Menu

This is all Header in this file:

1. Linux DeepLearning tools, view
2. Linux system, view
3. Check Envs for Deep Learning, view
4. Install Deeplearning Libraries, view
5. Python functions List, view
6. Keras funcs list, view

Link hay cần làm

Link hay cần làm

1. Colab

• colab running time:

!pip install ipython-autotime
%load_ext autotime

1. Linux DeepLearning tools

These are some useful tools to use with Linux (focus on Ubuntu) when working with Deep Learning. Tham khảo:

• 1. Install Ubuntu 18.04
  2. Install Nvidia-Driver: nvidia-smi
  3. Install cuda 9.0 : nvcc -V
  4. Install cudnn
  5. Install Anaconda
  6. install keras can run GPU: conda install tensorflow-gpu=1.12.0 cudatoolkit=9.0 -y (only need to install driver 390)
• Chạy trong Notebook, lấy thông tin của main, cpu, more

2. Linux system

Useful linux command

• htop: to view in %CPU uses by threads, %RAM: kill thread, filter, search,...
• ncdu: to view files/folders size over local or ssh
• find -type f -name "libcudnn.so" : find file/folder in Linux, more
• rm -fdr <dir_name>: Remove a dir with contents, more
• sudo lshw -class disk -short : List Physical Hardisk in Ubuntu, view
• Login to jupyter server from local machine , view
• Linux system: share folder / mount a remote dir to local , view
• Startup a program in ubuntu manualy , view
• Linux system: install code editor geany, better than gedit: code folding, show indent guide, show space,... , view
• Ubuntu: Install Sublime Text in linux: view
• Ubuntu: Create bootable install OS disk (ubuntu/window/...system): view
• Install VNCserver in ubuntu: good link
• Install Samba and Share file from linux to windows Xem
• Run Specific CUDA on GPU: CUDA_VISIBLE_DEVICES=2,3 python code.py or export CUDA_VISIBLE_DEVICES=0,1
• Add New sudo User ubuntu (add user): Xem
• Change language in ssh of ubuntu to english: Xem
• TensorBoard server client:(server): tensorboard --logdir=. --host=0.0.0.0 --port=6006 ;(Local): http://ip:6006 Xem
• Install Vivaldi, import/export pass: xem
• Install flash player ubuntu success [mothod1]: (view here)[https://www.wikihow.com/Install-Flash-Player-on-Ubuntu]
• FFMPEG Chia clip ra thành nhiều phần bằng nhau: Xem
• Ubuntu nén, giải nén, compress tar, zip folder: Xem
• Ubuntu/windows tìm tên thiết bị camera/webcam: /sbin/udevadm info --export-db | grep video
• ubuntu remove readonly files/folders permissions sudo chmod 777 -R *
• Tạo phím tắt/shortcut cho "Open terminal here" xem
• Ubuntu compress, zip, unzip files with progress: xem
• Copy file in Ubuntu with progress: xem
• folder size: !du -hs "dir/path/"
• Rút gọn path trong ubuntu terminal: PROMPT_DIRTRIM=1 (thêm vào cuôí ~/.bashrc)

3. Check Envs for Deep Learning

• nvidia-smi , watch -n 0.3 nvidia-smi, nvidia-smi -l 1
• nvcc -V
• find -type f -name "libcudnn.so"
• nvidia-smi --format=csv,noheader --query-gpu=index,name,temperature.gpu,fan.speed,pstate,power.draw,clocks.current.graphics, more
• Anaconda GUI in ubuntu: anaconda-navigator view more
• Ubuntu: Run ipython notebook with browser from on server: ipython notebook --no-browser --ip=0.0.0.0 --port=8889 :more
• ipython: ipython notebook width 100% , more
• Anaconda ipython Jupyter Notebook: Change startup folder of notebook: view
• Anaconda Jupyter Notebook: Run Remote Notebook on server from local: view

4. Install Deeplearning Libraries

• Install all librarys needing for Deeplearning with Keras for Speech: , View
• PYTHON - AUTO GENERATE REQUIREMENTS.TXT: pipreqs . (pipreqs /path/to/project): more
• pip install python-levenshtein
• How to install pytorch in Anaconda with conda or pip? View

5. Python functions List

• def LastNlines(NLs=15,LineContainKey="Key to Fine"), view
• Plot history and accuray when training with Keras to PDF, view
• Save data to json file, view
• Files_2csv_inDir: Find and Add All wav & label files to *.CSV, view
• Colab run time running time: !pip install ipython-autotime %load_ext autotime
• New calculate Running time + progress display: xem
• New calculate running time simple (rất đơn giản): xem
• Datetime Vietnam timezone+7: xem
• Timing: Calculate running time, view
• Get date time, month, day, hour, minute,...view
• Python: Many date time, unique file names: view
• Substring: Copy contends from txt file, add string of time, add ... view
• Get time of file to make filename
• Post (upload) file/string to PHP webpage, view
• Run a system Ubuntu command, view
• Move, Copy, delete file from Python, view
• Delay, sleep in python, view
• Pass arguments to program, call commandline args, view
• Load json with multiple json lines, view
• Python: Print json file out to screen: view
• Python: Padding a vector/matrix enlarge/make bigger view
• Python: Run ubuntu command without display with subprocess.Popen, view
• Python: Compress folders at current folder to tar.gz view
• Padding list 2D: view
• Convert list of 2D array to 3D array view
• Python: paralell CPUs, tính toán song song: pool.map_async()... view
• Python Notebook parallel CPUs:view
• Github markdown: auto make table of content: view
• Python Notebook run sudo command: view
• Python: install Pythable, view
• python Ramdom shuffle/choice list: random.shuffle(x) view
• Python Read image to list of array then Padding View
• Convert video+DarkLabel to YoLo label View
• import a file:

!wget -O taLibs_imports.py  https://github.com/holianh/Linux_DeepLearning_tools/raw/master/taLibs_imports.py
clear_output()
exec(open('taLibs_imports.py').read())

• TA progress: code
• get size of var:

    from sys import getsizeof
    print(getsizeof( datahome ),'bytes')

• get size of any object: import this
• download gdrive share file in colab: !gdown --id xxx hoặc: !gdown https://drive.google.com/uc?id=1-t9SO--H4WmP7wUl1tVNNeDkq47hjbv4 -O model.pth -q
• Python fastest way to get mp3 length code

6. Keras funcs list

this is all useful Keras functions, can be directly use

• Keras: save model + weight to files, here
• Keras: load model + weight from files to numpy array, here
• Tensorboard: local and remote , here
• Make model run in Multiple GPUs, From define input, to parallel model in multiple GPUs
• Keras: parallel GPUs Model training V2: here
• Plot, display model in jupyter notebook: here
• tensorboard : locate tensorboard/program.py | xargs sed -i -e "s/logdir', ''/logdir', os.getcwd()/g"
• Keras python jupyter notebook: Live plot Loss accuracy when training : here
• Keras check GPU exists: from tensorflow.python.client import device_lib;print(device_lib.list_local_devices())
• Cách dùng HDF5 để lưu/store large dataset code

7. WEB Services

• Build a front end web application: Send data without reload page Youtube
• Edit ipython Notebook html scroll field: div class="..." => div class="..." style="overflow-y: scroll; height:400px;"
• Server-Client Flask on Window: full code
• Server - Client predict: code
• Đọc file CSV=> array: code
• js post thông tin (jquery) yêu cầu xoá thông tin trên server: code
• Chỉnh sửa snippet của sublime text: code
• html to excel:code
• html max height, html-pre-p-div-max-height: code : <style>pre {max-height: 300px; overflow: auto; } </style>
• thu gọn/mở rộng html: <details> <summary>title or explanatory caption</summary> ... </details>
• Web Kết nối CSDL sqlite với PHP: code
• upload file and some other fields to php server: Code
• PHP sort multi column array sắp xếp mảng đa chiều

8. Ứng dụng khác

• Get link Fshare full speed: http://fullcrack.vn/get-link-fshare/ và https://tools.nhacm.com/
• vivaldi: darkmode: vivaldi://flags/#darken-websites-checkbox-in-themes-setting => Force Dark Mode for Web Contents = Enable

9. Tools

• Convert all imgs to jpg code
• List all filesize, dimension, duration,...code
• google spreadsheet: gsheet: Nhập data vào ô B4:E4, tự động copy, sắp xếp xuống vùng bên dưới code
• Cách active Window và office: Chạy CMD gán key mới tại đây: xem
• Window tool: Hiển thị pass wifi trên máy tính window: xem
• Xoá menu dual boot, xoá onekey ghost boot menu:

  1) chạy bcdedit
  2) Copy nội dung dòng Identifier (vd {curent} , {default}, {6fsdasdaa-qrtyytg-...}
  3) chạy bcdedit /delete <Identifier vừa copy> là xong, có thể chạy lại bcdedit để ktra.
  

10. Arduino

• Display: TFT 2.4" China Code
• Arduino send/get data to/from httpsCode

11. Best soft

• Quay phim ghi hình trực tiếp: OBS (dùng filter: MIC: noise suppression, Webcam: fileter, có nhiều cái hay lắm)
• Fastest video trimming: Avidemux (cắt bỏ từng khúc video siêu nhanh luôn, save cái là xong)
• Tải phần mềm bẻ khoá tại: http://fullcrack.vn khá chuẩn, server nhanh
• Sublime Text 4 tự bẻ khoá Ở đây, here
• License Pycharm pro: /home/aisolutions.vn/..../stores/phan_mem/pycharmpro: 88k 1 máy

12. QT-C++

• Ghi đọc file với C++ Code

13. Tạo bộ cài

• Tạo file chạy (1 file exe duy nhất) từ python (vào đúng chỗ, đang chạy bth): pyinstaller -F <tenfile.py> ví dụ: pyinstaller -F "VSM-cpp-h-txt-[backup].py"