Over the years, I have written a number functions to help making coding easier. I have used them in sparingly many machines, RedHat Linux, CentOS, Debian, macOS, etc. so I thought it might be a good idea to put these reusable functions into a collection so I can just source it and have all the functions available without a bunch of copy and paste. Perhaps the collection can be helpful to someone else too.
This is a collection of BASH functions, mostly for handling some form of automation for time-scheduled jobs. Here is a summary of the functions within the collection:
log()slog()num2str3()file_manager()remove_files_but_keep()remove_folders_but_keep()remove_minutes_old_files()remove_empty_dir()warn_if_low()check_process()fecho()textout()headtail()remove_minutes_old_files()mount_host()check_path()remove_old_logs()
More on blib.sh ...
SSHFS requires that FUSE framework and SSHFS tool are installed. On a Mac, they can be installed through Homebrew as
brew cask install osxfuse
brew install sshfsThese packages could make your shell experience more fun
fortunecowsaylolcat
A script that shows all the color codes for terminal.
A script to create a RAM disk. On macOS, the RAM disk is created at /Volumes/RAMDisk through diskutil. On Linux, the RAM disk is created at /mnt/ramdisk through the utility mount.
makeramdisk.sh SIZE [NAME]A simplified way to mount an remote NTFS path
mount_ntfs.sh REMOTE_ADDRESS_AND_PATHmount_ntfs.sh 10.203.1.100:/Dmounts the shared folder D on the machine 10.203.1.100 to /Volumes/ntfs
This script uses sshfs to mount a machine that can be reached with through SSH. Example usage is mount_host.sh deepsky to mount the root drive with partition name deepsky of the machine deepsky or deepsky.local. This naming scheme is what I usually practice.
mount_host.sh [COMPUTER_NAME]A script to use wget to download all the WSR-88D (NEXRAD) Level II .gz files from a URL, which comes with an email that is sent out when the order from the NCDC archive is ready. This was specifically developed for downloading multiple Level-II files but it may work for other products. It is set up to traverse two levels down so files with URL http://.../HASXXXX/0001/XXX_V06.gz, http://.../HASXXXX/0002/XXX_V06.gz, etc. will all be downloaded simply by providing the top level link, which is supplied in the email. All files will be downloaded in ~/Downloads/HAS000000000/ where the the last path component represents the order number. If the destination folder exists, the script will not overwrite it but quit.
ncdc.sh URL
A command line to remove extra attributes, which can be seen using ls -l@. Some annoying tags like com.apple.quarantine will be removed using this utility.
rmtags.sh FILES
A shell script wrapper to retrieve internet speed test
speedtest.sh
ping = 6.216 ms
download = 1666.00 Mbps
upload = 1505.66 MbpsWake up a machine connected through LAN. This is a convenient function to wake up several computers by name. It may not work for you.
wake.sh COMPUTER_NAMESome global variables for general purposes:
LOGFILEfor logging.VERBOSEfor printing out more information on screenLOG_DATE_FORMATfor short (1) or long (2) format time stamp
A collection of convenient functions:
log()slog()num2str3()file_manager()remove_files_but_keep()remove_folders_but_keep()remove_minutes_old_files()remove_empty_dir()warn_if_low()check_process()fecho()textout()headtail()remove_minutes_old_files()mount_host()check_path()remove_old_logs()
Some functions are better explained with examples so here they are:
logs an entry with timestamp.
log MESSAGE
log "Hello World."
logs an entry
1/28 02:49:29 PM : Hello World.
in the logfile, which is described by the global variable LOGFILE.
logs an entry with short timestamp. This is aimed for log files that are named by day so there is no need to put a date in each log entry.
slog MESSAGE
slog "All done."
logs an entry
02:49 : All done.
in the logfile.
frees up space or limits the usage until the targeted number is achieved.
file_manager MODE FOLDER SIZE
file_manager LIMIT ${HOME}/figs 1024*1024*1024
file_manager FREE ${HOME}/data 1024*1024*1024
where the first line limits usage of the folder to 1 GB; while the second line ensures the free space of the partition where the folder belongs to is at least 1 GB.
removes all files in a folder but keep the last specified number of files.
remove_files_but_keep FOLDER COUNT PATTERN
remove_files_but_keep "${HOME}/logs/drive_clean" 30 '*.log'
removes all files under the subfolder logs/drive_clean under home folder, which that match the file pattern that ends with "log" but keep the last 30.
removes all sub-folders in a folder but keep the specified number of sub-folders, which is determined by pre-sorting the list alphabetically.
remove_folders_but_keep FOLDER COUNT [PATTERN]
where the file pattern is assumed to be * if not supplied.
remove_folders_but_keep "${HOME}/data" 5 '20[0-9][0-9]*'
removes all sub-folders that match the naming pattern begins with 2000 - 2099 but keep the last 5.
removes files older than the specified age in minutes.
remove_minutes_old_files FOLDER COUNT [PATTERN]
where the file pattern is assumed to be anything if not supplied.
remove_minutes_old_files "${HOME}/px1000" 720 'PX*.tgz'
removes all files (up to level-2 deep) that are older than 720 minutes and satisfy the name pattern which begins with "PX" and ends with "tgz".
removes empty folders
remove_empty_folders FOLDER
remove_empty_folders "${HOME}/data"
removes all sub-folders that are empty. Note that hidden files that begin with ".", e.g., ".DS_Store", can prevent this function from removing the folders.
produces a message if the partition (of where the folder belongs to) is less than the test size.
warn_if_low FOLDER SIZE
warn_if_low "${HOME}/data" 5*1024*1024*1024
produces a message
Available: 4.9 GB < 5 GB
when the remaining space is 4.9 GB, which is less than the specified test size. Otherwise, nothing happens. This can be used in a script and scheduled as a cron job to notify the admin when the free space is too little.
checks for processes using ps and grep
check_process PROCESS_1 PROCESS_2 PROCESS_3 ...
check_process rpcd rcc lcc trxd trigd
produces output
Processes |
========= |
RUSER PID STAT %CPU %MEM NLWP COMMAND |
boonleng 32327 Sl 0.1 0.0 2 iqc |
boonleng 32330 Sl 0.1 0.0 4 iqd |
boonleng 32333 Sl 0.0 0.0 3 rcc |
prints out the piped-in text with color and title
textout TITLE [COLOR]
tail -n 5 ${LOG} | textout RCC green
produces output
RCC |
=== |
12:24:39 : [RCC] 00 <Kate>: 108-'l' (1) |
12:24:39 : [RCC] From RCC/LCC: ACK. LCC connected. |
12:24:39 : [RCC] LCC connected. |
12:24:58 : [RCC] 00 <Kate>: Hangging up... ST:Inactive |
shows the head and tail portions of a file list of a folder
headtail FOLDER [COLOR]
headtail ~/Downloads
produces
/Users/boonleng/Downloads ( 16 --> 59G) |
============================================ |
drwx------ 3 boonleng staff 102B Apr 16 2014 About Downloads.lpdf |
-rw-r--r-- 1 boonleng staff 1.1M Nov 25 2013 Scans.pdf |
-rw-r--r-- 1 boonleng staff 885M Nov 30 2014 simradar.mp4 |
: |
-rw-r--r-- 1 boonleng staff 33K Nov 17 22:03 nvrambak.bin |
-rw-r--r-- 1 boonleng staff 621K Nov 8 09:51 opencl-1-2-quick-reference|
drwxr-xr-x 7 boonleng staff 238B Nov 25 21:22 tables |
removes files in a folder that are older than a specified age in minutes and match a given pattern.
remove_minutes_old_files DIR MINUTES [PATTERN]
where PATTERN is assumed to be *.tgz if not supplied.
remove_minutes_old_files /home/ldm/data 86400 *.nc
mount the host using SSHFS
mount_host HOST_NAME
mount_host anastasia
mount_host cerulean.local
mount_host 10.203.6.227
checks if the path exist
check_path PATH
check_path /Volumes/Data
remove old log files that are in the patterns of:
cleanup-20171124.log
cleanup-20171125.log
cleanup-20171126.log
...
stitch-figure-20171124.log
stitch-figure-20171125.log
stitch-figure-20171126.log
...
remove_old_logs LOG_PATH [FILES_TO_KEEP]
where FILES_TO_KEEP is assumed to be 7 if not supplied.
remove_old_logs ${HOME}/logs 30