Various scripts made in the process of being a sysadmin, that are too random to get their own repository.
Sorted roughly in order of how much I seem to actually use them.
Most are very dumb and singular, and that's often the point.
There's a lot of "copy helpers into them" going on to make them independent scripts.
In cleaning up storage, you may also find https://github.com/scarfboy/file-count and https://github.com/scarfboy/duppy useful
List the largest files under the given path(s).
# file-largest /opt /tmp
Reading '/opt'...
Reading '/tmp'...
Done
100MB '/opt/google/chrome/chrome'
101MB '/opt/scipion-2016-01-30/.git/objects/pack/pack-043cf8c2d2435990ca2a8c6d489dd7c454eb29f0.pack'
104MB '/opt/scipion/.git/objects/pack/pack-f826a4cffb9f8afd8ade60c07d33a349d6ce4c28.pack'
124MB '/opt/PEET/ParticleRuntime/v84/sys/jxbrowser-chromium/glnxa64/chromium/libjxbrowser-chromium-lib.so'
177MB '/opt/PEET/ParticleRuntime/v84/bin/glnxa64/libnppi.so.6.0.37'
239MB '/opt/coot-Linux-x86_64-ubuntu-14.04-gtk2-python/libexec/coot-bin'
7.7GB '/opt/R2015b_glnxa64.iso'
Reports the most recently altered files under a path (via ctime/mtime). Helps find what you / other people / programs were recently working on.
# file-recent ~
Looking in '/root'
including files where max(ctime,mtime) is in the last 100 days
will sort results by age
and show the 5 most recent files
2mo 2wk : /root/.mozilla/firefox/Crash Reports/InstallTime20160606114208
5dy 2hr : /root/.lesshst
1hr 24min: /root/.Xauthority
40min 33sec: /root/.bash_history
35min 33sec: /root/.local/share/mc/history
You know when ssh refuses to do something because it doesn't like your permissions? This suggests chmod commands to fix that (and a few restrictive things, just because) (TODO: also consider chowns, in case you have the wrong owner)
$ sudo ssh-permission-check
chmod 750 '/root/.ssh' # currently: 770
chmod 600 '/root/.ssh/bridgetrust.pub' # currently: 644
# Checked users: repository, root, tunneler
lsof with a grep that filters out most not-really-file stuff. Verrry simple.
Tries to summarize what various people are up to
- shows process names -- and filters out boring stuff, particularly for root
- shows working dir for shells (and some others)
- mentions nontrivial CPU / wait state
# otherpeople
root
4 * bash
in '/var/www/coding/github-sys-tools'
in '/root'
in '/var/www/coding/github-file-tools'
3 * sshd
3 * smbd
2 * tmux
fancontrol
wolforward
fail2ban-server
john 99%CPU
in '/root'
master
emacs
docker
tunneler
sshd
me
Xtightvnc
x-window-manage
xstartup
dd 52%CPU waiting on IO <--------
bash
in '/home/me'
Reads a file. So that it's in the page cache. (note: you may care to know about the vmtouch utility)
# readin -w png -w jpg -r /data/images
Reading: /data/images/eb8f2be459aad473b5dd64680d78b810ddf2a8cf.jpg
Reading: /data/images/bdff38ce75cbbd76803c4e25478faa64553d0b08.jpg
$ file-caseclash /data
# Issues under '/data/NerdDocs/learning'
'/data/NerdDocs/learning/[learn] N-grams, smoothing.pdf'
'/data/NerdDocs/learning/[learn] N-grams, Smoothing.pdf'
# Issues under '/data/Installs/Game/Older - C64/unsorted/'
'/data/Installs/Game/Older - C64/unsorted/README.TXT
'/data/Installs/Game/Older - C64/unsorted/Readme.txt
'/data/Installs/Game/Older - C64/unsorted/README.txt
'/data/Installs/Game/Older - C64/unsorted/readme.txt
Checks directories for file and directory entries that will be confusing to case-insensitive filesystem APIs ( such as windows's)
When you want to handle files soon after they come in, but not while still being written, a bit of backoff time is useful.
Meant to feed things to xargs, in that it checks that files that match a given shellglob, that sit under a given directory, have not changed size/ctime in a while (currently hardcoded to ~10 seconds).
Once that's all true it prints their path once and never again.
I use this to compress data that's being copied into a storage server, as a more informed backoff so that it's surer to be done copying:
file-stable '*.dat' /data/archive | xargs -t -n 1 -P 3 pigz -3
If you break off rsync, it can leave behind temporary files, which are lonely (no file alongside) or stale (much older than the real file alongside).
This looks for them, checks that they're not recent, and reports and optionally removes them.
Reports total amount and size, per file extension under a path.
Helps answer questions like
- "if I delete just the cache files, how much space would that free?"
- "how useful might it be compress stuff in here?"
- "how long would this take to copy just the gzipped version?"
# file-summarize-extensions /home
Final report (top 10, by count):
html: 138.5MiB, 6088 files
None: 10.59GiB, 4696 files
gif: 41.1MiB, 3005 files
java: 19.5MiB, 1944 files
pyc: 11.78MiB, 1070 files
class: 2.98MiB, 1001 files
png: 72.1MiB, 922 files
htm: 12.61MiB, 631 files
jpg: 295.3MiB, 631 files
pdf: 1.44GiB, 563 files
Takes a list of filenames on stdin, stats them and sums their size and count.
I mostly use this as a more targeted variant of file-summarize-extensions
# /tmp files that haven't changed in the last two weeks
# find /tmp -ctime +14 | file-totalsize
63.4MiB / 66.5MB (66491289 bytes) in 240 files
# JPGs from cameras
# find /data/images -iname '*.jp*' | egrep ^DSC | file-totalsize
121MiB / 127MB (127194089 bytes) in 15 files
TODO: report both apparent and disk usage.
As a sysadmin, it's nice to quickly see who your inactive users and your abusers are.
# file-estimate-homediruse
User 'myusername'
INFO: Size: 9.7GB
INFO: Youngest file: 2mo 5dy
Some of the larger and recent files:
3mo 3dy Dropbox/morework/titan.png
4mo 7hr Dropbox/Zeck-PD6.12_8.12_12.16 pwrmix.pdf
3mo 3wk Dropbox/relay_h-550_en.pdf
3mo 1dy Dropbox/CMaps/project.idx
User 'repository'
INFO: Size: 13.2GB
INFO: Youngest file: 4min 9sec
Some of the larger and recent files:
4min 9sec workscripts/db/rep-cache.db
29min 13sec workscripts/db/revs/0/707
19hr 10min rep/db/revs/0/930
19hr 13min supportingdata/db/revs/0/6
User 'tunneler'
INFO: Size: 6KB
INFO: Youngest file: 7mo 3wk
No recently modified files.
Cleanup for all.
Option parsing for all.