Skip to content
A collection of small utilities written in shell or Python 3.
Python Shell
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore Added tests. Jun 10, 2016
LICENSE.txt Update license to MIT. Apr 16, 2018
README.rst Fix pylama warnings. Jul 29, 2019
clean.sh Update license to MIT. Apr 16, 2018
csv2tbl.py Replace str.format() calls with f-strings. Jul 7, 2018
csvcolumn.py
default_options.py Refactoring. Jul 30, 2019
denylog.py Fix pylama warnings. Jul 29, 2019
dicom2jpg.py Refactoring. Jul 30, 2019
dicom2png.py Simplify checks for requisites. Silence subprocesses. Jul 27, 2019
dvd2webm.py Simplify checks for requisites. Silence subprocesses. Jul 27, 2019
eps2png.sh Added eps2png. Nov 17, 2018
ffmutt.sh Update license to MIT. Apr 16, 2018
find-modified.sh Update license to MIT. Apr 16, 2018
find-pkg-upgrades.py Refactor configuration into separate function. Jul 30, 2019
fix-pdftitle.py Refactoring. Jul 30, 2019
fixbb.sh Update license to MIT. Apr 16, 2018
foto4lb.py Refactoring. Jul 30, 2019
genbackup.sh Update license to MIT. Apr 16, 2018
genotp.py Use secrets.choice in genotp. Aug 1, 2018
genpw.py Truncate passwords to size. Jun 30, 2019
get-tracks.py Refactoring. Jul 30, 2019
getbb.sh Add getbb.sh. Nov 13, 2018
git-check-all.py Refactoring. Jul 30, 2019
git-origdate.py Fix situation where git returns multiple dates. Jul 27, 2019
gitdates.py
graph-deps.py Replace str.format() calls with f-strings. Jul 7, 2018
histdata.py Fixes after tests run with pytest and bandit. Jul 27, 2019
img4latex.py Fixes after tests run with pytest and bandit. Jul 27, 2019
lk.py Replace str.format() calls with f-strings. Jul 7, 2018
make-flac.py
make-mp3.py Refactoring. Jul 30, 2019
markphotos.py Refactoring. Jul 30, 2019
missing-libs.py Fixes after tests run with pytest and bandit. Jul 27, 2019
mkhistory.py Fix pylama warnings. Jul 29, 2019
mkindexpic.sh Update license to MIT. Apr 16, 2018
mkpdf.sh Update license to MIT. Apr 16, 2018
nospaces.py Replace str.format() calls with f-strings. Jul 7, 2018
ntpclient.py Use context manager protocol with socket. Jul 30, 2019
offsetsrt.py Replace str.format() calls with f-strings. Jul 7, 2018
old.py Replace str.format() calls with f-strings. Jul 7, 2018
open.py Convert open.py to use a configuration file. Jul 29, 2019
osversion.py Print a warning and exit when not run on FreeBSD. Aug 19, 2018
pdfdiff.py Refactoring. Jul 30, 2019
pdfselect.sh
pdftopdf15.sh Process multiple files. Jul 8, 2019
povmake.sh Update license to MIT. Apr 16, 2018
py-ver.py Fix pylama warnings. Jul 29, 2019
pylama.ini Update pylama.ini. Apr 16, 2017
rename.py Replace str.format() calls with f-strings. Jul 7, 2018
scripts-tests.py Fixes after tests run with pytest and bandit. Jul 27, 2019
serve-git.sh Update license to MIT. Apr 16, 2018
set-ornata-chroma-rgb.py Add some extra debug messages. Jun 23, 2019
set-title.sh Update license to MIT. Apr 16, 2018
setres.sh Update license to MIT. Apr 16, 2018
sha256.py Replace str.format() calls with f-strings. Jul 7, 2018
standalone.sh Added standalone.sh May 2, 2018
statusline-i3.py Reconfigure stdout for line buffering. Jul 30, 2019
sync-to.sh Add license identifier. Oct 28, 2018
texfilehash.py
texfonts.sh Update license to MIT. Apr 16, 2018
tifftopdf.py Refactoring. Jul 30, 2019
tolower.sh Update license to MIT. Apr 16, 2018
vid2av1.py Simplify checks for requisites. Silence subprocesses. Jul 27, 2019
vid2mkv.py Refactoring. Jul 30, 2019
vid2mp4.py Refactoring. Jul 30, 2019
vid2webm.py Simplify checks for requisites. Silence subprocesses. Jul 27, 2019
webm.sh Update license to MIT. Apr 16, 2018
youtube-feed.py Set stdout to flush after every line. Jul 30, 2019

README.rst

Miscellaneous short utilities

tags:python, shell
Author: Roland Smith

Introduction

This is a collection of small utilities that I've written over the years. Some of them are simple front-ends for a utility with some standard options, to save me from having to recall the options every time I need them.

Another portion are basically Python front-ends to run a utility in parallel on different files.

All the functions in the python scripts come with documentation strings to explain what they do. The shell scripts have comments where necessary. They use basic sh syntax and to not use bash extensions.

All these programs are tested and in use on the FreeBSD operating system. The shell-scripts use the plain old sh that comes with FreeBSD, but should work with bash. Bug reports and patches welcome. Most of it should work on other BSD systems, Linux or OS-X without major problems.

All scripts use Python 3 specific features. Most recently I converted str.format calls to f-strings, meaning that you'll need Python 3.6 or later for the Python scripts.

Running tests

Tests for some of the functions used in the Python scripts are contained in scripts-tests.py. Running the tests requires pytest. Running the tests is done as follows:

pytest scripts-tests.py

Tests for security issues of the Python scripts can be done with bandit. I've run the tests as follows:

bandit -s B404 -x scripts-tests.py *.py | less

One might consider adding B603 and B607 to the exclusion list when the use of subprocess calls has been audited. These are:

  • B603 subprocess_without_shell_equals_true
  • B607 start_process_with_partial_path

When run with:

bandit -s B404,B603,B607 -x scripts-tests.py *.py

The file scripts-tests.py is excluded because it contains lots of assert calls as part of the testing mechanism. In this case, these are no cause for alarm.

The result should be: No issues identified.

Further general testing of Python scripts is done with pylama. The aim is to have no warnings or errors.

The programs

clean.sh

This script removes several types of generated files from the directory it is called from.

csv2tbl.py

Convert a CSV file to a LaTeX table.

default_options.py

This script generates a list of names of installed packages for which the options on the related port are equal to the default options.

denylog.py

This script reads /var/log/security or any other file that contains ipfw log messages, and makes an overview of incoming packages that have been logged.

This of course requires that blocked packets are logged!

If you are writing your own firewall script, make sure to use deny log instead of just deny.

dicom2jpg.py

A modification of the dicom2png program mentioned below to produce JPEG output. This is meant for situaties where lossy compression is acceptable.

dicom2png.py

Convert DICOM files from an x-ray machine to PNG format, remove blank areas. The blank area removal is based on the image size of a Philips flat detector. The image goes from 2048x2048 pixels to 1574x2048 pixels.

This program requires the convert` program from ImageMagick.

Multiple images are processed in parallel using a ThreadPoolExecutor from the concurrent.futures module to start subprocesses using as many worker processes as your CPU has cores. This number is determined by the os.cpu_count function, so this program requires at least Python 3.4.

dvd2webm.py

When I buy DVDs, I generally transfer their contents to my computer for easier viewing. However, the video and audio format used on DVD is not very compact. So I tend to use ffmpeg to convert it to smaller formats without losing quality. As of 2016, my favorite storage format is a webm container with a VP9 video stream and vorbis audio.

Initially I used the simple webm.sh script mentioned below. This had some shortcomings. It does not crop the video and cannot incorporate subtitles. It does enable multiple quality setting, but I seldomly used those.

The dvd2webm.py script performs a 2-pass encoding in constrained quality mode. Optionally it also adds subtitles to the video, and starts from an offset.

eps2png.sh

This script uses ghostscript to render encapsulated PostScript files to PNG format. Using command-line arguments the resolution and the type of PNG file can be changed.

ffmutt.sh

Small helper script to start mutt in an urxvt terminal for a mailto link.

find-modified.sh

Front-end for find to locate all files under the current directory that have been modified up to a given number of days ago.

find-pkg-upgrades.py

Script for FreeBSD to compare the versions of locally installed packages to the versions available from the FreeBSD package repo mirror. It will tell you which packages can be upgraded via pkg upgrade, and which have to be built from source.

fixbb.sh

Corrects the BoundingBox for single-page PostScript documents. It requires the ghostscript program.

foto4lb.py

Scales fotos for including them into LaTeX documents. The standard configuration sets the width to 886 pixels and sets the resolution to 300 dpi. This gives an image 75 mm (about 3 in) wide.

genbackup.sh

Generates a backup of the directory it is called from in the form of a tar-file. The name of the backup file generally consists of;

  • the word backup,
  • the date in the form YYYYMMDD,
  • the short hash-tag if the directory is managed by git.

These parts are separated by dashes, and the file gets the .tar extension. It requires the tar program. Tested with FreeBSD's tar. Should work with GNU tar as long as you don't use the -x option; the exclude syntax is different between BSD tar and GNU tar.

genotp.py

Generates an old-fashioned one-time pad; 65 lines of 12 groups of 5 random capital letters. Each pad has a header line containing a random identifier. It was inspired by reading Neal Stephenson's Cryptonomicon.

It uses random numbers from the operating system via Python's os.urandom function.

A partial example:

+++++ KWSNKYJLFF +++++
01  WAGGB HJVHQ TTQPD LQUMD KFRFS GGCKA SVLLA WEUCS HTXNI DITNW RBZKM SEGGW
02  GDSBB XECBL AUVLQ TUDPO DTXKW MWGAV DLRXT NRYAH HTGII YXEJJ JLNRC BIVDX
03  JDQUJ QPAUT CUEHN RHIHT QYBGV WOVAQ MKVZQ WPRGL QJAVA RPLRS AXIII FKLEP
04  WXYAD JNSAQ LBRXE QLCUX ZCLIE WPHSO OZBNH ZQLVN FAUEZ IDAJY VPQJN WVCAD
05  BEYRE WORKU CPEGE JKKWZ XUVYU WSZXQ NOULH QOFDQ PREMG YJBIT GMOAM USKLV
06  ZVATP YSRWH EEQDV LIPVQ FVYSY CIICG JKMOA RFJYE RUDJG HHJXI NNPNU VERMN
07  WAHFD WGGGN GHIUM BCJNN CVBCK QXYGZ PEYLW XOGMT SJFQJ NWEBE BFBPJ IDHDB
08  NPPEG HNONE YCJTG BFSFA NFYUR CMCGD XSKRO NSRBX WSDDX MEMLX BBMLC IMDJL
09  PZNAK OCOXA PEGNL UAWQW YCVDM WBNZZ YQICH MTLBG LDQTW TQMCS KUYBN RUNXT
...

Testing /dev/random on FreeBSD

My impression is that the random data device on FreeBSD is pretty good;

> ./ent -u
ent --  Calculate entropy of file.  Call
        with ent [options] [input-file]

        Options:   -b   Treat input as a stream of bits
                -c   Print occurrence counts
                -f   Fold upper to lower case letters
                -t   Terse output in CSV format
                -u   Print this message

By John Walker
http://www.fourmilab.ch/
January 28th, 2008

> dd if=/dev/random of=rdata.bin bs=1K count=1K
1024+0 records in
1024+0 records out
1048576 bytes transferred in 0.086200 secs (12164455 bytes/sec)

> ./ent rdata.bin
Entropy = 7.999857 bits per byte.

Optimum compression would reduce the size
of this 1048576 byte file by 0 percent.

Chi square distribution for 1048576 samples is 208.12, and randomly
would exceed this value 98.57 percent of the times.

Arithmetic mean value of data bytes is 127.5057 (127.5 = random).
Monte Carlo value for Pi is 3.137043522 (error 0.14 percent).
Serial correlation coefficient is 0.000771 (totally uncorrelated = 0.0).

According to the manual page, Wikipedia and other sources I could find the FreeBSD random device is intended to provide cryptographically secure pseudorandom data.

genpw.py

Generates random passwords. Like genotp, It uses random numbers from the operating system via Python's os.urandom function and converts them to text using base64 encoding. On FreeBSD I think this is secure enough given the previous section.

An example:

> python3 genpw.py -l 24 -g 4
BU_7 7RcI jjce zAKo 83v8 RAk_

getbb.sh

Determines the bounding box of PostScript files using ghostscript.

get-tracks.py

After using lsdvd to see the tracks on a DVD, this script can be used to extract the required tracks for viewing or transcoding.

It sxtracts the given tracks from a DVD using tccat from the transcode package.

git-check-all.py

Find all directories in the user's home directory that are managed with git, and run git gc on them unless they have uncommitted changes.

git-origdate.py

For all command-line arguments, print out when they were first checked into git.

gitdates.py

For each file in a directory managed by git, get the short hash and data of the most recent commit of that file.

graph-deps.py

Used with FreeBSD's pkg info and dot from the graphviz port to graph dependencies between packages.

histdata.py

Makes a histogram of the bytes in each input file, and calculates the entropy in each file.

img4latex.py

A program to check a PDF, PNG or JPEG file and return a suitable LaTeX figure environment for it.

this program requires ImageMagick program identify.

This program also requires the ghostscript interpreter to determine the size of PDF files.

As of version 1.4 it reads the text block width and height in mm from an INI-style configuration file named ~/.img4latexrc. A valid example is shown below.

[size]
width = 100
height = 200

The image is scaled so that it fits within the text block. If a bitmapped image does not have a defined resolution, 300 pixels/inch is assumed.

lk.py

Lock down files or directories.

This makes files read-only for the owner and inaccessible for the group and others. Then it sets the user immutable and user undeletable flag on the files. For directories, it recursively treats the files as mentioned above. It then sets the sets the directories to read/execute only for the owner and inaccessible for the group and others. Then it sets the user immutable and undeletable flag on the directories as well.

Using the -u flag unlocks the files or directories, making them writable for the owner only.

As usual, I wrote this to automate and simplify something that I was doing on a regular basis; safeguarding important but not often changed files.

The os.chflags function that is used in this script is only available on UNIX-like operating systems. So this doesn't work on ms-windows.

make-flac.py

Encodes WAV files from cdparanoia to FLAC format. Processing is done in parallel using as many subprocesses as the machine has cores. Album information is gathered from a text file called album.json.

This file has the following format:

{
    "title": "title of the album",
    "artist": "name of the artist",
    "year": 1985,
    "genre": "rock",
    "tracks": [
        "foo",
        "bar",
        "spam",
        "eggs"
    ]
}

make-mp3.py

Works like make-flac.py but uses lame to encode to variable bitrate MP3 files. It uses the same album.json file as make-flac.

markphotos.py

This scripts adds a copyright notice to pictures.

Warning

You should edit this script and update the cr string in the processfile function to contain your details before using this script!

Note

This script requires exiftool.

mkhistory.py

This script takes the git log --oneline history from the current working directory and formats it as LaTeX text with one commit per line. This is written to a given output file or standard output if - is used as the file name.

mkindexpic.sh

Use montage from the ImageMagick suite to create an index picture of all the files given on the command-line.

mkpdf.sh

Use convert from the ImageMagick suite to convert scanned images to PDF files.

It assumes that images are scanned at 150 PPI, and the target page is A4.

nospaces.py

Replaces whitespace in filenames with underscores.

ntpclient.py

A very simple NTP query and time setting program. It doesn't pretend to be extremely accurate.

offsetsrt.py

Reads an SRT file and applies the given offset to all times in the file. This time-shifts all subtitles.

old.py

Renames a directory by prefixing the name with old-, unless that directory already exists. If the directory name starts with a period, it removes the period and prefixes it with old-dot.

open.py

This Python script is a small helper to open files from the command line. It was inspired by a OS X utility of the same name.

A lot of my interaction with the files on my computers is done through a command-line shell, even though I use the X Window System. One of the things I like about the gvim editor is that it forks and detach from the shell it was started from. With other programs one usually has to explicitly add an & to the end of the command.

Then I read about the OS X open program, and I decided to write a simple program like it in Python.

The result is open.py. Note that it is pretty simple. and the programs that is uses to open files are geared towards common use. So text files are opened in an editor, while photos and most other types are opened in a viewer. This simplicity by design. It has no options and it only opens files and directories. I have no intention of it becoming like OS X's open or plan9's plumb.

This utility requires the python-magic module.

The filetypes and othertypes dictionaries in the beginning of this script should be changed to suit your preferences.

osversion.py

Prints the value __FreeBSD_version, aka OSVERSION.

pdfdiff.py

Uses pdftotext and diff to generate a unified diff between two PDF files.

pdfselect.sh

Select consecutive pages from a PDF document and put them in a separate document. Requires ghostscript.

pdftopdf.sh

Rewrite a PDF file using ghostscript.

povmake.sh

Front-end for POV-ray with a limited amount of choices for picture size and quality.

py-ver.py

List or set the __version__ string in all Python files given on the command line or recursively in all directories given on the command line.

rename.py

Renames files given on the command line to <prefix><number>, keeping the extension of the original file. Example:

> ls
img_3240.jpg  img_3246.jpg  img_3252.jpg  img_3258.jpg  img_3264.jpg
img_3271.jpg  img_3277.jpg  img_3241.jpg  img_3247.jpg  img_3253.jpg
img_3259.jpg  img_3265.jpg  img_3272.jpg  img_3278.jpg  img_3242.jpg
img_3248.jpg  img_3254.jpg  img_3260.jpg  img_3266.jpg  img_3273.jpg
img_3279.jpg  img_3243.jpg  img_3249.jpg  img_3255.jpg  img_3261.jpg
img_3267.jpg  img_3274.jpg  img_3280.jpg  img_3244.jpg  img_3250.jpg
img_3256.jpg  img_3262.jpg  img_3269.jpg  img_3275.jpg  img_3245.jpg
img_3251.jpg  img_3257.jpg  img_3263.jpg  img_3270.jpg  img_3276.jpg

> rename -p holiday2014- -w 3 img_32*

> ls
holiday2014-001.jpg  holiday2014-009.jpg  holiday2014-017.jpg
holiday2014-025.jpg  holiday2014-033.jpg  holiday2014-002.jpg
holiday2014-010.jpg  holiday2014-018.jpg  holiday2014-026.jpg
holiday2014-034.jpg  holiday2014-003.jpg  holiday2014-011.jpg
holiday2014-019.jpg  holiday2014-027.jpg  holiday2014-035.jpg
holiday2014-004.jpg  holiday2014-012.jpg  holiday2014-020.jpg
holiday2014-028.jpg  holiday2014-036.jpg  holiday2014-005.jpg
holiday2014-013.jpg  holiday2014-021.jpg  holiday2014-029.jpg
holiday2014-037.jpg  holiday2014-006.jpg  holiday2014-014.jpg
holiday2014-022.jpg  holiday2014-030.jpg  holiday2014-038.jpg
holiday2014-007.jpg  holiday2014-015.jpg  holiday2014-023.jpg
holiday2014-031.jpg  holiday2014-039.jpg  holiday2014-008.jpg
holiday2014-016.jpg  holiday2014-024.jpg  holiday2014-032.jpg
holiday2014-040.jpg

scripts-tests.py

This is just a collection of tests for functions from the different Python scripts.

serve-git.sh

Start a git daemon for every directory under the current working directory that is under git control.

set-ornata-chroma-rgb.py

This changes the color or the LEDs on a Razer Ornata Chroma keyboard to a static RGB color. It should work on operating systems that support pyusb, without requiring a kernel driver like the openrazer driver for Linux.

The openrazer driver served as an inspiration and source of information about Razer's USB protocol. At first I contemplated porting this driver to FreeBSD. But the differences between Linux and FreeBSD would make that a complete rewrite. Not to mention that the openrazer driver contains much more functionality than I need. Since FreeBSD comes with libusb, and supports pyusb you can pretty much control USB devices from user space with Python. So that's what I did.

set-title.sh

Set the title of the current terminal window to the hostname or to the first argument given on the command line.

setres.sh

Sets the resolution of pictures to the provided value in dots per inch. Uses the mogrify program from the ImageMagick suite.

sha256.py

A utility written in pure Python to calculate the SHA-256 checksum of files, for systems that don't come with such a utility.

standalone.sh

Compiles a LaTeX file with the standalone documentclass to Encapsulated PostScript format.

statusline-i3

A small Python script that replaces conky for me on FreeBSD with the i3 window manager.

sync-to.sh

This script was written to simplify the syncronization of data between different computers using rsync(1).

It assumes that:

  • The other host you are synchronizing to is running the rsync(1) daemon.
  • That host exposes /home as the [home] module.
  • You are syncronizing a directory in your $HOME to the same directory on the other host.

texfilehash.py

When given TeX file names, this program determines the short hash of last git commit that changed these file. When the original filename is <filename>.tex, this is written to a <filename>.hash. In the TeX file you can use \input to include the hash into the document. It is meant as a limited alternative to the vc bundle from CTAN.

texfonts.sh

This small shell script find Opentype fonts in my TeXlive installation and installs symbolic links to those font files in a single directory. This directory is then scanned by fc-cache to make the fonts available to all programs that use fontconfig.

tifftopdf.py

Convert TIFF files to PDF format using the utilities tiffinfo and tiff2pdf from the libtiff package.

tolower.sh

Changes the names of all the files that it is given on the command-line to lower case.

vid2mkv.py

Convert all video files given on the command line to theora / vorbis streams in a matroška container using ffmpeg. As of 3452c8a it uses a ThreadPoolExecutor.

vid2mp4.py

Analogue to vid2mkv.py, but converts to H.264 (using the x264 encoder) / AAC streams in an MP4 container.

webm.sh

Convert video files to VP9 video and Vorbis audio streams in a webm container, using a 2-pass process.

youtube-feed.py

Checks youtube for the latest video's from your favorite channels. Requires the requests module (version 2.x).

It also requires you to have a JSON-file called .youtube-feedrc in your $HOME directory. What this file should contain is documented in the script.

You can’t perform that action at this time.