poorquery

Matches regex-identified png-files anywhere on a filesystem and outputs an HTML file displaying them

NAME:

poorquery.sh

SIGNATURE:

poorquery.sh <bash path, including globs/wildcards, IN SINGLE/DOUBLE QUOTES>

DEPENDENCIES:

• Pandoc (This is rather large, around >700MB, as you must install the GHC Haskell compiler, unless you install a simple binary version. If you get to know/use Pandoc, though, you will NOT regret it, it's awesome!)
• a Unix/Linux OS (maybe?), or Cygwin on Windows (untested)
• sed (Only needs install on Windows. OS X version untested)
• this script makes, and then removes, a bunch of temp files like temp1.txt and temp6.md, so if you have files called that then this will delete them.

INSTALL:

0. (Install any dependencies listed above.)

1. Clone this repo into a directory.

2. Add aforementioned directory to your $PATH. This is most easily done by editing your .bashrc in your home directory and adding this line to the end:

    export PATH="$PATH:/<path>/<to>/<your>/<poorquery>/<git>/<repo>"
   

DESCRIPTION:

This tiny little bash script takes a single, bash-wildcard/regex-enabled path, finds all .png files in every subdirectory that matches, and then builds an HTML file (named poorquery_output.html in the working directory) to show the filename of all matches alongside the ACTUAL pictures. I.e., when you open the HTML file in your browser, you'll see both the filename and the corresponding picture.

The pictures are not stored in the HTML file but are rather referenced directly on your hard drive from them, so the HTML files are small. This way, if you have very many picture files that are named specifically enough for you to treat either their directories or filenames as bash-style regular expressions, you can then visualize a selected subset of them quickly via a single regex. Because the really optimal way would be serious SQL-querying a full on database of all your picture/simulation results, for the moment I'm calling this Poor Query, since this is essentially a low tech way to quickly visualize data that is distributed in regex-filename capable filesystem hierarchies rather than databases.

EXAMPLES:

Say you have some simulations' result pictures you want to visualize, which are, of course, organized in a complicated naming scheme, like

• sciencing
  • data_20150526
  • data_20150527
    • p001d001q022c001
    • p001d001q023c001
    • p001d001q023c002
      • raw_v
        • p001d001q023c002__T000.png
        • p001d001q023c002__T001.png
        • p001d001q023c002__T002.png
        • p001d001q023c002__T003.png
      • raw_spikes
        • p001d001q023c002__T000.png
        • p001d001q023c002__T001.png
      • raw_whatevs
        • p001d001q023c002_gH0pt001_gEsyn0pt02_T000.png
        • p001d001q023c002_gH0pt001_gEsyn0pt04_T001.png
        • p001d001q023c002_gH0pt002_gEsyn0pt02_T002.png
        • p001d001q023c002_gH0pt002_gEsyn0pt04_T003.png

Let's assume your current working directory is the sciencing directory. You could easily visualize all the pictures from 2015-05-27 with just:

    poorquery.sh 'data_20150527'

Or if you wanted all the raw_v figures for just the '...q023...' simulations, and for all of May:

    poorquery.sh 'data_201505*/*q023*/raw_v/*'

The best example there is is how I actually use it: when the individual figure filenames themselves have some parameter you're varying over, like for an N-dimensional batch of parameter sweeping simulations, you often want to hold one parameter fixed and vary the others. Say I want to see how changing the gH (in raw_whatevs) changes the outcome of the simulations, while holding gEsyn constant -- I would just run:

    poorquery.sh '*/*q023*/raw_whatevs/*gH*gEsyn0pt02*'

Or, equivalently in this case:

    poorquery.sh '*/*q023*/raw_whatevs/*gEsyn0pt02*'

Thus, you have "bash-style regular expression"-level control over viewing the hoowever-nested pictures you have...which is a lot of power!

TODO:

• Stop using temp files, and instead use bash variables (which are harder in terms of handling newlines, it seems...)