Skip to content
Toolkit for manipulating data in an ASCII table
Perl C Makefile Python Shell
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
completions
lib
packaging
test
.gitattributes
.gitignore
Changes
Makefile
README.org
README.template.org
b64_cencode.c
vnl-align
vnl-filter
vnl-gen-header
vnl-join
vnl-make-matrix
vnl-sort
vnl-tail
vnl-ts
vnl-uniq
vnlog-base64.h
vnlog.c
vnlog.h

README.org

Talk

I just gave a talk about this at SCaLE 17x. Presentation lives here.

Summary

Vnlog (pronounced “vanillog”) is a trivially-simple log format:

  • A whitespace-separated table of ASCII human-readable text
  • Lines beginning with # are comments
  • The first line that begins with a single # (not ## or #!) is a legend, naming each column
  • Empty fields reported as -

Example:

#!/usr/bin/whatever
# a b c
1 2 3
## another comment
4 5 6

Such data works very nicely with normal UNIX tools (awk, sort, join), can be easily read by fancier tools (numpy, matlab, excel, etc), and can be plotted with feedgnuplot. This tookit provides some tools to manipulate vnlog data and a few libraries to read/write it. The core philosophy is to avoid creating new knowledge as much as possible. Consequently, the toolkit relies heavily on existing (and familiar!) tools and workflows. As such, the toolkit is small, light, and has a very friendly learning curve.

Synopsis

I have two sets of historical stock data, from the start of 2018 until now (2018/11):

$ < dji.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000

$ < tsla.vnl head -n 4
# Date Open High Low Close AdjClose Volume
2018-11-15 342.33 348.58 339.04 348.44 348.44 4486339
2018-11-14 342.70 347.11 337.15 344.00 344.00 5036300
2018-11-13 333.16 344.70 332.20 338.73 338.73 5448600

I can pull out the closing prices:

$ < dji.vnl vnl-filter -p Close | head -n4
# Close
25289.27
25080.50
25286.49

And I can plot them:

$ < dji.vnl vnl-filter -p Close |
    feedgnuplot --lines --terminal 'dumb 80,30' --unset grid

  27000 +-------------------------------------------------------------------+
        |        *    +            +             +            +             |
        |        * *                                                        |
        |        ***                                            *           |
  26500 |-+     * ***                                           *         +-|
        |       * * *                                           *           |
        | *     *   **                                         * *          |
  26000 |-*     *   *** *                                      * **       +-|
        | *     *    ****                                      *  *         |
        | *    **       **                                     *  *         |
        | *   ***        ** *                              *   *   *        |
  25500 |*+*  ***         *** *                            *   *   *      +-|
        |* * ****         ** **       **                * **   *   *        |
        |* *** *          ** * **     **                * ***  *    *       |
        |*  ** *          *    **     * *               * ***  *    *       |
  25000 |-+ **                 ***   *  *  *           ** ***  *    *     +-|
        |   **                   **  *  * ***    *     ****** *      *      |
        |   **                   **  *  ** **    *    **  *  **             |
        |   **                    * **  **   *   *    **  *  **             |
  24500 |-+ *                     * **   *   *  * **  *      **           +-|
        |                         ***    *   **** *** *      **             |
        |                         **          *** *****      **             |
  24000 |-+                                   ***  ****       *           +-|
        |                                     *    ****       *             |
        |                                           ***                     |
        |             +            +             +  **        +             |
  23500 +-------------------------------------------------------------------+
        0             50          100           150          200           250

For this documentation I’m plotting in ascii (“dumb” terminal). Normally you’d use the default terminal (by omitting --terminal ...), and you’d get interactive, graphical plots. Here I kept only the closing price column, so the x-axis is just the row index. The data was in reverse chronological order, so this plot is also in reverse chronological order. Let’s fix that:

$ < dji.vnl vnl-sort -k Date |
    vnl-filter -p Close |
    feedgnuplot --lines --terminal 'dumb 80,30' --unset grid

  27000 +-------------------------------------------------------------------+
        |             +            +             +          * +             |
        |                                                 * *               |
        |    *                                            * *               |
  26500 |-+  *                                            ** *            +-|
        |    *                                            ** *              |
        |   ***                                         **   *     *        |
  26000 |-+** *                                      *****    *    **     +-|
        |  *  *                                      ****     *    **       |
        |  *   *                                    **        **   **       |
        |  *   *   *                             *  **        **   **       |
  25500 |-*    *   *                           * ***          **  * *     +-|
        | *    *  ** *                **       *****          *** * *       |
        |**    * *** *                **      *** **          * * * **      |
        |*     * *** *                ***    ***   *          * * * *       |
  25000 |*+    * *** **            *  * *   ***                 ***       +-|
        |*     * ******     *    * * *  *   *                   **          |
        |      * * ** **    **   ** **  *   *                   **          |
        |      **  **  *    **  *   **   *  *                   **          |
  24500 |-+    **      *  ****  *   **   * *                     *        +-|
        |      **       * ***** *   *    ***                                |
        |       *       * ** ****         **                                |
  24000 |-+     *       ***  ***                                          +-|
        |       *       ***    *                                            |
        |               **                                                  |
        |             + **         +             +            +             |
  23500 +-------------------------------------------------------------------+
        0             50          100           150          200           250

This is in the correct order, but it’d be nice to see the actual dates on the x-axis:

$ < dji.vnl vnl-filter -p Date,Close | head -n4
# Date Close
2018-11-15 25289.27
2018-11-14 25080.50
2018-11-13 25286.49

$ < dji.vnl vnl-sort -k Date |
    vnl-filter -p Date,Close |
    feedgnuplot --lines --unset grid --terminal 'dumb 80,30'
                --timefmt %Y-%m-%d --domain

  27000 +-------------------------------------------------------------------+
        |     +     +     +      +     +     +     +     +      *     +     |
        |                                                    *  *           |
        |    *                                               ****           |
  26500 |-+  **                                              *****        +-|
        |    **                                              **  *          |
        |   ***                                             **   *     *    |
  26000 |-+** *                                         *** **   *     *  +-|
        |  *  *                                         ****     *     *    |
        | **  *                                        **        **    *    |
        | *   *    *                                * * *        ***   **   |
  25500 |-*   *    **                             *****          ***   ** +-|
        | *   *    ** *                  *        ** **          **** ***   |
        |*    *  * ** *                 **      ** * **          ** * * **  |
        |*    *  **** *                 ***     **    *          *  * * *   |
  25000 |-+   *  *******            *   * *    * *                  * *   +-|
        |      **** ** *      *   **** *  *    *                    **      |
        |      ***  ** **     *   ******  **  **                    **      |
        |      ***  *   *    **   *  ***   *  *                     **      |
  24500 |-+    ***      *  *****  *   *    ** *                      *    +-|
        |      **       *  *** ****   *     ***                             |
        |       *       ** **  ***          **                              |
  24000 |-+     *       *****  ***                                        +-|
        |       *       ****     *                                          |
        |               * *                                                 |
        |     +     +   * *      +     +     +     +     +      +     +     |
  23500 +-------------------------------------------------------------------+
      01/01 02/01 03/01 04/01  05/01 06/01 07/01 08/01 09/01  10/01 11/01 12/01

What was the highest value of the Dow-Jones index, and when did it happen?

$ < dji.vnl vnl-sort -rgk Close |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume
2018-10-03 26833.47 26951.81 26789.08 26828.39 26828.39 280130000

Alrighty. Looks like the high was in October. Let’s zoom in on that month:

$ < dji.vnl vnl-sort -k Date |
    vnl-filter 'Date ~ /2018-10/' -p Date,Close |
    feedgnuplot --lines --unset grid --terminal 'dumb 80,30'
                --timefmt %Y-%m-%d --domain

  27000 +-------------------------------------------------------------------+
        |          ** +            +             +            +             |
        |        **  *                                                      |
        |       *     *                                                     |
  26500 |-+            *   ****                                           +-|
        |               ***    *                                            |
        |                      *                                            |
        |                       *                                           |
        |                       *                                           |
  26000 |-+                     *                                         +-|
        |                       *            **                             |
        |                        *           * *                            |
        |                        *          *   *                           |
  25500 |-+                      *          *   * ******                  +-|
        |                         *  *******     *      *                   |
        |                         * *      *             **                 |
        |                          **                      *              * |
  25000 |-+                        *                       *  *          *+-|
        |                                                  *  **        *   |
        |                                                   ** *        *   |
        |                                                   **  *****  *    |
        |                                                   *        * *    |
  24500 |-+                                                           *   +-|
        |                                                                   |
        |                                                                   |
        |             +            +             +            +             |
  24000 +-------------------------------------------------------------------+
      09/27         10/04        10/11         10/18        10/25         11/01

OK. Is this thing volatile? What was the largest single-day gain?

$ < dji.vnl | vnl-filter -p '.,d=diff(Close)' |
    head -n4 |
    vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume     d
2018-11-15 25061.48 25354.56 24787.79 25289.27 25289.27 383292840    -
2018-11-14 25388.08 25501.29 24935.82 25080.50 25080.50 384240000 -208.77
2018-11-13 25321.21 25511.03 25193.78 25286.49 25286.49 339690000  205.99


$ < dji.vnl | vnl-filter -p '.,d=diff(Close)' |
  vnl-sort -rgk d |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume     d
2018-02-02 26061.79 26061.79 25490.66 25520.96 25520.96 522880000 1175.21

Whoa. So the best single-gain day was 2018-02-02: the dow gained 1175.21 points between closing on Feb 1 and Feb 2. But it actually lost ground that day! What if I looked at the difference between the opening and closing in a single day?

< dji.vnl | vnl-filter -p '.,d=Close-Open' |
  vnl-sort -rgk d |
  head -n2 |
  vnl-align

#  Date      Open     High      Low     Close  AdjClose   Volume    d
2018-02-06 24085.17 24946.23 23778.74 24912.77 24912.77 823940000 827.6

I guess by that metric 2018-02-06 was better. Let’s join the Dow-jones index data and the TSLA data, and let’s look at them together:

$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  head -n4 |
  vnl-align

#  Date    Open_dji High_dji  Low_dji Close_dji AdjClose_dji Volume_dji Open_tsla High_tsla Low_tsla Close_tsla AdjClose_tsla Volume_tsla
2018-11-15 25061.48 25354.56 24787.79 25289.27  25289.27     383292840  342.33    348.58    339.04   348.44     348.44        4486339
2018-11-14 25388.08 25501.29 24935.82 25080.50  25080.50     384240000  342.70    347.11    337.15   344.00     344.00        5036300
2018-11-13 25321.21 25511.03 25193.78 25286.49  25286.49     339690000  333.16    344.70    332.20   338.73     338.73        5448600


$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  head -n4 |
  vnl-align

# Close_dji Close_tsla
25289.27    348.44
25080.50    344.00
25286.49    338.73


$ vnl-join --vnl-autosuffix dji.vnl tsla.vnl -j Date |
  vnl-filter -p '^Close' |
  feedgnuplot --domain --points --unset grid --terminal 'dumb 80,30'

  380 +---------------------------------------------------------------------+
      |         +         +         +         +         +         +         |
      |                             A           A                           |
      |                      A                                              |
  360 |-+                             A A         A                       +-|
      |                       A           A A A         A  AA               |
      |             A    A        AA    AAAA A           AA AA   A          |
  340 |-+         A  AA             AAA A A AAA  A  A   A           A     +-|
      |               A  A A       A  AAA AA   A    A           A           |
      |              A A    A A AAA     A A A                               |
      |                      A A AA     A  A                                |
  320 |-+                     A A AAAAAA         A AA    A                +-|
      |      A              A A   A AAA  AA                                 |
      |        A   AA  A A     A     A      A      A     A      AA   A      |
      |             A  AA AA   A     A         A A      A A                 |
  300 |A+     AA  A A  A     A  AA       A AAA         A        A AA A A  +-|
      |            A AA   AA A A  A     A                A A             A  |
      |         A    A   A  A  AA         A            AA  A                |
      |        AA A   AAAA    A     A                A AA    A       A      |
  280 |-+    A                AAAAA                 A  A                  +-|
      |                                           A                         |
      |          AA                                              A          |
  260 |-+                                AA AA        A          A        +-|
      |      A                             A    A                           |
      |  A                           A                            A         |
      |         +         +         +         +         +         +         |
  240 +---------------------------------------------------------------------+
    23500     24000     24500     25000     25500     26000     26500     27000

Huh. Apparently there’s no obvious, strong correlation between TSLA and Dow-Jones closing prices. And we saw that with just a few shell commands, without dropping down into a dedicated analysis system.

Description

Vnlog data is nicely readable by both humans and machines. Any time your application invokes printf() for either diagnostics or logging, consider writing out vnlog-formatted data. You retain human readability, but gain the power all the vnl-... tools provide.

Vnlog tools are designed to be very simple and light. There exist a number of other tools that do vaguely the same thing. Some of these:

Many of these provide facilities to run various analyses, and others focus on data types that aren’t just a table (json for instance). Vnlog by contrast doesn’t analyze anything, and targets the most trivial possible data format. This makes it very easy to run any analysis you like in any tool you like. The main envisioned use case is one-liners, and the tools are geared for that purpose. The above mentioned tools are much more powerful than vnlog, so they could be a better fit for some use cases. I claim that

  • 90% of the time you want to do simple things, and vnlog is a great fit for the task
  • If you really do need to do something complex, you really shouldn’t be in the shell writing oneliners anymore, and a fully-fledged analysis system (numpy, etc) is more appropriate

In the spirit of doing as little as possible, the provided tools are wrappers around tools you already have and are familiar with. The provided tools are:

  • vnl-filter is a tool to select a subset of the rows/columns in a vnlog and/or to manipulate the contents. This is an awk wrapper where the fields can be referenced by name instead of index. 20-second tutorial:
vnl-filter -p col1,col2,colx=col3+col4 'col5 > 10' --has col6

will read the input, and produce a vnlog with 3 columns: col1 and col2 from the input, and a column colx that’s the sum of col3 and col4 in the input. Only those rows for which both col5 > 10 is true and that have a non-null value for col6 will be output. A null entry is signified by a single - character.

vnl-filter --eval '{s += x} END {print s}'

will evaluate the given awk program on the input, but the column names work as you would hope they do: if the input has a column named x, this would produce the sum of all values in this column.

  • vnl-sort, vnl-uniq, vnl-join, vnl-tail, vnl-ts are wrappers around the corresponding commandline tools. These work exactly as you would expect also: the columns can be referenced by name, and the legend comment is handled properly. These are wrappers, so all the commandline options those tools have “just work” (except options that don’t make sense in the context of vnlog). As an example, vnl-tail -f will follow a log: data will be read by vnl-tail as it is written into the log (just like tail -f, but handling the legend properly). And you already know how to use these tools without even reading the manpages! Note: these were written for and have been tested with the Linux kernel and GNU Coreutils sort, join and tail. Other kernels and tools probably don’t (yet) work. Send me patches.
  • vnl-align aligns vnlog columns for easy interpretation by humans. The meaning is unaffected
  • Vnlog::Parser is a simple perl library to read a vnlog
  • vnlog is a simple python library to read a vnlog. Both python2 and python3 are supported
  • libvnlog is a C library to simplify writing a vnlog. Clearly all you really need is printf(), but this is useful if we have lots of columns, many containing null values in any given row, and/or if we have parallel threads writing to a log. In my usage I have hundreds of columns, so this is handy
  • vnl-make-matrix converts a one-point-per-line vnlog to a matrix of data. I.e.
$ cat dat.vnl
# i j x
0 0 1
0 1 2
0 2 3
1 0 4
1 1 5
1 2 6
2 0 7
2 1 8
2 2 9
3 0 10
3 1 11
3 2 12

$ < dat.vnl vnl-filter -p i,x | vnl-make-matrix --outdir /tmp
Writing to '/tmp/x.matrix'

$ cat /tmp/x.matrix
1 2 3
4 5 6
7 8 9
10 11 12

All the tools have manpages that contain more detail. And more tools will probably be added with time.

C interface

Basic usage

For most uses, these logfiles are simple enough to be generated with plain prints. But then each print statement has to know which numeric column we’re populating, which becomes effortful with many columns. In my usage it’s common to have a large parallelized C program that’s writing logs with hundreds of columns where any one record would contain only a subset of the columns. In such a case, it’s helpful to have a library that can output the log files. This is available. Basic usage looks like this:

In a shell:

$ vnl-gen-header 'int w' 'uint8_t x' 'char* y' 'double z' 'void* binary' > vnlog_fields_generated.h

In a C program test.c:

#include "vnlog_fields_generated.h"

int main()
{
    vnlog_emit_legend();

    vnlog_set_field_value__w(-10);
    vnlog_set_field_value__x(40);
    vnlog_set_field_value__y("asdf");
    vnlog_emit_record();

    vnlog_set_field_value__z(0.3);
    vnlog_set_field_value__x(50);
    vnlog_set_field_value__w(-20);
    vnlog_set_field_value__binary("\x01\x02\x03", 3);
    vnlog_emit_record();

    vnlog_set_field_value__w(-30);
    vnlog_set_field_value__x(10);
    vnlog_set_field_value__y("whoa");
    vnlog_set_field_value__z(0.5);
    vnlog_emit_record();

    return 0;
}

Then we build and run, and we get

$ cc -o test test.c -lvnlog

$ ./test

# w x y z binary
-10 40 asdf - -
-20 50 - 0.2999999999999999889 AQID
-30 10 whoa 0.5 -

The binary field in base64-encoded. This is a rarely-used feature, but sometimes you really need to log binary data for later processing, and this makes it possible.

So you

  1. Generate the header to define your columns
  2. Call vnlog_emit_legend()
  3. Call vnlog_set_field_value__...() for each field you want to set in that row.
  4. Call vnlog_emit_record() to write the row and to reset all fields for the next row. Any fields unset with a vnlog_set_field_value__...() call are written as null: -

This is enough for 99% of the use cases. Things get a bit more complex if we have have threading or if we have multiple vnlog ouput streams in the same program. For both of these we use vnlog contexts.

Contexts

To support independent writing into the same vnlog (possibly by multiple threads; this is reentrant), each log-writer should create a context, and use it when talking to vnlog. The context functions will make sure that the fields in each context are independent and that the output records won’t clobber each other:

void child_writer( // the parent context also writes to this vnlog. Pass NULL to
                   // use the global one
                   struct vnlog_context_t* ctx_parent )
{
    struct vnlog_context_t ctx;
    vnlog_init_child_ctx(&ctx, ctx_parent);

    while(records)
    {
        vnlog_set_field_value_ctx__xxx(&ctx, ...);
        vnlog_set_field_value_ctx__yyy(&ctx, ...);
        vnlog_set_field_value_ctx__zzz(&ctx, ...);
        vnlog_emit_record_ctx(&ctx);
    }

    vnlog_free_ctx(&ctx); // required only if we have any binary fields
}

If we want to have multiple independent vnlog writers to different streams (with different columns and legends), we do this instead:

file1.c:

#include "vnlog_fields_generated1.h"

void f(void)
{
    // Write some data out to the default context and default output (STDOUT)
    vnlog_emit_legend();
    ...
    vnlog_set_field_value__xxx(...);
    vnlog_set_field_value__yyy(...);
    ...
    vnlog_emit_record();
}

file2.c:

#include "vnlog_fields_generated2.h"

void g(void)
{
    // Make a new session context, send output to a different file, write
    // out legend, and send out the data
    struct vnlog_context_t ctx;
    vnlog_init_session_ctx(&ctx);
    FILE* fp = fopen(...);
    vnlog_set_output_FILE(&ctx, fp);
    vnlog_emit_legend_ctx(&ctx);
    ...
    vnlog_set_field_value__a(...);
    vnlog_set_field_value__b(...);
    ...
    vnlog_free_ctx(&ctx); // required only if we have any binary fields
    vnlog_emit_record();
}

Note that it’s the user’s responsibility to make sure the new sessions go to a different FILE by invoking vnlog_set_output_FILE(). Furthermore, note that the included vnlog_fields_....h file defines the fields we’re writing to; and if we have multiple different vnlog field definitions in the same program (as in this example), then the different writers must live in different source files. The compiler will barf if you try to #include two different vnlog_fields_....h files in the same source.

Remaining APIs

  • vnlog_printf(...) and vnlog_printf_ctx(ctx, ...) write to a pipe like

printf() does. This exists primarily for comments.

  • vnlog_clear_fields_ctx(ctx, do_free_binary) clears out the data in a context

and makes it ready to be used for the next record. It is rare for the user to have to call this manually. The most common case is handled automatically (clearing out a context after emitting a record). One area where this is useful is when making a copy of a context:

struct vnlog_context_t ctx1;
// .... do stuff with ctx1 ... add data to it ...

struct vnlog_context_t ctx2 = ctx1;
// ctx1 and ctx2 now both have the same data, and the same pointers to
// binary data. I need to get rid of the pointer references in ctx1

vnlog_clear_fields_ctx(&ctx1, false);
  • vnlog_free_ctx(ctx) frees memory for an vnlog context. Do this before

throwing the context away. Currently this is only needed for context that have binary fields, but this should be called for all contexts anyway, in case this changes in a later revision

Base64 interface

The C interface supports writing base64-encoded binary data using Chris Venter’s libb64. The base64-encoder used here was slightly modified: the output appears all on one line, making is suitable to appear in a vnlog field. If we’re writing a vnlog with printf() directly without using the vnlog.h interface described above, we allow this modified base64 encoder to be invoked by itself. Usage:

void* binary_buffer     = ...;
int   binary_buffer_len = ...;

char base64_buffer[vnlog_base64_dstlen_to_encode(binary_buffer_len)];
vnlog_base64_encode( base64_buffer, sizeof(base64_buffer),
                     binary_buffer, binary_buffer_len );

Clearly the above example allocates the base64 buffer on the stack, so it’s only suitable for small-ish data chunks. But if you have lots and lots of data, probably writing it as base64 into a vnlog isn’t the best thing to do.

numpy interface

The built-in numpy.loadtxt numpy.savetxt functions work well to read and write these files. For example to write to standard output a vnlog with fields a, b and c:

numpy.savetxt(sys.stdout, array, fmt="%g", header="a b c")

Note that numpy automatically adds the # to the header. To read a vnlog from a file on disk, do something like

array = numpy.loadtxt('data.vnl')

These functions know that # lines are comments, but don’t interpret anything as field headers. That’s easy to do, so I’m not providing any helper libraries. I might do that at some point, but in the meantime, patches are welcome.

Workflows and recipes

Storing disjoint data

A common use case is a complex application that produces several semi-related subsets of data at once. Example: a moving vehicle is reporting both its own position and the observed positions of other vehicles; at any given time any number of other vehicles may be observed. Two equivalent workflows are possible:

  • a single unified vnlog stream for all the data
  • several discrete vnlog streams for each data subset

Both are valid approaches

One unified vnlog stream

Here the application produces a single vnlog that contains all the columns, from all the data subsets. In any given row, many of the columns will be empty (i.e. contain only - ). For instance, a row describing a vehicle own position will not have data about any other vehicles, and vice versa. vnl-filter can be used to pull out the different subsets. Sample joint.vnl:

# time x_self x_other
1      10     -
2      20     -
2      -      100
3      30     -
3      -      200
3      -      300

Here we have 3 instances in time. We have no observations of other vehicles at time 1, one observation at time 2, and two observations at time 3. We can use vnl-filter to pull out the data we want. The + modifier in picked columns in vnl-filter is especially useful here:

$ < joint.vnl vnl-filter -p time,self

# time x_self
1 10
2 20
2 -
3 30
3 -
3 -


$ < joint.vnl vnl-filter -p time,+self

# time x_self
1 10
2 20
3 30


$ < joint.vnl vnl-filter -p time,+other

# time x_other
2 100
3 200
3 300

Several discrete vnlog streams

Conversely, the application can produce separate vnlog streams for each subset of data. Depending on what is desired, exactly, vnl-join can be used to re-join them:

$ cat self.vnl

# time x_self
1 10
2 20
3 30


$ cat other.vnl

# time x_other
2 100
3 200
3 300


$ vnl-join -j time -a- self.vnl other.vnl

# time x_self x_other
1 10 -
2 20 100
3 30 200
3 30 300

Data statistics

A common need is to compute basic statistics from your data. Many of the alternative toolkits listed above provide built-in facilities to do this, but vnlog does not: it’s meant to be unixy, where each tool has very limited scope. Thus you can either do this with awk like you would normally, or you can use other standalone tools to perform the needed computations. For instance, I can generate some data:

$ seq 2 100 | awk 'BEGIN {print "# x"} {print log($1)}' > /tmp/log.vnl

Then I can compute the mean with awk:

$ < /tmp/log.vnl vnl-filter --eval '{sum += x} END {print sum/NR}'
3.67414

Or I can compute the mean (and other stuff) with a separate standalone tool:

$ < /tmp/log.vnl ministat
x <stdin>
+----------------------------------------------------------------------------+
|                                                                         xx |
|                                                                  x xxxxxxx |
|                                                             xx xxxxxxxxxxxx|
|                                                x  x xxxxxxxxxxxxxxxxxxxxxxx|
|x       x    x    x  x  x  x x x xx xx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|
|                                         |_______________A____M___________| |
+----------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x  99      0.693147       4.60517       3.93183     3.6741353    0.85656382

ministat is not a part of the vnlog toolkit, but the vnlog format is generic so it works just fine.

Powershell-style filtering of common shell commands

Everything about vnlog is generic and simple, so it’s easy to use it to process data that wasn’t originally meant to be used this way. For instance filtering the output of ls -l to report only file names and sizes, skipping directories, and sorting by file sizes:

$ ls -l

total 320
-rw-r--r-- 1 dima dima  5044 Aug 25 15:04 Changes
-rw-r--r-- 1 dima dima 12749 Aug 25 15:04 Makefile
-rw-r--r-- 1 dima dima 69789 Aug 25 15:04 README.org
-rw-r--r-- 1 dima dima 33781 Aug 25 15:04 README.template.org
-rw-r--r-- 1 dima dima  5359 Aug 25 15:04 b64_cencode.c
drwxr-xr-x 4 dima dima  4096 Aug 25 15:04 completions
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 lib
drwxr-xr-x 3 dima dima  4096 Aug 25 15:04 packaging
drwxr-xr-x 2 dima dima  4096 Aug 25 15:04 test
-rwxr-xr-x 1 dima dima  5008 Aug 25 15:04 vnl-align
-rwxr-xr-x 1 dima dima 56637 Aug 25 15:04 vnl-filter
-rwxr-xr-x 1 dima dima  5678 Aug 25 15:04 vnl-gen-header
-rwxr-xr-x 1 dima dima 29815 Aug 25 15:04 vnl-join
-rwxr-xr-x 1 dima dima  3631 Aug 25 15:04 vnl-make-matrix
-rwxr-xr-x 1 dima dima  8372 Aug 25 15:04 vnl-sort
-rwxr-xr-x 1 dima dima  5822 Aug 25 15:04 vnl-tail
-rwxr-xr-x 1 dima dima  4439 Aug 25 15:04 vnl-ts
-rw-r--r-- 1 dima dima   559 Aug 25 15:04 vnlog-base64.h
-rw-r--r-- 1 dima dima  8169 Aug 25 15:04 vnlog.c
-rw-r--r-- 1 dima dima 12677 Aug 25 15:04 vnlog.h


$ (echo '# permissions num_links user group size month day time name';
   ls -l | tail -n +2) |
  vnl-filter 'permissions !~ "^d"' -p name,size |
  vnl-sort -gk size |
  vnl-align

#       name         size
vnlog-base64.h        559
vnl-make-matrix      3631
vnl-ts               4439
vnl-align            5008
Changes              5044
b64_cencode.c        5359
vnl-gen-header       5678
vnl-tail             5822
vnlog.c              8169
vnl-sort             8372
vnlog.h             12677
Makefile            12749
vnl-join            29815
README.template.org 33781
vnl-filter          56637
README.org          69789

With a bit of shell manipulation, these tools can be applied to a whole lot of different data streams that know nothing of vnlog.

Caveats and bugs

The tools that wrap standard commandline utilities (vnl-sort, vnl-uniq, vnl-join, vnl-tail, vnl-ts) are written specifically to work with the Linux kernel and the GNU coreutils. None of these have been tested with BSD tools or with non-Linux kernels, and I’m sure things don’t just work. It’s probably not too effortful to get that running, but somebody needs to at least bug me for that. Or better yet, send me nice patches :)

These tools are meant to be simple, so some things are hard requirements. A big one is that columns are whitespace-separated. There is no mechanism for escaping or quoting whitespace into a single field. I think supporting something like that is more trouble than it’s worth.

Installation

On Debian-based boxes

At this time vnlog is a part of Debian/buster and Ubuntu/cosmic (18.10). On those boxes you can simply

$ sudo apt install vnlog libvnlog-dev libvnlog-perl python-vnlog

to get the binary tools, the C API, the perl and python2 interfaces respectively.

On a Debian (or Ubuntu) machine that’s too old to have the packages already available, you can build and install them:

$ git clone git@github.com:dkogan/vnlog.git
$ cd vnlog
$ cp -r packaging/debian .
$ dpkg-buildpackage -us -uc -b
$ sudo dpkg -i ../vnlog*.deb ../libvnlog-dev*.deb ../libvnlog-perl*.deb ../python-vnlog*.deb

On non-Debian-based boxes

With the exception of the C API, every part of the toolkit is written in an interpreted language, and there’s nothing to “install”. You can run everything directly from the source tree:

$ git clone git@github.com:dkogan/vnlog.git
$ cd vnlog
$ ./vnl-filter .....

If you do want to install to some location, do this:

$ make
$ PREFIX=/usr/local make install

Manpages

vnl-filter

NAME
    vnl-filter - filters vnlogs to select particular rows, fields

SYNOPSIS
     $ cat run.vnl

     # time x   y   z   temperature
     3      1   2.3 4.8 30
     4      1.1 2.2 4.7 31
     6      1   2.0 4.0 35
     7      1   1.6 3.1 42


     $ <run.vnl vnl-filter -p x,y,z | vnl-align

     # x  y   z
     1   2.3 4.8
     1.1 2.2 4.7
     1   2.0 4.0
     1   1.6 3.1


     $ <run.vnl vnl-filter -p i=NR,time,'dist=sqrt(x*x + y*y + z*z)' | vnl-align

     # i time   dist
     1   3    5.41572
     2   4    5.30471
     3   6    4.58258
     4   7    3.62905


     $ <run.vnl vnl-filter 'temperature >= 35' | vnl-align

     # time x  y   z  temperature
     6      1 2.0 4.0 35
     7      1 1.6 3.1 42



     $ <run.vnl vnl-filter --eval '{s += temperature} END { print "mean temp: " s/NR}'

     mean temp: 34.5


     $ <run.vnl vnl-filter -p x,y | feedgnuplot --terminal 'dumb 80,30' --unset grid --domain --lines --exit

       2.3 +---------------------------------------------------------------------+
           |           +          +          ***************         +           |
           |                                                **************       |
           |                                                              *******|
       2.2 |-+                                                       ************|
           |                                                 ********            |
           |                                         ********                    |
       2.1 |-+                              *********                          +-|
           |                        ********                                     |
           |                ********                                             |
           |            ****                                                     |
         2 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *                                                         |
       1.9 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *                                                         |
       1.8 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
       1.7 |-+         *                                                       +-|
           |           *                                                         |
           |           *                                                         |
           |           *          +           +           +          +           |
       1.6 +---------------------------------------------------------------------+
          0.98         1         1.02        1.04        1.06       1.08        1.1

DESCRIPTION
    This tool is largely a frontend for awk to operate on vnlog files. Vnlog
    is both an input and an output. This tool makes it very simple to select
    specific rows and columns for output and to manipulate the data in
    various ways.

    This is a UNIX-style tool, so the input/output of this tool is strictly
    STDIN/STDOUT. Furthermore, in its usual form this tool is a filter, so
    the format of the output is *exactly* the same as the format of the
    input. The exception to this is when using "--eval", in which the output
    is dependent on whatever expression we're evaluating.

    This tool is convenient to process both stored data or live data; in the
    latter case, it's very useful to pipe the streaming output to
    "feedgnuplot --stream" to get a realtime visualization of the incoming
    data.

    This tool reads enough of the input file to get a legend, at which point
    it constructs an awk program to do the main work, and execs to awk (it's
    possible to use perl as well, but this isn't as fast).

  Input/output data format
    The input/output data is vnlog: a plain-text table of values. Any lines
    beginning with "#" are treated as comments, and are passed through. The
    first line that begins with "#" but not "##" or "#!" is a *legend* line.
    After the "#", follow whitespace-separated field names. Each subsequent
    line is whitespace-separated values matching this legend. For instance,
    this is a valid vnlog file:

     #!/usr/bin/something
     ## more comments
     # x y z
     -0.016107 0.004362 0.005369
     -0.017449 0.006711 0.006711
     -0.018456 0.014093 0.006711
     -0.017449 0.018791 0.006376

    "vnl-filter" uses this format for both the input and the output. The
    comments are preserved, but the legend is updated to reflect the fields
    in the output file.

    A string "-" is used to indicate an undefined value, so this is also a
    valid vnlog file:

     # x y z
     1 2 3
     4 - 6
     - - 7

  Filtering
    To select specific *columns*, pass their names to the "-p" option (short
    for "--print" or "--pick", which are synonyms). In its simplest form, to
    grab only columns "x" and "y", do

     vnl-filter -p x,y

    See the detailed description of "-p" below for more detail.

    To select specific *rows*, we use *matches* expressions. Anything on the
    "vnl-filter" commandline and not attached to any "--xxx" option is such
    an expression. For instance

     vnl-filter 'size > 10'

    would select only those rows whose "size" column contains a value > 10.
    See the detailed description of matches expressions below for more
    detail.

  Context lines
    "vnl-filter" supports the context output options ("-A", "-B" and "-C")
    exactly like the "grep" tool. I.e to print out all rows whose "size"
    column contains a value > 10 *but also* include the 3 rows immediately
    before *and* after such matching rows, do this:

     vnl-filter -C3 'size > 10'

    "-B" reports the rows *before* matching ones and "-A" the rows *after*
    matching ones. "-C" reports both. Note that this applies *only* to
    *matches* expressions: records skipped because they fail "--has" or
    "--skipempty" are *not* included in contextual output.

  Backend choice
    By default, the parsing of arguments and the legend happens in perl,
    which then constructs a simple awk script, and invokes "mawk" to
    actually read the data and to process it. This is done because awk is
    lighter weight and runs faster, which is important because our data sets
    could be quite large. We default to "mawk" specifically, since this is a
    simpler implementation than "gawk", and runs much faster. If for
    whatever reason we want to do everything with perl, this can be
    requested with the "--perl" option.

  Special functions
    For convenience we support several special functions in any expression
    passed on to awk or perl (named expressions, matches expressions,
    "--eval" strings). These generally maintain some internal state, and
    vnl-filter makes sure that this state is consistent. Note that these are
    evaluated *after* "--skipcomments" and "--has". So any record skipped
    because of a "--has" expression, for instance, will *not* be considered
    in "prev()", "diff()" and so on.

    *   rel(x) returns value of "x" relative to the first value of "x". For
        instance we might want to see the time or position relative to the
        start, not relative to some absolute beginning. Example:

         $ cat tst.vnl

         # time x
         100    200
         101    212
         102    209


         $ <tst.vnl vnl-filter -p 't=rel(time),x=rel(x)

         # t x
         0 0
         1 12
         2 9

    *   diff(x) returns the difference between the current value of "x" and
        the previous value of "x". The first row will always be -. Example:

         $ <tst.vnl vnl-filter -p x,'d1=diff(x),d2=diff(diff(x))' | vnl-align

         # x d1 d2
           1  -  -
           8  7  7
          27 19 12
          64 37 18
         125 61 24

    *   sum(x) returns the cumulative sum of "x". As diff(x) can be thought
        of as a derivative, sum(x) can be thought of as an integral. So
        "diff(sum(x))" would return the same value as "x" (except for the
        first row; "diff()" always returns - for the first row).

        Example:

         $ <tst.vnl vnl-filter -p 'x,s=sum(x),ds=diff(sum(x))' | vnl-align

         # x  s   ds
           1   1   -
           8   9   8
          27  36  27
          64 100  64
         125 225 125

    *   prev(x) returns the previous value of "x". One could construct
        "sum()" and "rel()" using this, if they weren't already available.

ARGUMENTS
  -p|--print|--pick expr
    These option provide the mechanism to select specific columns for
    output. For instance to pull out columns called "lat", "lon", and any
    column whose name contains the string "feature_", do

     vnl-filter -p lat,lon,'feature_.*'

    or, equivalently

     vnl-filter --print lat --print lon --print 'feature_.*'

    We look for exact column name matches first, and if none are found, we
    try a regex. If there was no column called exactly "feature_", then the
    above would be equivalent to

     vnl-filter -p lat,lon,feature_

    This mechanism is much more powerful than just selecting columns. First
    off, we can rename chosen fields:

     vnl-filter -p w=feature_width

    would pick the "feature_width" field, but the resulting column in the
    output would be named "w". When renaming a column in this way regexen
    are *not* supported, and exact field names must be given. But the string
    to the right of the "=" is passed on directly to awk (after replacing
    field names with column indices), so any awk expression can be used
    here. For instance to compute the length of a vector in separate columns
    "x", "y", and "z" you can do:

     vnl-filter -p 'l=sqrt(x*x + y*y + z*z)'

    A single column called "l" would be produced.

    We can also *exclude* columns by preceding their name with "!". This
    works like you expect. Rules:

    *   The pick/exclude directives are processed in order given to produce
        the output picked-column list

    *   If the first "-p" item is an exclusion, we implicitly pick *all* the
        columns prior to processing the "-p".

    *   The exclusion expressions match the *output* column names, not the
        *input* names.

    *   We match the exact column names first. If that fails, we match as a
        regex

    Example. To grab all the columns *except* the temperature(s) do this:

     vnl-filter -p !temperature

    To grab all the columns that describe *something* about a robot (columns
    whose names have the string "robot_" in them), but *not* its temperature
    (i.e. *not* "robot_temperature"), do this:

     vnl-filter -p robot_,!temperature

  --has a,b,c,...
    Used to select records (rows) that have a non-empty value in a
    particular field (column). A *null* value in a column is designated with
    a single "-". If we want to select only records that have a value in the
    "x" column, we pass "--has x". To select records that have data for
    *all* of a given set of columns, the "--has" option can be repeated, or
    these multiple columns can be given in a whitespace-less comma-separated
    list. For instance if we want only records that have data in *both*
    columns "x" *and* "y" we can pass in "--has x,y" or "--has x --has y".
    If we want to combine multiple columns in an *or* (select rows that have
    data in *any* of a given set of columns), use a matches expression, as
    documented below.

    If we want to select a column *and* pick only rows that have a value in
    this column, a shorthand syntax exists:

     vnl-filter --has col -p col

    is equivalent to

     vnl-filter -p +col

    Note that just like the column specifications in "-p" the columns given
    to "--has" must match exactly *or* as a regex. In either case, a unique
    matching column must be found.

  Matches expressions
    Anything on the commandline not attached to any "--xxx" option is a
    *matches* expression. These are used to select particular records (rows)
    in a data file. For each row, we evaluate all the expressions. If *all*
    the expressions evaluate to true, that row is output. This expression is
    passed directly to the awk (or perl) backend.

    Example: to select all rows that have valid data in column "a" *or*
    column "b" *or* column "c" you can

     vnl-filter 'a != "-" || b != "-" || c != "-"'

    or

     vnl-filter --perl 'defined a || defined b || defined c'

    As with the named expressions given to "-p" (described above), these are
    passed directly to awk, so anything that can be done with awk is
    supported here.

  -A N|--after-context N
    Output N lines following each *matches* expression, even those lines
    that do not themselves match. This works just like the "grep" options of
    the same name. See "Context lines"

  -B N|--before-context N
    Output N lines preceding each *matches* expression, even those lines
    that do not themselves match. This works just like the "grep" options of
    the same name. See "Context lines"

  -C N|--context N
    Output N lines preceding and following each *matches* expression, even
    those lines that do not themselves match. This works just like the
    "grep" options of the same name. See "Context lines"

  --eval expr
    Instead of printing out all matching records and picked columns, just
    run the given chunk of awk (or perl). In this mode of operation,
    "vnl-filter" acts just like a glorified awk, that allows fields to be
    accessed by name instead of by number, as it would be in raw awk.

    Since the expression may print *anything* or nothing at all, the output
    in this mode is not necessarily itself a valid vnlog stream. And no
    column-selecting arguments should be given, since they make no sense in
    this mode.

    In awk the expr is a full set of pattern/action statements. So to print
    the sum of columns "a" and "b" in each row, and at the end, print the
    sum of all values in the "a" column

     vnl-filter --eval '{print a+b; suma += a} END {print suma}'

    In perl the arbitrary expression fits in like this:

     while(<>) # read each line
     {
       next unless matches; # skip non-matching lines
       eval expression;     # evaluate the arbitrary expression
     }

  --function|--sub
    Evaluates the given expression as a function that can be used in other
    expressions. This is most useful when you want to print something that
    can't trivially be written as a simple expression. For instance:

     $ cat tst.vnl
     # s
     1-2
     3-4
     5-6

     $ < tst.vnl
       vnl-filter --function 'before(x) { sub("-.*","",x); return x }' \
                  --function 'after(x)  { sub(".*-","",x); return x }' \
                  -p 'b=before(s),a=after(s)'
     # b a
     1 2
     3 4
     5 6

    See the CAVEATS section below if you're doing something
    sufficiently-complicated where you need this.

  --[no]skipempty
    Do [not] skip records where all fields are blank. By default we *do*
    skip all empty records; to include them, pass "--noskipempty"

  --skipcomments
    Don't output non-legend comments

  --perl
    By default all procesing is performed by "mawk", but if for whatever
    reason we want perl instead, pass "--perl". Both modes work, but "mawk"
    is noticeably faster. "--perl" could be useful because it is more
    powerful, which could be important since a number of things pass
    commandline strings directly to the underlying language (named
    expressions, matches expressions, "--eval" strings). Note that while
    variables in perl use sigils, column references should *not* use sigils.
    To print the sum of all values in column "a" you'd do this in awk

     vnl-filter --eval '{suma += a} END {print suma}'

    and this in perl

     vnl-filter --perl --eval '{$suma += a} END {say $suma}'

    The perl strings are evaluated without "use strict" or "use warnings" so
    I didn't have to declare $suma in the example.

  --dumpexprs
    Used for debugging. This spits out all the final awk (or perl) program
    we run for the given commandline options and given input. This is the
    final program, with the column references resolved to numeric indices,
    so one can figure out what went wrong.

  --unbuffered
    Flushes each line after each print. This makes sure each line is output
    as soon as it is available, which is crucial for realtime output and
    streaming plots.

  --stream
    Synonym for "--unbuffered"

CAVEATS
    This tool is very lax in its input validation (on purpose). As a result,
    columns with names like %CPU and "TIME+" do work (i.e. you can more or
    less feed in output from "top -b"). The downside is that shooting
    yourself in the foot is possible. This tradeoff is currently set to work
    well for my use cases, but I'd be interested in hearing other people's
    experiences. Potential pitfalls/unexpected behaviors:

    *   When substituting column names I match *either* a word-nonword
        transition ("\b") *or* a whitespace-nonword transition. The word
        boundaries is what would be used 99% of the time. But the keys may
        have special characters in them, which don't work with "\b". This
        means that whitespace becomes important: "1+%CPU" will not be parsed
        as expected, which is correct since "+%CPU" is also a valid field
        name. But "1+ %CPU" will be parsed correctly, so if you have weird
        field names, put the whitespace into your expressions. It'll make
        them more readable anyway.

    *   Strings passed to "-p" are split on "," *except* if the "," is
        inside balanced "()". This makes it possible to say things like
        "vnl-filter --function 'f(a,b) { ... }' -p 'c=f(a,b)'". This is
        probably the right behavior, although some questionable looking
        field names become potentially impossible: "f(a" and "b)" *could*
        otherwise be legal field names, but you're probably asking for
        trouble if you do that.

    *   All column names are replaced in all eval strings without regard to
        context. The earlier example that reports the sum of values in a
        column: "vnl-filter --eval '{suma += a} END {print suma}'" will work
        fine if we *do* have a column named "a" and do *not* have a column
        named "suma". But will not do the right thing if any of those are
        violated. It's the user's responsibility to make sure we're talking
        about the right columns. The focus here was one-liners so hopefully
        nobody has so many columns, they can't keep track of all of them in
        their head. I don't see any way to resolve this without seriously
        impacting the scope of the tool, so I'm leaving this alone. Comments
        welcome.

    *   Currently there're two modes: a pick/print mode and an "--eval"
        mode. Then there's also "--function", which adds bits of "--eval" to
        the pick/print mode, but it feels maybe insufficient. I don't yet
        have strong feelings about what this should become. Comments welcome


vnl-align

NAME
    vnl-align - aligns vnlog columns for easy interpretation by humans

SYNOPSIS
     $ cat tst.vnl

     # w x y z
     -10 40 asdf -
     -20 50 - 0.300000
     -30 10 whoa 0.500000


     $ vnl-align tst.vnl

     # w  x   y      z
     -10 40 asdf -
     -20 50 -    0.300000
     -30 10 whoa 0.500000

DESCRIPTION
    The basic usage is

     vnl-align logfile

    The arguments are assumed to be the vnlog files. If no arguments are
    given, the input comes from STDIN.

    This is very similar to "column -t", but handles "#" lines properly:

    1. The first "#" line is the legend. For the purposes of alignment, the
    leading "#" character and the first column label are treated as one
    column

    2. All other "#" lines are output verbatim.


vnl-sort

NAME
    vnl-sort - sorts an vnlog file, preserving the legend

SYNOPSIS
     $ cat a.vnl
     # a b
     AA 11
     bb 12
     CC 13
     dd 14
     dd 123

     Sort lexically by a:
     $ <a.vnl vnl-sort -k a
     # a b
     AA 11
     CC 13
     bb 12
     dd 123
     dd 14

     Sort lexically by a, ignoring case:
     $ <a.vnl vnl-sort -k a --ignore-case
     # a b
     AA 11
     bb 12
     CC 13
     dd 123
     dd 14

     Sort lexically by a, then numerically by b:
     $ <a.vnl vnl-sort -k a -k b.n
     # a b
     AA 11
     CC 13
     bb 12
     dd 14
     dd 123

     Sort lexically by a, then numerically by b in reverse:
     $ <a.vnl vnl-sort -k a -k b.nr
     # a b
     AA 11
     CC 13
     bb 12
     dd 123
     dd 14


     Sort by month and then day:
     $ cat dat.vnl
     # month day
     March 5
     Jan 2
     Feb 1
     March 30
     Jan 21

     $ <dat.vnl vnl-sort -k month.M -k day.n
     # month day
     Jan 2
     Jan 21
     Feb 1
     March 5
     March 30

DESCRIPTION
      Usage: vnl-sort [options] logfile logfile logfile ... < logfile

    This tool sorts given vnlog files in various ways. "vnl-sort" is a
    wrapper around the GNU coreutils "sort" tool. Since this is a wrapper,
    most commandline options and behaviors of the "sort" tool are present;
    consult the sort(1) manpage for detail. The differences from GNU
    coreutils "sort" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The columns are referenced by name, not index. So instead of saying

          sort -k1

        to sort by the first column, you say

          sort -k time

        to sort by column "time".

    *   The fancy "KEYDEF" spec from "sort" is only partially supported. I
        only allow us to sort by full *fields*, so the start/stop positions
        don't make sense. I *do* support the "OPTS" to change the type of
        sorting in a given particular column. For instance, to sort by month
        and then by day, do this (see example above):

          vnl-sort -k month.M -k day.n

    *   "--files0-from" is not supported due to lack of time. If somebody
        really needs it, talk to me.

    *   "--output" is not supported due to an uninteresting technical
        limitation. The output always goes to standard out.

    *   "--field-separator" is not supported because vnlog assumes
        whitespace-separated fields

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    Past that, everything "sort" does is supported, so see that man page for
    detailed documentation. Note that all non-legend comments are stripped
    out, since it's not obvious where they should end up.

BUGS
    This and the other "vnl-xxx" tools that wrap coreutils are written
    specifically to work with the Linux kernel and the GNU coreutils. None
    of these have been tested with BSD tools or with non-Linux kernels, and
    I'm sure things don't just work. It's probably not too effortful to get
    that running, but somebody needs to at least bug me for that. Or better
    yet, send me nice patches :)

SEE ALSO
    sort(1)


vnl-uniq

NAME
    vnl-uniq - uniq a log file, preserving the legend

SYNOPSIS
     $ cat colors.vnl
     # color
     blue
     yellow
     yellow
     blue
     yellow
     orange
     orange

     $ < colors.vnl | vnl-sort | vnl-uniq -c
     # count color
           2 blue
           2 orange
           3 yellow

DESCRIPTION
      Usage: vnl-uniq [options] < logfile

    This tool runs "uniq" on a given vnlog dataset. "vnl-uniq" is a wrapper
    around the GNU coreutils "uniq" tool. Since this is a wrapper, most
    commandline options and behaviors of the "uniq" tool are present;
    consult the uniq(1) manpage for detail. The differences from GNU
    coreutils "uniq" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   One *one* input is supported (a file on the cmdline or data on
        standard input), and the output *always* goes to standard output.
        Specifying the output as a file on the commandline is not supported.

    *   "--vnl-count NAME" can be given to name the "count" column. "-c" is
        still supported to add the default new column named "count", but if
        another name is wanted, "--vnl-count" does that. "--vnl-count"
        implies "-c"

    *   In addition to the normal behavior of skipping fields at the start,
        "-f" and "--skip-fields" can take a negative argument to skip the
        *all but the last* N fields. For instance, to use only the one last
        field, pass "-f -1" or "--skip-fields=-1".

    Past that, everything "uniq" does is supported, so see that man page for
    detailed documentation.

BUGS
    This and the other "vnl-xxx" tools that wrap coreutils are written
    specifically to work with the Linux kernel and the GNU coreutils. None
    of these have been tested with BSD tools or with non-Linux kernels, and
    I'm sure things don't just work. It's probably not too effortful to get
    that running, but somebody needs to at least bug me for that. Or better
    yet, send me nice patches :)

SEE ALSO
    uniq(1)


vnl-join

NAME
    vnl-join - joins two log files on a particular field

SYNOPSIS
     $ cat a.vnl
     # a b
     AA 11
     bb 12
     CC 13
     dd 14
     dd 123

     $ cat b.vnl
     # a c
     aa 1
     cc 3
     bb 4
     ee 5
     - 23

     Try to join unsorted data on field 'a':
     $ vnl-join -j a a.vnl b.vnl
     # a b c
     join: /dev/fd/5:3: is not sorted: CC 13
     join: /dev/fd/6:3: is not sorted: bb 4

     Sort the data, and join on 'a':
     $ vnl-join --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a  b c
     bb  12 4

     Sort the data, and join on 'a', ignoring case:
     $ vnl-join -i --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a b c
     AA 11 1
     bb 12 4
     CC 13 3

     Sort the data, and join on 'a'. Also print the unmatched lines from both files:
     $ vnl-join -a1 -a2 --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # a  b   c
     -   -   23
     AA   11 -
     CC   13 -
     aa  -    1
     bb   12  4
     cc  -    3
     dd  123 -
     dd   14 -
     ee  -    5

     Sort the data, and join on 'a'. Print the unmatched lines from both files,
     Output ONLY column 'c' from the 2nd input:
     $ vnl-join -a1 -a2 -o 2.c --vnl-sort - -j a a.vnl b.vnl | vnl-align
     # c
     23
     -
     -
      1
      4
      3
     -
     -
      5

DESCRIPTION
      Usage: vnl-join [join options]
                      [--vnl-sort -|[sdfgiMhnRrV]+]
                      [ --vnl-[pre|suf]fix[1|2] xxx    |
                        --vnl-[pre|suf]fix xxx,yyy,zzz |
                        --vnl-autoprefix               |
                        --vnl-autosuffix ]
                      logfile1 logfile2

    This tool joins two vnlog files on a given field. "vnl-join" is a
    wrapper around the GNU coreutils "join" tool. Since this is a wrapper,
    most commandline options and behaviors of the "join" tool are present;
    consult the join(1) manpage for detail. The differences from GNU
    coreutils "join" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The columns are referenced by name, not index. So instead of saying

          join -j1

        to join on the first column, you say

          join -j time

        to join on column "time".

    *   -1 and -2 are supported, but *must* refer to the same field. Since
        vnlog knows the identify of each field, it makes no sense for -1 and
        -2 to be different. So pass "-j" instead, it makes more sense in
        this context.

    *   "-a-" is available as a shorthand for "-a1 -a2": this is a full
        outer join, printing unmatched records from both of the inputs.
        Similarly, "-v-" is available as a shorthand for "-v1 -v2": this
        will output *only* the unique records in both of the inputs.

    *   "vnl-join"-specific options are available to adjust the field-naming
        in the output:

          --vnl-prefix1
          --vnl-suffix1
          --vnl-prefix2
          --vnl-suffix2
          --vnl-prefix
          --vnl-suffix
          --vnl-autoprefix
          --vnl-autosuffix

        See "Field names in the output" below for details.

    *   A "vnl-join"-specific option "--vnl-sort" is available to sort the
        input and/or output. See below for details.

    *   If no "-o" is given, we output the join field, the remaining fields
        in logfile1, the remaining fields in logfile2, .... This is what "-o
        auto" does, except we also handle empty vnlogs correctly.

    *   "-e" is not supported because vnlog uses "-" to represent undefined
        fields.

    *   "--header" is not supported because vnlog assumes a specific header
        structure, and "vnl-join" makes sure that this header is handled
        properly

    *   "-t" is not supported because vnlog assumes whitespace-separated
        fields

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    *   Rather than only 2-way joins, this tool supports N-way joins for any
        N > 2. See below for details.

    Past that, everything "join" does is supported, so see that man page for
    detailed documentation. Note that all non-legend comments are stripped
    out, since it's not obvious where they should end up.

  Field names in the output
    By default, the field names in the output match those in the input. This
    is what you want most of the time. It is possible, however that a column
    name adjustment is needed. One common use case for this is if the files
    being joined have identically-named columns, which would produce
    duplicate columns in the output. Example: we fixed a bug in a program,
    and want to compare the results before and after the fix. The program
    produces an x-y trajectory as a function of time, so both the bugged and
    the bug-fixed programs produce a vnlog with a legend

     # time x y

    Joining this on "time" will produce a vnlog with a legend

     # time x y x y

    which is confusing, and *not* what you want. Instead, we invoke
    "vnl-join" as

     vnl-join --vnl-suffix1 _buggy --vnl-suffix2 _fixed -j time buggy.vnl fixed.vnl

    And in the output we get a legend

     # time x_buggy y_buggy x_fixed y_fixed

    Much better.

    Note that "vnl-join" provides several ways of specifying this. The above
    works *only* for 2-way joins. An alternate syntax is available for N-way
    joins, a comma-separated list. The same could be expressed like this:

     vnl-join -a- --vnl-suffix _buggy,_fixed -j time buggy.vnl fixed.vnl

    Finally, if passing in structured filenames, "vnl-join" can infer the
    desired syntax from the filenames. The same as above could be expressed
    even simpler:

     vnl-join --vnl-autosuffix -j time buggy.vnl fixed.vnl

    This works by looking at the set of passed in filenames, and stripping
    out the common leading and trailing strings.

  Sorting of input and output
    The GNU coreutils "join" tool expects sorted columns because it can then
    take only a single pass through the data. If the input isn't sorted,
    then we can use normal shell substitutions to sort it:

     $ vnl-join -j key <(vnl-sort -s -k key a.vnl) <(vnl-sort -s -k key b.vnl)

    For convenience "vnl-join" provides a "--vnl-sort" option. This allows
    the above to be equivalently expressed as

     $ vnl-join -j key --vnl-sort - a.vnl b.vnl

    The "-" after the "--vnl-sort" indicates that we want to sort the
    *input* only. If we also want to sort the output, pass the short codes
    "sort" accepts instead of the "-". For instance, to sort the input for
    "join" and to sort the output numerically, in reverse, do this:

     $ vnl-join -j key --vnl-sort rg a.vnl b.vnl

    The reason this shorthand exists is to work around a quirk of "join".
    The sort order is *assumed* by "join" to be lexicographical, without any
    way to change this. For "sort", this is the default sort order, but
    "sort" has many options to change the sort order, options which are
    sorely missing from "join". A real-world example affected by this is the
    joining of numerical data. If you have "a.vnl":

     # time a
     8 a
     9 b
     10 c

    and "b.vnl":

     # time b
     9  d
     10 e

    Then you cannot use "vnl-join" directly to join the data on time:

     $ vnl-join -j time a.vnl b.vnl
     # time a b
     join: /dev/fd/4:3: is not sorted: 10 c
     join: /dev/fd/5:2: is not sorted: 10 e
     9 b d
     10 c e

    Instead you must re-sort both files lexicographically, *and* then
    (because you almost certainly want to) sort it back into numerical
    order:

     $ vnl-join -j time <(vnl-sort -s -k time a.vnl) <(vnl-sort -s -k time b.vnl) |
       vnl-sort -s -n -k time
     # time a b
     9 b d
     10 c e

    Yuck. The shorthand described earlier makes the interface part of this
    palatable:

     $ vnl-join -j time --vnl-sort n a.vnl b.vnl
     # time a b
     9 b d
     10 c e

    Note that the input sort is stable: "vnl-join" will invoke "vnl-sort
    -s". If you want a stable post-sort, you need to ask for it with
    "--vnl-sort s...".

  N-way joins
    The GNU coreutils "join" tool is inherently designed to join *exactly*
    two files. "vnl-join" extends this capability by chaining together a
    number of "join" invocations to produce a generic N-way join. This works
    exactly how you would expect with the following caveats:

    *   Full outer joins are supported by passing "-a-", but no other "-a"
        option is supported. This is possible, but wasn't obviously worth
        the trouble.

    *   "-v" is not supported. Again, this is possible, but wasn't obviously
        worth the trouble.

    *   Similarly, "-o" is not supported. This is possible, but wasn't
        obviously worth the trouble, especially since the desired behavior
        can be obtained by post-processing with "vnl-filter".

BUGS
    This and the other "vnl-xxx" tools that wrap coreutils are written
    specifically to work with the Linux kernel and the GNU coreutils. None
    of these have been tested with BSD tools or with non-Linux kernels, and
    I'm sure things don't just work. It's probably not too effortful to get
    that running, but somebody needs to at least bug me for that. Or better
    yet, send me nice patches :)

SEE ALSO
    join(1)


vnl-tail

NAME
    vnl-tail - tail a log file, preserving the legend

SYNOPSIS
     $ read_temperature | tee temp.vnl
     # temperature
     29.5
     30.4
     28.3
     22.1
     ... continually produces data

     ... at the same time, in another terminal
     $ vnl-tail -f temp.vnl
     # temperature
     28.3
     22.1
     ... outputs data as it comes in

DESCRIPTION
      Usage: vnl-tail [options] logfile logfile logfile ... < logfile

    This tool runs "tail" on given vnlog files in various ways. "vnl-tail"
    is a wrapper around the GNU coreutils "tail" tool. Since this is a
    wrapper, most commandline options and behaviors of the "tail" tool are
    present; consult the tail(1) manpage for detail. The differences from
    GNU coreutils "tail" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   "-c" is not supported because vnlog really doesn't want to break up
        lines

    *   "--zero-terminated" is not supported because vnlog assumes
        newline-separated records

    Past that, everything "tail" does is supported, so see that man page for
    detailed documentation.

BUGS
    This and the other "vnl-xxx" tools that wrap coreutils are written
    specifically to work with the Linux kernel and the GNU coreutils. None
    of these have been tested with BSD tools or with non-Linux kernels, and
    I'm sure things don't just work. It's probably not too effortful to get
    that running, but somebody needs to at least bug me for that. Or better
    yet, send me nice patches :)

SEE ALSO
    tail(1)


vnl-ts

NAME
    vnl-ts - add a timestamp to a vnlog stream

SYNOPSIS
     $ read_temperature
     # temperature
     29.5
     30.4
     28.3
     22.1
     ... continually produces data at 1Hz

     $ read_temperature | vnl-ts -s %.s
     # time-rel temperature
     0.013893 30.2
     1.048695 28.6
     2.105592 29.3
     3.162873 22.0
     ...

DESCRIPTION
      Usage: vnl-ts [-i | -s] [-m] [--vnl-field t] format < pipe

    This tool runs "ts" on given vnlog streams. "vnl-ts" is a wrapper around
    the "ts" tool from Joey Hess's moreutils
    <https://joeyh.name/code/moreutils/> toolkit. Since this is a wrapper,
    most commandline options and behaviors of the "ts" tool are present;
    consult the ts(1) manpage for details. The differences from "ts" are

    *   The input and output to this tool are vnlog files, complete with a
        legend

    *   The format *must* be passed-in by the user; no default is assumed.

    *   The given format *must not* contain whitespace, so that it fits a
        single vnlog field.

    *   "-r" is not supported: it assumes input timestamps with whitespace,
        which is incompatible with vnlog

    *   A "vnl-ts"-specific option "--vnl-field" is available to set the
        name of the new field. If omitted, a reasonable default will be
        used.

    Past that, everything "ts" does is supported, so see that man page for
    detailed documentation.

BUGS
    This and the other "vnl-xxx" tools that wrap standard utilities are
    written specifically to work with the Linux kernel. None of these have
    been tested with BSD tools or with non-Linux kernels, and I'm sure
    things don't just work. It's probably not too effortful to get that
    running, but somebody needs to at least bug me for that. Or better yet,
    send me nice patches :)

SEE ALSO
    ts(1)


Repository

https://github.com/dkogan/vnlog/

Authors

Dima Kogan (dima@secretsauce.net) wrote this toolkit for his work at the Jet Propulsion Laboratory, and is delighted to have been able to release it publically

Chris Venter (chris.venter@gmail.com) wrote the base64 encoder

License and copyright

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

Copyright 2016-2017 California Institute of Technology

Copyright 2017-2018 Dima Kogan (dima@secretsauce.net)

b64_cencode.c comes from cencode.c in the libb64 project. It is written by Chris Venter (chris.venter@gmail.com) who placed it in the public domain. The full text of the license is in that file.

You can’t perform that action at this time.