Filter programs

[passcod]

This is a persistent thread to contribute --filter-prog jaq programs.

(Upvote those you find useful, and they may be made into native features!)

How to use

Copy into a file e.g. filter-name.jq, then invoke with watchexec using -j=@filter-name.jq.

How to test or debug

Sprinkling log("info")s around and enabling -vv level logging can be very helpful.

Syntax and program errors will be output at error (-v) level.

Documentation

Programs are given an event and must return a boolean.

Programs are run in order, after all other filters, and short-circuit: if a filter (jaq or not) rejects an event, execution stops there, and no other filters are run. Additionally, they stop after outputting the first value, so you'll want to use any or all when iterating, otherwise only the first item will be processed, which can be quite confusing!

Note that jaq only supports the any(expression; predicate) syntax, not the shorthand any(expression that returns booleans).

All filtering programs, and especially those using filesystem access, are much slower than the other filtering methods. If filtering is too slow, events will back up and stall watchexec. Take care when designing your filters. That said, for most uses there's plenty of time budget to go! Don't be too cautious, have fun :)

Watchexec filters

In addition to the jaq stdlib, watchexec adds some custom filter definitions:

• path | file_meta returns file metadata or null if the file does not exist.
• path | file_size returns the size of the file at path, or null if it does not exist.
• path | file_read(bytes) returns a string with the first n bytes of the file at path. If the file is smaller than n bytes, the whole file is returned. There is no filter to read the whole file at once to encourage limiting the amount of data read and processed.
• string | hash, and path | file_hash return the hash of the string or file at path. No guarantee is made about the algorithm used: treat it as an opaque value.
• any | kv_store(key), kv_fetch(key), and kv_clear provide a simple key-value store. Data is kept in memory only, there is no persistence. Consistency is not guaranteed.
• any | printout, any | printerr, and any | log(level) will print or log any given value to stdout, stderr, or the log (levels = "error", "warn", "info", "debug", "trace"), and pass the value through (so [1] | log("debug") | .[] will produce a 1 and log [1]).

Event object

  {
    "tags": [
      {
        "kind": "path",
        "absolute": "/home/user/your/new-folder",
        "filetype": "dir"
      },
      {
        "kind": "fs",
        "simple": "create",
        "full": "Create(Folder)"
      },
      {
        "kind": "source",
        "source": "filesystem",
      }
    ],
    "metadata": {
      "notify-backend": "inotify"
    }
  }

• tags, structured event data.
• tags[].kind, which can be:
  • path, along with:
    • absolute, an absolute path.
    • filetype, a file type if known (dir, file, symlink, other).
  • fs:
    • simple, the "simple" event type (access, create, modify, remove, or other).
    • full, the "full" event type, which is too complex to fully describe here, but looks like General(Precise(Specific)).
  • source, along with:
    • source, the source of the event (filesystem, keyboard, mouse, os, time, internal).
  • keyboard, along with:
    • keycode. Currently only the value eof is supported.
  • process, for events caused by processes:
    • pid, the process ID.
  • signal, for signals sent to Watchexec:
    • signal, the normalised signal name (hangup, interrupt, quit, terminate, user1, user2).
  • completion, for when a command ends:
    • disposition, the exit disposition (success, error, signal, stop, exception, continued).
    • code, the exit, signal, stop, or exception code.
• metadata, additional information about the event.

[passcod]

Check actual file contents

Only passes events that touch files if either:

• it's the first event we receive for a file, or
• the stored hash of any affected file has changed.

def has_changed:
	. as $file |
	file_hash |
	{ prev: kv_fetch($file), cur: (. | kv_store($file)) } |
	.prev != .cur
;

if any(.tags[]; .kind == "path") then
	any(
		.tags[] | select((.kind == "path") and (.filetype == "file"));
		.absolute | has_changed
	)
else true end

[unphased]

I'm new to jaq, somewhat familiar with jq (though not proficient at all). How would you implement logic to prevent hashing from taking too long for files that are too large? I would imagine that rather than giving up, dealing with checking large files could at least fall back to something like testing filesize, then testing first 1MB of content, then testing last 1MB of content, etc.