watch directories

[anarcat]

One thing I'm missing in my workflow is a "fire and forget" pattern. Let me explain: what I'm doing right now is I have a remote server with sigal install, where I dump pictures. Right now, I mount the pictures directory through SSH and dump photos there. But then I need to ssh on the remote server and run sigal by hand.

I'd like to have some sort of daemon that would sit and watch on a root directory (e.g. ~sigal/pictures in my case) and fire up sigal build once changes are detected. This could be done with inotify or similar tools, but this is tricky: we don't want to start the build as soon as a file is created. What we want, actually, is to trigger the build once changes are done and no activity is occuring after a specific timeout.

I wonder if you have ever thought of implementing this: I know I could do something with inotifywait or similar, but I figured I would look into implementing this in sigal directly as well.

Opinions? Ideas? would a PR to that effect be welcome?

[anarcat]

Fundamentally, there are two main approaches to do this in Linux land: inotify and fanotify. inotify's issue is that it doesn't recursively monitor directories: you need to implement that yourself. it also takes up memory linearly with the number of monitored files. fanotify is a little better because it can recursively look into directories, but it requires root and doesn't detect renames or delete. i take this from this excellent overview of the linux alternatives there.

both have stable python wrappers (inotify and python-fanotify, respectively). but they are both linux-specific, so if we worry about portability at all, we'd probably want to use a wrapper. i found those:

• butter is a modern asyncio thing that does way more than just watch files, not really packaged in linux distros
• watchdog is more minimal and completely multiplatform (including windows), very well packaged across linux distros

Then there's stuff like gamin and fam that have their own python bindings

and of course there are commandline tools like entr and tons more (fswatch, fsnotify, a whole HN! thread about this) that we could connect with without writing our own daemon and leave that in the hands of users + FAQ docs.

[saimn]

I usually use dedicated tools for this, but I agree it could be useful to have a builtin subcommand, possibly with an optional dependency. Another option is to do something more basic, similar to pelican's autorelad (https://github.com/getpelican/pelican/blob/master/pelican/__init__.py#L413). I used watchdog for another project, it seems a reasonable choice.

[anarcat]

i'd love to hear more about how you're doing this with "dedicated tools" :) maybe another FAQ? would save me from writing new code maybe...

[anarcat]

oh and a --autoreload option makes perfect sense as well. maybe it would just be a separate thing to add inotify/watchdog/whatever hooks as we parse the pictures folder, and then fire up the build as needed. even better than what i had in mind, because then we don't need to reparse everything like we would do if we would just call sigal build again.

[anarcat]

looks like pelican rolled their own file watcher using a busy loop... (https://github.com/getpelican/pelican/blob/master/pelican/utils.py#L727) kind of inefficient which is find for pelican as it's a dev tool, but might not be so good for us in production. but the flag idea is good.

[saimn]

i'd love to hear more about how you're doing this with "dedicated tools" :)

I meant tools similar to entr, I typically use peat, so nothing fancy.

... inefficient which is find for pelican as it's a dev tool, but might not be so good for us in production. but the flag idea is good.

By production you mean you want to have something that can run continuously on a server ? In this case yes you probably need something based in inotify.

[anarcat]

but peat just takes a list of files to watch - does it notice new files as well? and won't it fire as soon as the first file is done transferring and repeatedly fire for every new file?

[anarcat]

there's now a FAQ proposed in #285 which could be expanded with this.

[saimn]

but peat just takes a list of files to watch - does it notice new files as well?

Yes, with the --dynamic option the list of files is generated every 1000 ms (by default). But it is just one way to do it without specific development, and other tools could do the same (entr etc.).

This works quite well:

peat --dynamic 'find pictures/' 'sigal build'

Maybe we can add this to the FAQ for now, then if you or someone else want to add a more powerful watch command (or flag) it would be great as well. I like the pelican solution because it is simple enough, has no dependency, and would satisfy most use cases (I think) but a solution based on inotify/watchdog would be fine as well.

[anarcat]

okay, maybe we can start with the FAQ and i can test your answer as an ad-hoc implementation. if i find it unsatisfactory, i'll implement the real thing. ;)

[anarcat]

sorry, i'm confused actually; will you or should i add a new section to the FAQ about this here? i'm not sure how you proceed, so i guess I'd let you do it if you don't mind.

thanks!

[saimn]

Feel free to add something if you want to do it, help is always much appreciated ;). Otherwise I will add something.

[anarcat]

On 2018-01-12 18:41:08, Simon Conseil wrote: Feel free to add something if you want to do it, help is always much appreciated ;). Otherwise I will add something.

I'd rather let you, if you don't mind. :) You have a better grasp of this than i have, obviously.

[saimn]

Okay ;)

[saimn]

Done in 244bc70.
I think we can keep the issue open as it would be great to have a builtin option for this, make something like pelican.

[anarcat]

Excellent, thanks for the hint..

it looks like an equivalent command with entr (which is in debian, and not peat) is:

while true; do  find pictures/ | entr -d sigal build; done

Unfortunately, it only works for a while - at some point, entr crashes with:

entr: directory altered

... which seems rather silly. Now, it could be my shell job control messing around: i had suspended and resumed the process and maybe that broke the while loop. But that's not exactly convenient.

It would also be tricky to setup an actual system-level service for this, which is my ultimate goal... I'll try to see how to implement an --autoreload option to build eventually, and ship a .service file to hook multiple galleries into systemd... But for now, this works. :)

[anarcat]

Update here: for what it's worth, I'm using git-annex and git to manage my galleries now. I dump the high resolution pics in pictures/, add that all to git annex, then run git annex sync --content and a git hook post-receive hook takes care of calling sigal on the remote side:

https://gitlab.com/anarcat/git-hooks/blob/master/sigal-git-hook

I'll see if I can wrap this up in a docs PR now.

[anarcat]

see #351