hledger-ui --watch does not work with certain editors (gedit, IDEA)

[5833tP2]

Further to a recent thread on reddit about methods for reconciliation, there appears to be a bug with hledger-ui where it does not update automatically when changes to the journal file are saved. The workaround is to manually refresh using g.

System specifications

Arch Linux 5.12.13-arch1-2
hledger 1.21

Steps to reproduce

1. Edit a journal file
   $ vim myfile.journal
   Change the flag on any unreconciled transaction in Account A by changing <blank> to *.
   Do not save yet.

2. Open a new terminal window and start hledger-ui
   $ hledger-ui -f myfile.journal --watch
   Navigate to Account A and list the transactions so that the just-edited transaction is visible. It should not show a * flag.

3. Return to the editor window and Save the file.

Expected behavior

The edited transaction is updated in the hledger-ui window and now shows *

Actual behavior

The edited transaction is not updated in the hledger-ui window.

Workaround

In the hledger-ui window, refresh the underlying data by pressing g. The edited transaction is updated and now correctly shows as being reconciled *.

[simonmichael]

Thank you for bringing this to my attention. I see we had #836 some time ago. Do the sysctl parameters mentioned there give any clues ?

[5833tP2]

I just increased the number of inotify watchers using these instructions.

No change to hledger-ui behavior :(

I added the --debug=n option but do not know where to find the debug file.

[simonmichael]

I confirm your exact scenario works normally on mac. Also, starting with hledger-ui -f myfile.journal --watch --debug=1, after saving two file changes the following output is shown at the bottom of the window:

─────────────── ?:help t:list/tree -+:depth H:end-bals/changes F:forecast a:add q:quit ──────────────
fsnotify:   "Modified "/Users/simon/src/hledger/myfile.journal" 2021-07-19 16:48:46.539822 UTC False"
fsnotify:   "Modified "/Users/simon/src/hledger/myfile.journal" 2021-07-19 16:48:51.731687 UTC False"

(until next keypress).

[simonmichael]

I didn't see anything obvious at https://github.com/haskell-fswatch/hfsnotify/issues . Probably the next step is to make a minimal reproducing program and open a bug report there..

[5833tP2]

I made one change to the journal and saved it, and the --debug=1 information is as follows:

[me@mymachine ~Documents/Ledger]$ hledger-ui -f my.journal --watch --debug=1
fsnotify using polling ?: False
files:      [ "/home/me/Documents/Ledger/my.journal" ]
directories to watch: [ "/home/me/Documents/Ledger" ]
[me@mymachine ~Documents/Ledger]$

I do not know whether or not this is helpful.

I am quite happy to hit g to refresh the hledger-ui window after an edit. It might not be optimal but I can live with it.

[simonmichael]

Thanks! We'll get this fixed.

[simonmichael]

How to watch hledger-ui fsnotify debug output:

$ hledger-ui -f a.j --watch --debug=1 >a.log
files:      [ "/home/simon/src/hledger/a.j" ]
directories to watch: [ "/home/simon/src/hledger" ]
(SAVE FILE CHANGES)
fsnotify:   "Modified "/home/simon/src/hledger/a.j" 2021-07-19 17:44:51.80486088 UTC False"
fsnotify:   "Modified "/home/simon/src/hledger/a.j" 2021-07-19 17:45:19.545594613 UTC False"
(PRESS q)
$

[simonmichael]

At least two people have reported this problem, but I can't reproduce on a fresh arch system. @5833tP2, if you have time for further debugging, here or (quicker) on http://matrix.hledger.org or http://irc.hledger.org, I would appreciate your help; if not, no problem. Likewise if anyone else can reproduce this, please get in touch. Here's some version info from my arch linux system:

[root@localhost ~]# uname -a
Linux localhost 5.12.15-arch1-1 #1 SMP PREEMPT Wed, 07 Jul 2021 23:35:29 +0000 x86_64 GNU/Linux
[root@localhost ~]# pacman -Q | grep -iE '(glibc|notify)'
glibc 2.33-5
haskell-fsnotify 0.3.0.1-209
haskell-hinotify 0.4.1-64

hledger-ui version doesn't seem to matter too much, I have tested both 1.21 and 1.22.

[Xitian9]

I am unable to reproduce on the following system.

$ uname -a
Linux localhost 5.13.1 #1-NixOS SMP Wed Jul 7 12:24:58 UTC 2021 x86_64 GNU/Linux
$ nix-env -qa glibc
glibc-2.32-48
$ nix-env -f "<nixpkgs>" -qa -A haskellPackages "fsnotify|hinotify"
fsnotify-0.3.0.1
hinotify-0.4.1
$ findmnt -n -o FSTYPE -T .
ext4

[simonmichael]

Also @5833tP2, what kind of filesystem are you on ? Mine is ext4:

# findmnt -n -o FSTYPE -T .
ext4

[5833tP2]

$ pacman -Q | grep -iE '(glibc|notify)'
glibc 2.33-5
haskell-fsnotify 0.3.0.1-209
haskell-hinotify 0.4.1-64
inotify-tools 3.20.11.0-1
libnotify 0.7.9-1
ruby-rb-inotify 0.10.1-1

I am also on ext4

$findmnt -n -o FSTYPE -T .
ext4

[simonmichael]

It's a mystery. Is there anything else unusual about your test journal ? If it has more than just the one entry, try that:

2021-01-01 *
  (a)   1

[5833tP2]

My journal is just an ordinary ledger journal file. Plain text. Nothing weird.

I just tested this ...

2021-01-01 *
  (a)   1

... and the hledger-ui --watch behavior is the same. No auto-updating but the view does refresh when I press g.

Honestly, this is not a show-stopper problem so maybe we could park this for the time being. Pressing g is not onerous as a workaround.

[simonmichael]

If you need to drop it I totally understand. You've been helpful already. For me and the project, it's important to fix this and you're the only person reproducing it so I'm just getting what info I can. :)

[5833tP2]

I'm happy to help in any way I can, but might reach the limit of my technical competence soon :)

[agander]

On a Fedora 32 setup I cannot reproduce the error:

uname -a
Linux toast 5.11.22-100.fc32.x86_64 #1 SMP Wed May 19 18:58:25 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
qa glibc
glibc-2.31-6.fc32.x86_64
Completed    fsnotify-0.3.0.1 (lib)
Completed    vty-5.33 (lib)
Completed    hinotify-0.4.1 (lib)
findmnt -n -o FSTYPE -T .
ext4

[simonmichael]

@5833tP2, I don't suppose it would be possible for me to log in to your machine ?

I believe there are tools, like debug-me, where you can monitor and supervise the session.

[5833tP2]

@simonmichael, I've emailed you.

[simonmichael]

Progress! Thanks to @5833tP2 we found that it's editor related. gedit saves files in some way that defeats inotify. I now remember having the same problem with IDEA.

[simonmichael]

#836 might be relevant, if gedit uses this "gocryptfs mount" somehow. If so we'd expect the fs.inotify.max_user_watches workaround, discussed in more detail at flathub/com.visualstudio.code#29, to help - but you already tried that. Perhaps it didn't quite get applied to the current terminal environment or something. [Perhaps other Gnome users could try it as well ?]

[simonmichael]

See also #911.

I have added https://hledger.org/dev/hledger-ui.html#watch-mode docs mentioning this issue.

[simonmichael]

Open questions:

• when does the fs.inotify.max_user_watches hack work ?
• why didn't it work for @5833tP2 ?
• does the number of files (on the filesystem / below the data directory) affect whether this issue occurs ?