File system change events (inotify) support for mounted directories

[charleskorn]

Are there any plans to support file system change events for mounted directories through inotify?

This is a key feature for us as we use containers in scenarios like auto-reloading dev servers and re-running unit tests on changes.

If there are no plans to support it at the moment, what kind of effort would be involved in adding this? Depending on the effort required, we might be open to putting together a PR to add support for this.

(If this is really an issue for lima, let me know and I'll raise this there.)

[jandubois]

(If this is really an issue for lima, let me know and I'll raise this there.)

Yes, it is a Lima issue. I don't think it is possible to do inotify over sshfs, as sftp does not support it.

There is some hope that we'll eventually be able to replace sshfs with 9p, but I think that doesn't support inotify either.

If you have ideas, please file issues/prs against lima!

[jandubois]

I think your best bet is probably some kind of notification forwarder, like https://github.com/mhallin/notify-forwarder (there are others). If you do get something working, please post about it to the Lima discussions, or at least update this issue here!

[jandubois]

like https://github.com/mhallin/notify-forwarder

On second looks it seems to just set the atime and mtime fields on the guest to the current time; it doesn't trigger an inotify event, which would require kernel support.

I'm out of ideas; maybe polling is all you can do. Please let us all know if you come up with something better!

[charleskorn]

Yes, it is a Lima issue. I don't think it is possible to do inotify over sshfs, as sftp does not support it.

There is some hope that we'll eventually be able to replace sshfs with 9p, but I think that doesn't support inotify either.

If you have ideas, please file issues/prs against lima!

Done: lima-vm/lima#615

[abiosoft]

I will be closing this in favour of the upstream issue lima-vm/lima#615

[Rylon]

Hi @abiosoft do you know if there was any movement from Lima on this, the most recent comment suggests it's supported in virtiofs now, but I'm not able to get it to work.

[abiosoft]

My tests indicate it is indeed not supported. Some work still needs to be done on Colima's end to provide the feature.

[abiosoft]

You can try out the development version brew install --HEAD colima and try the experimental --mount-inotify flag?

colima start --mount-inotify

[henrik242]

@abiosoft Very cool that there are things happening! I tried it, but there is no change :/ Are there any logs you'd like me to share?

I see loglines like this:

time="2023-03-20T08:25:08+01:00" level=info msg="setting modtime to 2023-03-20 08:25:03 for /Users/foo/src/baz/bar/web/build/next/cache/webpack/client-development/0.pack " context=inotify

But no lines match the files I am changing (/Users/foo/src/baz/bar/web/src/components/navigation/Navigation.tsx). docker-compose.yml is in /Users/foo/src/baz/bar. I have a volume mount like this:

        volumes:
            - ./web:/app

touch /app/web/src/components/navigation/Navigation.tsx from inside the container works.

I'm on colima version HEAD-30efd6c

[abiosoft]

@henrik242 can you share a sample project to replicate your scenario?

I think I know the cause.
Long polling is being used and therefore some checks are put it place to only respond to relevant file changes in order to avoid stressing the CPU.

[henrik242]

@abiosoft This is part of a large project, so it's not all that trivial. "Sadly" it worked in a next.js demo project I tried, but I'll try to find time to create something.

[henrik242]

I get a few of these in the log, are they related? time="2023-03-20T09:10:41+01:00" level=error msg="r.CreateEndpoint() = connection was refused"

[abiosoft]

That does not look related. Are there many log entries with same error?

[henrik242]

No, just about 1 in every 15:

time="2023-03-20T09:29:59+01:00" level=error msg="r.CreateEndpoint() = connection was refused"
time="2023-03-20T09:30:09+01:00" level=info msg="setting modtime to 2023-03-20 09:30:04 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:30:23+01:00" level=info msg="setting modtime to 2023-03-20 09:30:18 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:30:37+01:00" level=info msg="setting modtime to 2023-03-20 09:30:32 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:30:51+01:00" level=info msg="setting modtime to 2023-03-20 09:30:46 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:31:04+01:00" level=info msg="setting modtime to 2023-03-20 09:30:59 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:31:18+01:00" level=info msg="setting modtime to 2023-03-20 09:31:13 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:31:31+01:00" level=info msg="setting modtime to 2023-03-20 09:31:26 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:31:45+01:00" level=info msg="setting modtime to 2023-03-20 09:31:40 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:31:59+01:00" level=info msg="setting modtime to 2023-03-20 09:31:54 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:32:13+01:00" level=info msg="setting modtime to 2023-03-20 09:32:08 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:32:29+01:00" level=info msg="setting modtime to 2023-03-20 09:32:24 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:32:43+01:00" level=info msg="setting modtime to 2023-03-20 09:32:38 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:32:56+01:00" level=info msg="setting modtime to 2023-03-20 09:32:51 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:33:10+01:00" level=info msg="setting modtime to 2023-03-20 09:33:05 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify
time="2023-03-20T09:33:23+01:00" level=info msg="setting modtime to 2023-03-20 09:33:18 for /Users/foo/src/baz/bar/web/build/next/build-manifest.json " context=inotify

[henrik242]

With --very-verbose logging I see this in daemon.log:

time="2023-03-20T10:31:16+01:00" level=trace msg="changed file modTime 2023-03-20 10:30:49.846457862 +0100 CET->2023-03-20 10:31:12.116122505 +0100 CET: /Users/foo/src/baz/bar/web/src/components/navigation/Navigation.tsx"

But no setting modtime and no context=inotify

[abiosoft]

Anyone interested in this feature should follow this issue #261.