inotify doesn't work on 9p filesystem mounts

[bryanlarsen]

Bug Report

Minikube version: v0.19.0

Environment:

• OS: Ubuntu 16.04.2 LTS
• VM Driver: kvm
• ISO version: listed as 0.18.0, but was built from master
• Install tools:
• Others:

What happened:

inotify does not trigger on mounted files.

What you expected to happen:

inotify works, allowing hot reload

How to reproduce it (as minimally and precisely as possible):

Minikube mount a directory:

minikube mount .:/mount-9p

Mount that into a container:

volumeMounts:
- mountPath: /app/src
  name: src
volumes:
- name: src
  hostPath:
    path: /mount-9p/src

kubectl exec into that container,

 apt-get install inotify-tools
 inotifywait -m /app/src/index.js

Edit that file on the host, and note that inotifywait never outputs anything.

Note that the reverse works fine, touching the file from inside the container triggers inotify events on the host.

Even more strange, touching the file from inside the container doesn't trigger inotify events in the container. Touching a file that isn't shared triggers events, but touching a shared file doesn't.

Anything else do we need to know:

[r2d4]

ref #821

Our mount command uses a different 9p implementation than the xhyve driver, but since we control the filesystem mounting now, we might have a little more ability to fix this.

[eav]

I have the same issue in Ubuntu but with VirtualBox and it's a real blocker, it's would be nice to have this resolved!

[jicksta]

@r2d4 Still having this issue.

We've been able to create a minimalistic reproduction of this the issue here:

https://github.com/the-jackalope/repro-fs-reloading

Should help diagnose and verify any fixes. Our README.md there has instructions.

This is quite an impediment to our rolling out of our new Kubernetes developer tooling since our JS dev environment build tools, Rails processes, and anything else in our local dev environment that needs to react to changes of source code files by reloading/rebuilding simply can't without some gnarly workarounds. I would assume other folks will be hitting this issue as minikube is used more often for non-Go developer environments?

Are filesystem events something that 9P2000 doesn't natively support? My cursory search couldn't find any occurrence of filesystem events for the protocol.

Notably, we don't have any issue getting filesystem events via the xhyve-based Docker for Mac VM. Just with minikube. How different are the two mounting mechanisms between those two projects?

Many thanks! ✊

[chrisslater]

Has any progress been made regarding this issue or are there any workarounds that can be used in the meantime?

[RaeesBhatti]

This is a blocker for our use-case. The whole point of minikube is to allow local development on a real Kubernetes environment, which is not possible if file-system events are not propagated. Please take some time out to fix this or point out where it is broken so that people can find a fix themselves.

[christopherL91]

@elsteelbrain This doesn't solves the problem, but the latest docker for mac with kubernetes support does forward inotify events, so if you're using mac's then that's a valid workaround until minikube supports forwarding inotify events.

[ericb]

Note: I've discovered the following over a few cups of coffee and half a Saturday. I've made a lot of assumptions here, and know very little of the 9p protocol.

Unfortunately, I don't think this is possible. At least for inotify, but probably for other systems like OSX File System Events too. The root problem is inotify events can only be generated by the Linux kernel.

So, inotify events will generate on the host when the file is modified when it has been modified on the host directly, as well as when it has been modified by the container (because the modification ultimately happens on the host). However, events will only be generated on the container when the file is modified directly from the container. Changes from the host will not reflect on the container because the host 9p server does not send changes to the clients. (I don't know enough about the protocol, but I assume there's nothing about event notifications to clients).

Workaround:
I've come up with a workaround, but it is application specific and it may not work for you based on your tech stack or tooling. Due to it's specific nature I'm going to try and state it as a series of high-level steps that you'll need to solve for your setup.

1. Develop a small docker image that runs a simple tcp server. No specific implementation is required unless you have additional needs outside of 'some file changed'. If a connection receives data, it should simply touch a file, run a bash script, or some other process you want to happen.

2. Update your existing application deployment with a sidecar container using the image you just built. I suggest using a shared folder between containers for simple 'touch this file' setups.

3. Write a small script to process fs events on your system and send data to the tcp server you just setup on events you want to trigger changes for.

Caveats:

• If you run more than a single application that you need to have update on changes, the script you develop for step 3 should know which files should trigger events on each specific sidecar container for that application.
• The tcp server you create for step 1 will need to do whatever your application needs it to do in order to trigger updates for your app. I think in many cases this will simply be touching a file. (Thinking of webpack builds, web server restarts, etc)
• I have no idea how this will perform, but I think in most cases, it will be better than using polling alternatives.

Thoughts:

• Step 3 could potentially be developed as a minikube addon.
• The image in Step 1 could most likely be reused for most applications (assuming they need something simple like touching a single file).
• If the above could be accomplished, it would make this workaround fairly simple. Something that a few example repos could guide other implementations.

Sorry for the text bomb... hope this helps at least one person.
Cheers 🍻

[bryanlarsen]

Our workaround is to create a Linux VM using something with working file sharing and then use minikube with --vm-driver=None.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[bryanlarsen]

/remove-lifecycle rotten

This issue is still preventing us from rolling out a minikube based dev environment company-wide.

[74io]

/remove-lifecycle rotten

As amazing as the minikube project is, this is a real show stopper. Is a fix even on the road map? Any idea when it will be worth revisiting this project?

[tstromberg]

I suspect that making inotify operational is more work than one might think. NFS doesn't support inotify either, for similar reasons: https://stackoverflow.com/questions/4231243/inotify-with-nfs - this isn't a minikube shortcoming, as much as a kernel module shortcoming.

The obvious workaround is to use inotify outside of minikube, and trigger events to an endpoint (such as over SSH or HTTP) within minikube when changes occur.

[akatebi]

inotify.txt

I have come up with a solution that works. The solution is to measure the size of files periodically and copy the file to itself if it has changed. I have a bash script that proves the concept works. If you improve on this please share it. We need a complete script with more capabilities. I have attached the shell script as text so you can download and rename into a proper shell script name inotify.sh
Here is my simple script:

while [ 1 ]; do
files=$(du -b $(find . -name *.js))
files=$(echo $files | sed 's/ / /g')
declare -a ary1=($files)
sleep 3
files2=$(du -b $(find . -name *.js))
files2=$(echo $files2 | sed 's/ / /g')
declare -a ary2=($files2)
total=${#ary1[@]}
for (( i=0; i<=$(( $total -1 )); i += 2 ))
do
# echo "ary1 => ${ary1[$i]} ${ary1[$i+1]}"
# echo "ary2 => ${ary2[$i]} ${ary2[$i+1]}"
if [ ! ${ary1[$i]} = ${ary2[$i]} ]; then
echo "${ary1[$i+1]} => ${ary1[$i]} != ${ary2[$i]}"
cp ${ary1[$i+1]} /tmp/poll-notify
cp /tmp/poll-notify ${ary1[$i+1]}
fi
done
done

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[reefbarman]

just want to add my two cents here as well, and say this is an important use case for my company as well and makes it difficult to have an effective dev environment with minikube involved

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

As painful as this is, I'm closing this as infeasible.

Currently, inotify does not support notifications from user-space or remote filesystems - and requires kernel level support to be implemented. There are some proposals floating around to make it possible to fix this for user-space drivers such as FUSE: https://github.com/libfuse/libfuse/wiki/Fsnotify-and-FUSE

Once it is possible to inotify on user-space filesystems, I'll be happy to see this issue get re-opened.

[activescott]

Just some notes on workarounds suggested here:
Touching the file, even from within the container (let alone a sidecar) won't trigger inotify on these user-space volumes.
Even modifying the file doesn't.
You can't copy a file to itself as far as I know. For example, on alpine I get cp: 'src/main.ts' and 'src/main.ts' are the same file.

https://github.com/MinikubeAddon/watchpod might be an option but it is pretty heavy handed as it appears to bounce the pod and the image isn't available anymore, but if you look at the issues, you can find a link to how it works. I might try that, but haven't yet.