New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inotify doesn't work on 9p filesystem mounts #1551

Open
bryanlarsen opened this Issue Jun 2, 2017 · 14 comments

Comments

Projects
None yet
@bryanlarsen
Contributor

bryanlarsen commented Jun 2, 2017

Bug Report

Minikube version: v0.19.0

Environment:

  • OS: Ubuntu 16.04.2 LTS
  • VM Driver: kvm
  • ISO version: listed as 0.18.0, but was built from master
  • Install tools:
  • Others:

What happened:

inotify does not trigger on mounted files.

What you expected to happen:

inotify works, allowing hot reload

How to reproduce it (as minimally and precisely as possible):

Minikube mount a directory:

minikube mount .:/mount-9p

Mount that into a container:

volumeMounts:
- mountPath: /app/src
  name: src
volumes:
- name: src
  hostPath:
    path: /mount-9p/src

kubectl exec into that container,

 apt-get install inotify-tools
 inotifywait -m /app/src/index.js

Edit that file on the host, and note that inotifywait never outputs anything.

Note that the reverse works fine, touching the file from inside the container triggers inotify events on the host.

Even more strange, touching the file from inside the container doesn't trigger inotify events in the container. Touching a file that isn't shared triggers events, but touching a shared file doesn't.

Anything else do we need to know:

@r2d4 r2d4 added drv/kvm kind/bug and removed drv/kvm labels Jun 2, 2017

@r2d4

This comment has been minimized.

Member

r2d4 commented Jun 2, 2017

ref #821

Our mount command uses a different 9p implementation than the xhyve driver, but since we control the filesystem mounting now, we might have a little more ability to fix this.

@eav

This comment has been minimized.

eav commented Jun 20, 2017

I have the same issue in Ubuntu but with VirtualBox and it's a real blocker, it's would be nice to have this resolved!

@jicksta

This comment has been minimized.

jicksta commented Aug 1, 2017

@r2d4 Still having this issue.

We've been able to create a minimalistic reproduction of this the issue here:

https://github.com/the-jackalope/repro-fs-reloading

Should help diagnose and verify any fixes. Our README.md there has instructions.

This is quite an impediment to our rolling out of our new Kubernetes developer tooling since our JS dev environment build tools, Rails processes, and anything else in our local dev environment that needs to react to changes of source code files by reloading/rebuilding simply can't without some gnarly workarounds. I would assume other folks will be hitting this issue as minikube is used more often for non-Go developer environments?

Are filesystem events something that 9P2000 doesn't natively support? My cursory search couldn't find any occurrence of filesystem events for the protocol.

Notably, we don't have any issue getting filesystem events via the xhyve-based Docker for Mac VM. Just with minikube. How different are the two mounting mechanisms between those two projects?

Many thanks!

@chrisslater

This comment has been minimized.

chrisslater commented Oct 13, 2017

Has any progress been made regarding this issue or are there any workarounds that can be used in the meantime?

@raeesbhatti

This comment has been minimized.

Contributor

raeesbhatti commented Dec 29, 2017

This is a blocker for our use-case. The whole point of minikube is to allow local development on a real Kubernetes environment, which is not possible if file-system events are not propagated. Please take some time out to fix this or point out where it is broken so that people can find a fix themselves.

@christopherL91

This comment has been minimized.

christopherL91 commented Jan 8, 2018

@elsteelbrain This doesn't solves the problem, but the latest docker for mac with kubernetes support does forward inotify events, so if you're using mac's then that's a valid workaround until minikube supports forwarding inotify events.

@ericb

This comment has been minimized.

ericb commented Jan 27, 2018

Note: I've discovered the following over a few cups of coffee and half a Saturday. I've made a lot of assumptions here, and know very little of the 9p protocol.

Unfortunately, I don't think this is possible. At least for inotify, but probably for other systems like OSX File System Events too. The root problem is inotify events can only be generated by the Linux kernel.

So, inotify events will generate on the host when the file is modified when it has been modified on the host directly, as well as when it has been modified by the container (because the modification ultimately happens on the host). However, events will only be generated on the container when the file is modified directly from the container. Changes from the host will not reflect on the container because the host 9p server does not send changes to the clients. (I don't know enough about the protocol, but I assume there's nothing about event notifications to clients).

Workaround:
I've come up with a workaround, but it is application specific and it may not work for you based on your tech stack or tooling. Due to it's specific nature I'm going to try and state it as a series of high-level steps that you'll need to solve for your setup.

  1. Develop a small docker image that runs a simple tcp server. No specific implementation is required unless you have additional needs outside of 'some file changed'. If a connection receives data, it should simply touch a file, run a bash script, or some other process you want to happen.

  2. Update your existing application deployment with a sidecar container using the image you just built. I suggest using a shared folder between containers for simple 'touch this file' setups.

  3. Write a small script to process fs events on your system and send data to the tcp server you just setup on events you want to trigger changes for.

Caveats:

  • If you run more than a single application that you need to have update on changes, the script you develop for step 3 should know which files should trigger events on each specific sidecar container for that application.
  • The tcp server you create for step 1 will need to do whatever your application needs it to do in order to trigger updates for your app. I think in many cases this will simply be touching a file. (Thinking of webpack builds, web server restarts, etc)
  • I have no idea how this will perform, but I think in most cases, it will be better than using polling alternatives.

Thoughts:

  • Step 3 could potentially be developed as a minikube addon.
  • The image in Step 1 could most likely be reused for most applications (assuming they need something simple like touching a single file).
  • If the above could be accomplished, it would make this workaround fairly simple. Something that a few example repos could guide other implementations.

Sorry for the text bomb... hope this helps at least one person.
Cheers 🍻

@bryanlarsen

This comment has been minimized.

Contributor

bryanlarsen commented Apr 17, 2018

Our workaround is to create a Linux VM using something with working file sharing and then use minikube with --vm-driver=None.

@fejta-bot

This comment has been minimized.

fejta-bot commented Jul 16, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@fejta-bot

This comment has been minimized.

fejta-bot commented Aug 15, 2018

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@bryanlarsen

This comment has been minimized.

Contributor

bryanlarsen commented Aug 15, 2018

/remove-lifecycle rotten

This issue is still preventing us from rolling out a minikube based dev environment company-wide.

@74io

This comment has been minimized.

74io commented Sep 3, 2018

/remove-lifecycle rotten

As amazing as the minikube project is, this is a real show stopper. Is a fix even on the road map? Any idea when it will be worth revisiting this project?

@tstromberg tstromberg changed the title from inotify doesn't work on shared files to inotify doesn't work on 9p filesystem mounts Sep 19, 2018

@tstromberg tstromberg added kind/feature and removed kind/bug labels Sep 19, 2018

@tstromberg

This comment has been minimized.

Collaborator

tstromberg commented Sep 19, 2018

I suspect that making inotify operational is more work than one might think. NFS doesn't support inotify either, for similar reasons: https://stackoverflow.com/questions/4231243/inotify-with-nfs - this isn't a minikube shortcoming, as much as a kernel module shortcoming.

The obvious workaround is to use inotify outside of minikube, and trigger events to an endpoint (such as over SSH or HTTP) within minikube when changes occur.

@s12chung s12chung referenced this issue Oct 5, 2018

Merged

Setup Kubernetes #3

@akatebi

This comment has been minimized.

akatebi commented Oct 13, 2018

inotify.txt

I have come up with a solution that works. The solution is to measure the size of files periodically and copy the file to itself if it has changed. I have a bash script that proves the concept works. If you improve on this please share it. We need a complete script with more capabilities. I have attached the shell script as text so you can download and rename into a proper shell script name inotify.sh
Here is my simple script:

while [ 1 ]; do
files=$(du -b $(find . -name *.js))
files=$(echo $files | sed 's/ / /g')
declare -a ary1=($files)
sleep 3
files2=$(du -b $(find . -name *.js))
files2=$(echo $files2 | sed 's/ / /g')
declare -a ary2=($files2)
total=${#ary1[@]}
for (( i=0; i<=$(( $total -1 )); i += 2 ))
do
# echo "ary1 => ${ary1[$i]} ${ary1[$i+1]}"
# echo "ary2 => ${ary2[$i]} ${ary2[$i+1]}"
if [ ! ${ary1[$i]} = ${ary2[$i]} ]; then
echo "${ary1[$i+1]} => ${ary1[$i]} != ${ary2[$i]}"
cp ${ary1[$i+1]} /tmp/poll-notify
cp /tmp/poll-notify ${ary1[$i+1]}
fi
done
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment