New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

no automatic restart when using boot2docker docker volume #419

Closed
chrishiestand opened this Issue Nov 20, 2014 · 65 comments

Comments

@chrishiestand

nodemon doesn't seem to restart node when using a docker volume, or in this case a boot2docker ( os x -> virtualbox -> docker) volume. nodemon starts okay, but then fails to restart node when a watched file has been modified in the shared OS X volume.

root@760d5d2ff8cf:/src# nodemon --watch index.js 
20 Nov 21:06:58 - [nodemon] v1.2.1
20 Nov 21:06:58 - [nodemon] to restart at any time, enter `rs`
20 Nov 21:06:58 - [nodemon] watching: index.js
20 Nov 21:06:58 - [nodemon] starting `nodejs index.js`
Running on http://localhost:8080
request received
#index.js changed and saved here but output does not change as it should
request received
request received

nodemon -L causes restarts at least once a second even when no files have changed - way too frequently.

forever -w seems to work just fine. That seems like the best workaround for now.

output of dump:

root@b36ab2778133:/src# nodemon --dump
20 Nov 22:08:27 - [nodemon] v1.2.1
20 Nov 22:08:27 - [nodemon] to restart at any time, enter `rs`
20 Nov 22:08:27 - [nodemon] watching: *.*
--------------
node: v0.10.25
nodemon: v1.2.1
command: node /usr/local/bin/nodemon --dump
cwd: /src
OS: linux x64
--------------
{ run: false,
  system: 
   { cwd: '/src',
     useFind: false,
     useWatch: true,
     useWatchFile: false },
  required: false,
  dirs: [ '/src' ],
  timeout: 1000,
  options: 
   { scriptPosition: 0,
     dump: true,
     script: 'index.js',
     args: [],
     ignore: 
      [ '.git',
        'node_modules',
        'bower_components',
        '.sass-cache',
        re: /\.git|node_modules|bower_components|\.sass\-cache/ ],
     watch: [ '*.*', re: /.*\..*/ ],
     restartable: 'rs',
     execMap: { py: 'python', rb: 'ruby' },
     stdin: true,
     verbose: false,
     stdout: true,
     execOptions: 
      { script: 'index.js',
        exec: 'node',
        args: [],
        nodeArgs: undefined,
        ext: 'js',
        env: {},
        execArgs: [] },
     ext: 'js',
     monitor: 
      [ '*.*',
        '!.git',
        '!/src/node_modules/**/*',
        '!bower_components',
        '!.sass-cache' ] },
  load: [Function],
  reset: [Function: reset],
  lastStarted: 0,
  loaded: [],
  command: 
   { raw: { executable: 'node', args: [ 'index.js' ] },
     string: 'node index.js' } }
@chisleu

This comment has been minimized.

Show comment
Hide comment
@chisleu

chisleu Nov 23, 2014

When I use boot2docker with my branch off of library/node, when I try to use nodemon /path/to/app --watch, the cpu goes nuts like it is constantly restarting, although I can't verify.

I tried -L, but it also caused the CPU to go nuts.

I changed to pm2 for now, but it has 2-3 seconds of delay before it restarts. Still, better than nothing.

chisleu commented Nov 23, 2014

When I use boot2docker with my branch off of library/node, when I try to use nodemon /path/to/app --watch, the cpu goes nuts like it is constantly restarting, although I can't verify.

I tried -L, but it also caused the CPU to go nuts.

I changed to pm2 for now, but it has 2-3 seconds of delay before it restarts. Still, better than nothing.

@tanis2000

This comment has been minimized.

Show comment
Hide comment
@tanis2000

tanis2000 Nov 25, 2014

+1 this happens to me as well with a mounted volume on boot2docker (OSX + VirtualBox).

+1 this happens to me as well with a mounted volume on boot2docker (OSX + VirtualBox).

@remy remy added the help wanted label Dec 4, 2014

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Dec 4, 2014

Owner

I'm marking as "help wanted" - I don't have the infrastructure to even begin testing and debugging this.

I'm sure it's linked to clocks being out of whack somehow, but can't replicate myself with simple tests.

Owner

remy commented Dec 4, 2014

I'm marking as "help wanted" - I don't have the infrastructure to even begin testing and debugging this.

I'm sure it's linked to clocks being out of whack somehow, but can't replicate myself with simple tests.

@tanis2000

This comment has been minimized.

Show comment
Hide comment
@tanis2000

tanis2000 Dec 4, 2014

@remy if I can help you in any way, just let me know what I should do.

@remy if I can help you in any way, just let me know what I should do.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Dec 4, 2014

Owner

@tanis2000 if you can pull the project inside of your virtual machine. I'd start by trying to discover why -L sets the CPU insane. The watching logic is all in watch.js - it's a little complicated because there's multiple methods to actually watch.

Really you want to get a handle on what's causing the issue in the first place. I usually console log the crap out of the code when I'm spelunking for errors.

Owner

remy commented Dec 4, 2014

@tanis2000 if you can pull the project inside of your virtual machine. I'd start by trying to discover why -L sets the CPU insane. The watching logic is all in watch.js - it's a little complicated because there's multiple methods to actually watch.

Really you want to get a handle on what's causing the issue in the first place. I usually console log the crap out of the code when I'm spelunking for errors.

@mattapperson

This comment has been minimized.

Show comment
Hide comment
@mattapperson

mattapperson Dec 16, 2014

still not seeing an automatic restart with a volume mounted nodejs project in b2d

still not seeing an automatic restart with a volume mounted nodejs project in b2d

@splodingsocks

This comment has been minimized.

Show comment
Hide comment
@splodingsocks

splodingsocks Jan 2, 2015

Remy, thanks for nodemon. Did you close this issue because you resolved it? Or did you not have enough helpful feedback to fix?

We've been experiencing the same problems as mentioned above.

Remy, thanks for nodemon. Did you close this issue because you resolved it? Or did you not have enough helpful feedback to fix?

We've been experiencing the same problems as mentioned above.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 3, 2015

Owner

No, it looks like it was closed by mistake in the associated PR. I'll reopen and ask for help (from you lot!).

Owner

remy commented Jan 3, 2015

No, it looks like it was closed by mistake in the associated PR. I'll reopen and ask for help (from you lot!).

@remy remy reopened this Jan 3, 2015

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 4, 2015

Owner

If anyone wants to push a Docker image that I can test and debug this with – I'd be grateful.

Owner

remy commented Jan 4, 2015

If anyone wants to push a Docker image that I can test and debug this with – I'd be grateful.

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 5, 2015

👍 same problem here

olalonde commented Jan 5, 2015

👍 same problem here

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

@olalonde then same question to you: "[do you have] a Docker image that I can test and debug this with – I'd be grateful.".

Owner

remy commented Jan 5, 2015

@olalonde then same question to you: "[do you have] a Docker image that I can test and debug this with – I'd be grateful.".

@chisleu

This comment has been minimized.

Show comment
Hide comment
@chisleu

chisleu Jan 5, 2015

I will make one if someone else doesn't. I can't get to it until tonight though, which is why I had not yet replied.

Assuming Ubuntu 12.04 is ok unless I hear otherwise.

A faster solution would be to use vagrant. If you have vagrant and virtualbox, you can install a VM with

vagrant up ubuntu/precise64 --provider virtualbox
vagrant ssh

Then install node/npm and nodemon. Place a node file in a shared resource (such as /vagrant) and you should find these problems to be easy to reproduce.

I'm not sure about docker as I no longer use docker for development (because of this issue, and because I use MacOSX and the boot to docker image causes CPU and sleep problems for my little Macbook Air.)

It still affects my work in Vagrant and seems to have the same results.

BTW, PM2 suffers a similar fate now in Vagrant. It simply fails to restart at all.

chisleu commented Jan 5, 2015

I will make one if someone else doesn't. I can't get to it until tonight though, which is why I had not yet replied.

Assuming Ubuntu 12.04 is ok unless I hear otherwise.

A faster solution would be to use vagrant. If you have vagrant and virtualbox, you can install a VM with

vagrant up ubuntu/precise64 --provider virtualbox
vagrant ssh

Then install node/npm and nodemon. Place a node file in a shared resource (such as /vagrant) and you should find these problems to be easy to reproduce.

I'm not sure about docker as I no longer use docker for development (because of this issue, and because I use MacOSX and the boot to docker image causes CPU and sleep problems for my little Macbook Air.)

It still affects my work in Vagrant and seems to have the same results.

BTW, PM2 suffers a similar fate now in Vagrant. It simply fails to restart at all.

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 5, 2015

Ok, I just made a test but can't push it to Github at the moment.

Install boot2docker. Run $(boot2docker shellinit).

Put all the following files in the same directory (e.g. nodemontest).

./Dockerfile

FROM node:0.10-slim
RUN mkdir /app
WORKDIR /app

CMD ["./node_modules/nodemon/bin/nodemon.js", "index.js"]

./index.js

console.log(new Date());

./package.json

{
  "name": "testnodemon",
  "devDependencies": {
    "nodemon": "^1.2.1"
  }
}

Once, you have those files, run the following commands.

npm install;
docker build -t testnodemon .;
docker run -v $(pwd):/app testnodemon;

Edit index.js, nodemon doesn't restart the script.

You can use the following to get a shell in the container:

docker run -i -t -v $(pwd):/app testnodemon /bin/bash;
root@762c18df76ee:/app# mount | grep app
none on /app type vboxsf (rw,nodev,relatime)

Related? https://www.virtualbox.org/ticket/10660

olalonde commented Jan 5, 2015

Ok, I just made a test but can't push it to Github at the moment.

Install boot2docker. Run $(boot2docker shellinit).

Put all the following files in the same directory (e.g. nodemontest).

./Dockerfile

FROM node:0.10-slim
RUN mkdir /app
WORKDIR /app

CMD ["./node_modules/nodemon/bin/nodemon.js", "index.js"]

./index.js

console.log(new Date());

./package.json

{
  "name": "testnodemon",
  "devDependencies": {
    "nodemon": "^1.2.1"
  }
}

Once, you have those files, run the following commands.

npm install;
docker build -t testnodemon .;
docker run -v $(pwd):/app testnodemon;

Edit index.js, nodemon doesn't restart the script.

You can use the following to get a shell in the container:

docker run -i -t -v $(pwd):/app testnodemon /bin/bash;
root@762c18df76ee:/app# mount | grep app
none on /app type vboxsf (rw,nodev,relatime)

Related? https://www.virtualbox.org/ticket/10660

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

@olalonde thanks for this. I'll go through the steps this week to try to replicate. I'm sure it's linked the latest information in this issue #11 but I need to see it first hand.

Owner

remy commented Jan 5, 2015

@olalonde thanks for this. I'll go through the steps this week to try to replicate. I'm sure it's linked the latest information in this issue #11 but I need to see it first hand.

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 5, 2015

The problem I believe is that vboxsf doesn't support inotify notifications. The alternative would be to poll the directory but that could be end up eating all the CPU on large code bases. Another solution which I believe was adopted by Ruby's guard is to have a client on the host machine which sends restart messages to a another process running on the guest machine.

Anyways, I hope you can come up with something better :)

olalonde commented Jan 5, 2015

The problem I believe is that vboxsf doesn't support inotify notifications. The alternative would be to poll the directory but that could be end up eating all the CPU on large code bases. Another solution which I believe was adopted by Ruby's guard is to have a client on the host machine which sends restart messages to a another process running on the guest machine.

Anyways, I hope you can come up with something better :)

@chisleu

This comment has been minimized.

Show comment
Hide comment
@chisleu

chisleu Jan 5, 2015

It is such a problem to resolve that we started running node locally with a remote Cassandra DB (since C* would be the only service in docker.)

chisleu commented Jan 5, 2015

It is such a problem to resolve that we started running node locally with a remote Cassandra DB (since C* would be the only service in docker.)

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 5, 2015

Another solution would be to watch files normally on the host and whenever a file changes, write to .file-changed. The nodemon running in the container could just tail .file-changed and reload the process whenever it receives new data.

olalonde commented Jan 5, 2015

Another solution would be to watch files normally on the host and whenever a file changes, write to .file-changed. The nodemon running in the container could just tail .file-changed and reload the process whenever it receives new data.

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 5, 2015

I just submitted a PR with the solution I posted above. It's not elegant, but it's simple and it works.

Basically, nodemon needs to run on the host with the --local flag:

nodemon --local

and the VM/container can listen for file change events by using the --remote flag:

nodemon --remote index.js

The local nodemon communicates with the remote one by toggling a file called .nodemon-ping

olalonde commented Jan 5, 2015

I just submitted a PR with the solution I posted above. It's not elegant, but it's simple and it works.

Basically, nodemon needs to run on the host with the --local flag:

nodemon --local

and the VM/container can listen for file change events by using the --remote flag:

nodemon --remote index.js

The local nodemon communicates with the remote one by toggling a file called .nodemon-ping

remy added a commit that referenced this issue Jan 5, 2015

Use find on linux (as per mac)
Though the command is slightly different, so fork the code logic. This will still fall back to alternative method if find isn't available.

Ref #419
@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

Can you try the development version of nodemon that I've pushed? It contains a lot of other tweaks, but the main thing I've changed is it'll use find instead of fs.watchFile (which I think uses the broken IO notify...).

To install:

npm install -g nodemon@dev

Can someone quickly give me an idea of whether that works or not? I tried it myself, but I'm still pretty new to docker so wasn't quite sure of what I was doing...

Owner

remy commented Jan 5, 2015

Can you try the development version of nodemon that I've pushed? It contains a lot of other tweaks, but the main thing I've changed is it'll use find instead of fs.watchFile (which I think uses the broken IO notify...).

To install:

npm install -g nodemon@dev

Can someone quickly give me an idea of whether that works or not? I tried it myself, but I'm still pretty new to docker so wasn't quite sure of what I was doing...

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

(it should be nodemon@1.3.0-7)

Owner

remy commented Jan 5, 2015

(it should be nodemon@1.3.0-7)

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

Nope, okay, I'm pretty sure nodemon@1.3.0-7 farts and makes a mess in docker. Anyone else want to confirm that's cool, otherwise I'll take this as a fail.

Owner

remy commented Jan 5, 2015

Nope, okay, I'm pretty sure nodemon@1.3.0-7 farts and makes a mess in docker. Anyone else want to confirm that's cool, otherwise I'll take this as a fail.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

Shoot, the issue I'm seeing is this: #454.

Owner

remy commented Jan 5, 2015

Shoot, the issue I'm seeing is this: #454.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 5, 2015

Owner

Sod. I've fixed up all the issues, but now the clock is the main issue. The container says it's 9am, and current time is 10pm, so when I change on my local machine, the container doesn't get the file change correctly, and actually thinks everything is changing all the time...

Owner

remy commented Jan 5, 2015

Sod. I've fixed up all the issues, but now the clock is the main issue. The container says it's 9am, and current time is 10pm, so when I change on my local machine, the container doesn't get the file change correctly, and actually thinks everything is changing all the time...

@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jan 6, 2015

The container says it's 9am, and current time is 10pm

Yes I recall having a similar problem (files modified in the future from the POV of the container).

olalonde commented Jan 6, 2015

The container says it's 9am, and current time is 10pm

Yes I recall having a similar problem (files modified in the future from the POV of the container).

@keithamus

This comment has been minimized.

Show comment
Hide comment
@keithamus

keithamus Jan 8, 2015

VMs will fall out of clock sync when you sleep the host machine - so if your clock is messing up on your VM, then you need to have it query the NTP servers to adjust its clock.

For boot2docker (which I assume, as we're talking about docker you're using) you can run the following:

boot2docker ssh sudo ntpclient -s -h pool.ntp.org

and it'll fix clock sync issues. For other VMs you can install the ntp service, e.g. for ubuntu:

sudo apt-get install ntp && sudo start ntp

VMs will fall out of clock sync when you sleep the host machine - so if your clock is messing up on your VM, then you need to have it query the NTP servers to adjust its clock.

For boot2docker (which I assume, as we're talking about docker you're using) you can run the following:

boot2docker ssh sudo ntpclient -s -h pool.ntp.org

and it'll fix clock sync issues. For other VMs you can install the ntp service, e.g. for ubuntu:

sudo apt-get install ntp && sudo start ntp
@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 8, 2015

Owner

@keithamus this is fine, but I suspect there's times when aligning the clock isn't the solution. Say you're creating a Dockerfile for your US based deploy that runs on US time, but you're developing in the UK.

I've got an idea that I think is fairly solid (and @olalonde has proposed a PR #453 that ping/pongs across the VM), but it all still boils down to somehow telling the user that running nodemon "regular" style ain't going to fly. Something extra is required. Either ntp, or an extra flag/step in running nodemon.

Owner

remy commented Jan 8, 2015

@keithamus this is fine, but I suspect there's times when aligning the clock isn't the solution. Say you're creating a Dockerfile for your US based deploy that runs on US time, but you're developing in the UK.

I've got an idea that I think is fairly solid (and @olalonde has proposed a PR #453 that ping/pongs across the VM), but it all still boils down to somehow telling the user that running nodemon "regular" style ain't going to fly. Something extra is required. Either ntp, or an extra flag/step in running nodemon.

@keithamus

This comment has been minimized.

Show comment
Hide comment
@keithamus

keithamus Jan 8, 2015

To add, if your VM/docker seems to be an exact number of hours out (i.e. the minute and second values are both the same) then it is likely your VM or Docker container have a different timezone set to your local machine. In this case, you need to change your timezone in the guest.

For docker, moby/moby#3359 seems to suggest you can copy over your host machines /etc/localtime file over to your container:

docker run -v /etc/localtime:/etc/localtime:ro`

For other VMs, you need to override your systems timezone; for example Ubuntu:

echo "Australia/Adelaide" | sudo tee /etc/timezone && sudo dpkg-reconfigure --frontend noninteractive tzdata

To add, if your VM/docker seems to be an exact number of hours out (i.e. the minute and second values are both the same) then it is likely your VM or Docker container have a different timezone set to your local machine. In this case, you need to change your timezone in the guest.

For docker, moby/moby#3359 seems to suggest you can copy over your host machines /etc/localtime file over to your container:

docker run -v /etc/localtime:/etc/localtime:ro`

For other VMs, you need to override your systems timezone; for example Ubuntu:

echo "Australia/Adelaide" | sudo tee /etc/timezone && sudo dpkg-reconfigure --frontend noninteractive tzdata
@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 8, 2015

Owner

I'm actually seeing something very strange here.

When I create a file (with node) in the container, and then stat the file, I get this:

{ dev: 25,
  mode: 33188,
  nlink: 1,
  uid: 1000,
  gid: 50,
  rdev: 0,
  blksize: 4096,
  ino: 448,
  size: 11,
  blocks: 8,
  atime: Wed Jan 07 2015 23:54:22 GMT+0000 (UTC),
  mtime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC),
  ctime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC) }

Notice the mtime compared to the atime. The access time is from where I read it, but I created in the VM whose date is: Wed Jan 7 23:56:34 UTC 2015 - so the creation time is actually based on the host machine...

Is this normal? If so, this would be enough to confirm the offset.

Anyone able to try this in different environments?

Owner

remy commented Jan 8, 2015

I'm actually seeing something very strange here.

When I create a file (with node) in the container, and then stat the file, I get this:

{ dev: 25,
  mode: 33188,
  nlink: 1,
  uid: 1000,
  gid: 50,
  rdev: 0,
  blksize: 4096,
  ino: 448,
  size: 11,
  blocks: 8,
  atime: Wed Jan 07 2015 23:54:22 GMT+0000 (UTC),
  mtime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC),
  ctime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC) }

Notice the mtime compared to the atime. The access time is from where I read it, but I created in the VM whose date is: Wed Jan 7 23:56:34 UTC 2015 - so the creation time is actually based on the host machine...

Is this normal? If so, this would be enough to confirm the offset.

Anyone able to try this in different environments?

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Jan 8, 2015

Owner

Update: I currently have an offline dev version of nodemon that runs inside of a container and picks up changes made on the host.

It's not ready yet, but it's working without any extra flags or hoops to jump...

Owner

remy commented Jan 8, 2015

Update: I currently have an offline dev version of nodemon that runs inside of a container and picks up changes made on the host.

It's not ready yet, but it's working without any extra flags or hoops to jump...

remy added a commit that referenced this issue Jan 8, 2015

Find on linux and detect container offset
Closes #455
Closes #453
Fixes #419

This method creates a file and checks the stat to check of an offset between the date on the machine and the created time on the file system. Docker (in my tests) show that you can get the offset using this method.

Once we have the offset, we tell the user (once) and then we use that to find files that have been modified according to the offset (either looking in the last using -mmin or in the future using -newer + a touched file).
@olalonde

This comment has been minimized.

Show comment
Hide comment
@olalonde

olalonde Jul 2, 2015

@ChrisCinelli there was unfortunately a previous bug report that was marked as "wontfix": https://www.virtualbox.org/ticket/10660

olalonde commented Jul 2, 2015

@ChrisCinelli there was unfortunately a previous bug report that was marked as "wontfix": https://www.virtualbox.org/ticket/10660

@ChrisCinelli

This comment has been minimized.

Show comment
Hide comment
@ChrisCinelli

ChrisCinelli Jul 2, 2015

I saw it. It was also 3 years ago. And the new one is limited in scope: asking just support for Mac OS X host and Linux guests (ext3/ext4)

I saw it. It was also 3 years ago. And the new one is limited in scope: asking just support for Mac OS X host and Linux guests (ext3/ext4)

@ChrisCinelli

This comment has been minimized.

Show comment
Hide comment
@ChrisCinelli

ChrisCinelli Jul 2, 2015

It may not be a bad idea to have other people to comment on it to "upvote it"

It may not be a bad idea to have other people to comment on it to "upvote it"

@pyper

This comment has been minimized.

Show comment
Hide comment
@pyper

pyper Jul 9, 2015

I think I may have found what issue is, and a simple workaround to correct it. I am using boot2docker and docker-compose for a multi container environment in OS X. When I add the --watch path to my command path in a docker-compose.yml file it works perfectly, and when I remove it I have the same issue as everyone here. Tested many times over.

I looked into it further I found something interesting. When I do a --dump with the --watch path set I get the following:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/src/code' ],

Now watch what happens without the --watch path:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/' ],

I think it is watching over the whole filesystem. From docker it thinks the current working directory is / and the default behaviour is the cwd without --watch set, so it watches over more the whole filesystem instead of a limited part of it.

As long as you give it much less to chew on with --watch=/somedir everything works fine.

pyper commented Jul 9, 2015

I think I may have found what issue is, and a simple workaround to correct it. I am using boot2docker and docker-compose for a multi container environment in OS X. When I add the --watch path to my command path in a docker-compose.yml file it works perfectly, and when I remove it I have the same issue as everyone here. Tested many times over.

I looked into it further I found something interesting. When I do a --dump with the --watch path set I get the following:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/src/code' ],

Now watch what happens without the --watch path:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/' ],

I think it is watching over the whole filesystem. From docker it thinks the current working directory is / and the default behaviour is the cwd without --watch set, so it watches over more the whole filesystem instead of a limited part of it.

As long as you give it much less to chew on with --watch=/somedir everything works fine.

@ryanbennettvoid

This comment has been minimized.

Show comment
Hide comment
@ryanbennettvoid

ryanbennettvoid Jul 14, 2015

@pyper You're a lifesaver. I've tried Supervisor, Nodemon, and Forever in Docker and they all performed terribly, if at all. Nodemon responded to the first few file changes, then stopped. Here's what I did to get it working:

nodemon --watch /app/server /app/server/server.js

@pyper You're a lifesaver. I've tried Supervisor, Nodemon, and Forever in Docker and they all performed terribly, if at all. Nodemon responded to the first few file changes, then stopped. Here's what I did to get it working:

nodemon --watch /app/server /app/server/server.js
@lox

This comment has been minimized.

Show comment
Hide comment
@lox

lox Jul 20, 2015

I don't think it's actually possible to "implement inotify". It's triggered via the Linux VFS, not the underlying filesystem.

lox commented Jul 20, 2015

I don't think it's actually possible to "implement inotify". It's triggered via the Linux VFS, not the underlying filesystem.

@aisipos

This comment has been minimized.

Show comment
Hide comment
@aisipos

aisipos Sep 18, 2015

I'm still seeing this on an NFS mounted share in boot2docker from an OS X host. I can verify that the host modified file produces a change in the container's "view" of the file, but no reload is triggered by nodemon. Which is a little odd, since the fallback of polling should in theory allow nodemon to see the changes. Is there a way to force recent versions of nodemon to use polling? The -L legacy flag is no longer here. I also tried @pyper 's method of explicitly setting --watch, but sadly it also isn't working for me.

aisipos commented Sep 18, 2015

I'm still seeing this on an NFS mounted share in boot2docker from an OS X host. I can verify that the host modified file produces a change in the container's "view" of the file, but no reload is triggered by nodemon. Which is a little odd, since the fallback of polling should in theory allow nodemon to see the changes. Is there a way to force recent versions of nodemon to use polling? The -L legacy flag is no longer here. I also tried @pyper 's method of explicitly setting --watch, but sadly it also isn't working for me.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Sep 18, 2015

Owner

@aisipos I think when this lands it'll fix the issue: #633 (if anyone wants to get on with a PR, otherwise it'll probably be next week...earliest).

Owner

remy commented Sep 18, 2015

@aisipos I think when this lands it'll fix the issue: #633 (if anyone wants to get on with a PR, otherwise it'll probably be next week...earliest).

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Sep 24, 2015

Owner

@chrishiestand have you tried the latest build with -L? 1.7.x now supports the polling mode. I've generally had positive feedback that this resolves the docker mounted volumes problem.

Owner

remy commented Sep 24, 2015

@chrishiestand have you tried the latest build with -L? 1.7.x now supports the polling mode. I've generally had positive feedback that this resolves the docker mounted volumes problem.

@aisipos

This comment has been minimized.

Show comment
Hide comment
@aisipos

aisipos Sep 24, 2015

I can confirm for myself at least using -L in 1.7.0 in a docker container is working for me. Without it, it doesn't work for me. (I'll note also, with some sadness, that while polling does work it does use a bit of CPU).

aisipos commented Sep 24, 2015

I can confirm for myself at least using -L in 1.7.0 in a docker container is working for me. Without it, it doesn't work for me. (I'll note also, with some sadness, that while polling does work it does use a bit of CPU).

@chrishiestand

This comment has been minimized.

Show comment
Hide comment
@chrishiestand

chrishiestand Sep 25, 2015

@remy Thanks for your care on this issue. I've changed my workflow and haven't used this feature in some time. If other people have reported success with -L that is good enough for me. Thanks!

@remy Thanks for your care on this issue. I've changed my workflow and haven't used this feature in some time. If other people have reported success with -L that is good enough for me. Thanks!

@pencilcheck

This comment has been minimized.

Show comment
Hide comment
@pencilcheck

pencilcheck Sep 26, 2015

-L works but the CPU will soar when run in boot2docker vm.

-L works but the CPU will soar when run in boot2docker vm.

@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy Sep 26, 2015

Owner

@pencilcheck you should also be wary of the number of files its watching.

Owner

remy commented Sep 26, 2015

@pencilcheck you should also be wary of the number of files its watching.

@d6u

This comment has been minimized.

Show comment
Hide comment
@d6u

d6u Sep 30, 2015

@remy -L works for me with docker-compose (previously boot2docker) version 1.4.2 on Node 0.10 with Nodemon 1.7.1

d6u commented Sep 30, 2015

@remy -L works for me with docker-compose (previously boot2docker) version 1.4.2 on Node 0.10 with Nodemon 1.7.1

@pencilcheck

This comment has been minimized.

Show comment
Hide comment
@pencilcheck

pencilcheck Sep 30, 2015

In a VM -L is still needed it seems but I have switched to a different van that setup NFS instead of vboxfs, now it runs much much quieter but unfortunately writing speed is not as responsive as native

In a VM -L is still needed it seems but I have switched to a different van that setup NFS instead of vboxfs, now it runs much much quieter but unfortunately writing speed is not as responsive as native

@oncletom

This comment has been minimized.

Show comment
Hide comment
@oncletom

oncletom Oct 7, 2015

-L works perfectly, thanks!

oncletom commented Oct 7, 2015

-L works perfectly, thanks!

@akselnordal

This comment has been minimized.

Show comment
Hide comment
@akselnordal

akselnordal Oct 15, 2015

Nothing of the above worked for me, so I ended up creating a small bash-script that starts my servers, restarts the pm2 instance inside docker from the host OS and tails docker logs. To avoid zombie processes, I'vce included a trap that kills the docker processes started by docker-compose

#!/bin/bash

function clean_up {
  docker-compose kill
  exit
}

trap clean_up SIGHUP SIGINT SIGTERM

docker-compose build
docker-compose start

# tail all logs 
docker-compose logs&

# fswatch is available for linux and osx.
fswatch -0 src | while read -d "" event
  do
    echo "file changed: ${event}"
    image=$(docker ps | grep <docker app-name> | cut -d " " -f1 )
    docker exec -t $image ./node_modules/.bin/pm2 restart <pm2 app-name>
  done

Nothing of the above worked for me, so I ended up creating a small bash-script that starts my servers, restarts the pm2 instance inside docker from the host OS and tails docker logs. To avoid zombie processes, I'vce included a trap that kills the docker processes started by docker-compose

#!/bin/bash

function clean_up {
  docker-compose kill
  exit
}

trap clean_up SIGHUP SIGINT SIGTERM

docker-compose build
docker-compose start

# tail all logs 
docker-compose logs&

# fswatch is available for linux and osx.
fswatch -0 src | while read -d "" event
  do
    echo "file changed: ${event}"
    image=$(docker ps | grep <docker app-name> | cut -d " " -f1 )
    docker exec -t $image ./node_modules/.bin/pm2 restart <pm2 app-name>
  done
@Omnomios

This comment has been minimized.

Show comment
Hide comment
@Omnomios

Omnomios Dec 17, 2015

I was chasing this issue for a while. Nodemon was catching the change no problem but node never received the signal to terminate. I was looking through the nodemon source and saw that there was an issue with child processes and as a result it was using psTree to get to the children.

The docker image I was using (fedora) didn't come with PS pre installed and so nodemon was using a fallback to pull the process that wasn't working for me.

Installing the 'procps' package in the dockerbuild got it working for me.

Hope this helps someone.

I was chasing this issue for a while. Nodemon was catching the change no problem but node never received the signal to terminate. I was looking through the nodemon source and saw that there was an issue with child processes and as a result it was using psTree to get to the children.

The docker image I was using (fedora) didn't come with PS pre installed and so nodemon was using a fallback to pull the process that wasn't working for me.

Installing the 'procps' package in the dockerbuild got it working for me.

Hope this helps someone.

@thalesfsp

This comment has been minimized.

Show comment
Hide comment
@thalesfsp

thalesfsp Dec 29, 2015

So, what's the final solution and how to, to get nodemon watching for changes in Mac OS X -> Boot2Docker -> Docker Container? It's possible or not? There's some documentation/README.md/Wiki? @remy

So, what's the final solution and how to, to get nodemon watching for changes in Mac OS X -> Boot2Docker -> Docker Container? It's possible or not? There's some documentation/README.md/Wiki? @remy

@pyper

This comment has been minimized.

Show comment
Hide comment
@pyper

pyper Dec 29, 2015

@thalesfsp It looks as though there is a few issues possibly causing this. @Omnomios has done some next level bug chasing and found something. For me, I simply had to explicitly set --watch.

pyper commented Dec 29, 2015

@thalesfsp It looks as though there is a few issues possibly causing this. @Omnomios has done some next level bug chasing and found something. For me, I simply had to explicitly set --watch.

@altieres

This comment has been minimized.

Show comment
Hide comment
@altieres

altieres Jan 3, 2016

The best way I have found to have filesystem notifications visible within the container was as follows:
1- Create two folders, one to map the project and a "mirror"
2- Map the project in the first folder
3- Keep a background script running in the container, rsyncing the project folder to "mirror"
4- Run the project from "mirror"

It may not be the most efficient or most elegant way, but this way was transparent to users of the container. No additional script needs to be run.
Not tested in a larger project, but in my case I did not realize performance issues.

https://github.com/altieres/docker-jekyll-s3

altieres commented Jan 3, 2016

The best way I have found to have filesystem notifications visible within the container was as follows:
1- Create two folders, one to map the project and a "mirror"
2- Map the project in the first folder
3- Keep a background script running in the container, rsyncing the project folder to "mirror"
4- Run the project from "mirror"

It may not be the most efficient or most elegant way, but this way was transparent to users of the container. No additional script needs to be run.
Not tested in a larger project, but in my case I did not realize performance issues.

https://github.com/altieres/docker-jekyll-s3

@nettofarah

This comment has been minimized.

Show comment
Hide comment
@nettofarah

nettofarah Jan 5, 2016

I have similar solution to @Omnomios.
I wrote this bash script

#!/bin/bash

image=$(docker ps | grep <container_name> | cut -d " " -f1 )
docker exec -t $image pm2 restart server

and hooked it up to webpack in my gulpfile

gulp.task('watch-backend', function() {
  webpack(backendConfig).watch(100, function(err, stats) {
    onBuild(err, stats)
    restartServer()
  })
})

var exec = require('child_process').exec

function restartServer() {
  exec('./restart-container.sh', function(err, stdout, stderr) {
    if (stdout) {
      console.log('stdout: ' + stdout)
    }

    if (stderr) {
      console.log('stderr: ' + stderr)
    }

    if (err !== null) {
      console.log('exec error: ' + err)
    }
  })
}

It has been working pretty well for me.
You have to run webpack (or whatever other watcher) in the host machine though.

I have similar solution to @Omnomios.
I wrote this bash script

#!/bin/bash

image=$(docker ps | grep <container_name> | cut -d " " -f1 )
docker exec -t $image pm2 restart server

and hooked it up to webpack in my gulpfile

gulp.task('watch-backend', function() {
  webpack(backendConfig).watch(100, function(err, stats) {
    onBuild(err, stats)
    restartServer()
  })
})

var exec = require('child_process').exec

function restartServer() {
  exec('./restart-container.sh', function(err, stdout, stderr) {
    if (stdout) {
      console.log('stdout: ' + stdout)
    }

    if (stderr) {
      console.log('stderr: ' + stderr)
    }

    if (err !== null) {
      console.log('exec error: ' + err)
    }
  })
}

It has been working pretty well for me.
You have to run webpack (or whatever other watcher) in the host machine though.

@darrinholst

This comment has been minimized.

Show comment
Hide comment
@darrinholst

darrinholst Jan 30, 2016

I cannot figure out how to get this working either. Shelling into the container and touching a file triggers nodemon...doing the same thing from osx doesn't do anything.

I cannot figure out how to get this working either. Shelling into the container and touching a file triggers nodemon...doing the same thing from osx doesn't do anything.

@darrinholst

This comment has been minimized.

Show comment
Hide comment
@darrinholst

darrinholst Jan 30, 2016

whoops...I missed the need for -L. Working once I added that.

whoops...I missed the need for -L. Working once I added that.

@kirkstrobeck

This comment has been minimized.

Show comment
Hide comment
@kirkstrobeck

kirkstrobeck May 2, 2016

-L ftw ! 👍

-L ftw ! 👍

@yordis

This comment has been minimized.

Show comment
Hide comment
@yordis

yordis Aug 7, 2016

I am using Docker for Mac and it doesn't work either but -L flag fixed it

yordis commented Aug 7, 2016

I am using Docker for Mac and it doesn't work either but -L flag fixed it

@mshick

This comment has been minimized.

Show comment
Hide comment
@mshick

mshick Mar 10, 2017

Contributor

Using a barebones security distro and was having this issue. @Omnomios solution installing procps worked for me.

Contributor

mshick commented Mar 10, 2017

Using a barebones security distro and was having this issue. @Omnomios solution installing procps worked for me.

@bradcosine

This comment has been minimized.

Show comment
Hide comment
@bradcosine

bradcosine Mar 30, 2017

Only -L fixed the issue for me

Only -L fixed the issue for me

@JCWolf

This comment has been minimized.

Show comment
Hide comment
@JCWolf

JCWolf May 22, 2018

Has anyone found a better solution, more official fix (maybe without any parameter)t o this -L parameter?

JCWolf commented May 22, 2018

Has anyone found a better solution, more official fix (maybe without any parameter)t o this -L parameter?

@mshick

This comment has been minimized.

Show comment
Hide comment
@mshick

mshick May 22, 2018

Contributor

@JCWolf have you tried the solution that worked for me? procps? I haven't been using legacy mode for over a year now and get automatic restarts.

Here's a snippet of my Dockerfile

FROM node:8.9.4-wheezy
RUN apt-get update
RUN apt-get install -y procps make gcc g++ python libpq-dev
Contributor

mshick commented May 22, 2018

@JCWolf have you tried the solution that worked for me? procps? I haven't been using legacy mode for over a year now and get automatic restarts.

Here's a snippet of my Dockerfile

FROM node:8.9.4-wheezy
RUN apt-get update
RUN apt-get install -y procps make gcc g++ python libpq-dev
@remy

This comment has been minimized.

Show comment
Hide comment
@remy

remy May 23, 2018

Owner

@mshick would you be able to add this to the FAQ? Sounds like you've solved a problem that many people run in to! Thanks in advance

Owner

remy commented May 23, 2018

@mshick would you be able to add this to the FAQ? Sounds like you've solved a problem that many people run in to! Thanks in advance

mshick added a commit to mshick/nodemon that referenced this issue May 23, 2018

Note about procps on Docker
Adding a note about installing procps on some Node.js Docker images to deal with long standing issue #419
@mshick

This comment has been minimized.

Show comment
Hide comment
@mshick

mshick May 23, 2018

Contributor

@remy Added a note. Credit goes to @Omnomios. Hope it helps!

Contributor

mshick commented May 23, 2018

@remy Added a note. Credit goes to @Omnomios. Hope it helps!

remy added a commit that referenced this issue May 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment