New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker volumes (bind mounts) pointing to a concrete file aren't updated when the file changes #4797
Comments
Looks similar to #4770 |
I've stumbled upon a user with the exact same issue: https://groups.google.com/forum/#!topic/nomad-tool/s9cw-GeFF8M |
I had this issue, mounting the directory rather than the file directly resolved it. |
Seems that consul template copies the file into place rather than editing in place. |
The same problem.
file don't update ( |
I am also experiencing this issue. Current workaround is to build my own container layer that symlinks from |
@mud5150 it's not really solvable. It's described in the issue that was linked to in an earlier comment. Basically when you do a bind mount, the directory or file has this thing called an inode which points to the file/directory. If you make a change to this file or directory, some software may replace the file causing a new inode to appear. For a single file, the filepath that was mounted, points to the original file inode, but on the host system, your filepath may now point to a new inode, but the docker container is only aware of the inode of the original file. Some text editors have different settings to avoid the changing of an inode, thus it's not an issue with Docker or Nomad, you must take care of it on your end. Mounting a directory instead should work, as the inode is now being watched on the directory, which the software like text editors don't tamper with, thus notifications of updates to it's contents works properly. |
@polarathene I'm not sure you're right in saying this isn't a Nomad issue. See my original comment:
Also as others have pointed out it could be an issue with consul-template. Either way I think this issue should remain open for investigation. Especially since it's something quite a few folks have run into. |
@rkettelerij I'm not actually a Nomad or Consul user (I am aware of them and might check them out sometime in future) I just noticed this issue on the thread/issue that the comment I mentioned linked to. Your comparing the echo update to another method updating the file, it's not the same. Verify the files inode for both types of modification and I bet you'll have a difference in behavior. If the inode remains the same, it'll update and work as expected by anything watching it. Alternatively, mount a directory with the file instead, and you should also note that any updates to the file should now behave as you expect them to? If so that'll confirm that what I chimed in to share is correct. |
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
I believe this is still the case. Re-open. |
Hey there Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this. Thanks! |
This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍 |
Still affected |
This is not a bug. It is an unfortunate interaction between two foundational design choices in two different products. Both of them are performing as designed.
The combination of these two factors is what prevents consul-template and docker single file mounts from working like people expect. The workaround from Nomad and Docker's perspective is to use directory mounts in these cases. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.8.3 (c85483d)
Operating system and Environment details
Ubuntu 16.04
Docker version 18.03.0-ce, build 0520e24
Issue
If you create a file using the
template
stanza and map this to a location in a Docker container then the contents of the file in Docker are never updated, even though Nomad (through the template stanza) has rendered a new file. So basically you're dealing with stale files in Docker.Reproduction steps
template
stanza in the job that reads something from an external source (like Consul or Vault). Set thechange_mode
tosignal
and chooseSIGHUP
as the signal.volumes = [ "local/foo.txt:/var/foo.txt" ]
.docker exec -it <name> bash
and observe the contents of/var/foo.txt
/var/lib/nomad/alloc/.../.../foo.txt
)/var/foo.txt
hasn't changed! << this is the bug/local/foo.txt
has changed, the default mappings (alloc/, local/, secrets/) are always correctly updated.One way around this bug to use
restart
as the template'schange_mode
since this kills the docker container and therefore always creates a new volume mount. But this causes downtime. Since I'm using the template to render a Nginx config I'd rather not kill the Docker container but prefer to sent a SIGHUP to trigger a reload in Nginx.Note that this is a Nomad issue not a Docker issue. When I manually create a Docker container and map a concrete file to a location in the container (e.g.
docker run -d --name devtest --mount type=bind,source=/home/myname/foo/foo.conf,target=/etc/foo/foo.conf nginx:mainline
) and update the file (e.g.echo 'test' >> /home/myname/foo/foo.conf
) the changes are always reflected in the container.Job file (if appropriate)
The text was updated successfully, but these errors were encountered: