Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bind mount exclusion to prevent overwriting generated files with local files #6470

Open
steevehook opened this issue Jan 17, 2019 · 6 comments

Comments

@steevehook
Copy link

steevehook commented Jan 17, 2019

It's always annoying when I want to create a bind mount in order to synchronize local source code I'm editing, with code from docker container and all of the generated files.

Whether it's vendor directory or a binary file which eventually is used as an ENTRYPOINT or CMD they are getting removed when the bind mount is done. And since these files/directories are not available on local machine to avoid copying different binaries for different OS & ARCH inside linux container, docker removes them inside container as well when binding the source from local with the one from container

A solution to this might be to to do the bind mount first then use the snapshot fs which was created when the image was build, which will first bind mount the sources then allow the scripts which generate the binaries/directories to execute without being removed

Another solution would be to indicate in the volumes section some sort of exclude list when binding mount for the first time, somethings that would look like this:

volumes:
      - .:/usr/app:
          exclude:
            - fileA
            - fileB
            - dirA/

As a workaround to avoid this situation with volumes instead of running the generated binary in CMD or ENTRYPOINT we run some sort of shell script which generates again the binary and then executes the binary and now the CMD/ENTRYPOINT is referencing the shell script.
However would be a nice option with for initial bind mount some files which we choose would not be overwritten/removed in order to sync with local source code

docker-compose.yaml

version: "3.7"

services:
  redis:
    image: redis
    restart: always
    volumes:
      - redis-data:/data
    networks:
      - visits
    command: redis-server --appendonly yes
  visits-app:
    build: .
    restart: always
    ports:
      - "8080:8080"
    volumes:
      - ./:/usr/app
    networks:
      - visits
    

    # if this custom command is missing then we get an error that there is no node_modules
    # but in Dockerfile we indeed genereated node_modules by running npm install
    command: sh /usr/app/init.sh

networks:
  visits:

volumes:
  redis-data:

init.sh

npm install
npm start

Dockerfile

FROM node:alpine

ENV PORT 8080

EXPOSE $PORT

WORKDIR /usr/app

COPY ./package.json ./
RUN npm install
COPY . .

CMD npm start

Thank you for the awesome project and let me know if such a feature is a valid one and does not come against docker philosophy or if you need any help in implementing it

@ulyssessouza
Copy link
Collaborator

ulyssessouza commented Jan 21, 2019

Thanks for the suggestion @steevehook !

One thing you can do is restructuring your source tree so that you can mount (in docker-compose.yml:18) just important part which is, lets say a src folder. Leading to something like ./:/usr/app/src in the volume mounting line.

I think implementing volume exclusions could lead to awkward corner cases.

If the mounting is for a performance issue then, looking at your Dockerfile, if you use buildkit your image layers should be cached until the RUN npm install.

@montanaflynn
Copy link

montanaflynn commented Apr 13, 2021

The biggest problem that this would alleviate is for node_modules.

There are countless stackoverflow threads and blog posts about this, here's a few:

https://stackoverflow.com/questions/29181032/add-a-volume-to-docker-but-exclude-a-sub-folder
https://burnedikt.com/dockerized-node-development-and-mounting-node-volumes/

The basic gist is that when doing local development you want to have the files except for node modules shared between host and container. The docker image already installs the dependencies for the linux version, which is then overwritten with either an empty node_modules or in some cases node_modules for a different OS like windows or mac:

  example:
    build: example
    volumes:
      - ./example:/usr/app
    ports:
      - "8080:3000"

The common workaround is a named volume:

volumes:
  example_node_modules:

services:
  example:
    build: example
    volumes:
      - ./example:/usr/app
      - example_node_modules:/usr/app/node_modules/
    ports:
      - "8080:3000"

The big downside is then you have some extra volume to deal with and remember to remove when updating dependencies.

@adamkdean
Copy link

adamkdean commented Oct 26, 2021

This would be a great feature to add, something we've lacked for almost a decade now. There was a workaround with symlinks and a second container but, sadly, as of 2020 that has also stopped working.

@malachid
Copy link

malachid commented Dec 15, 2021

I'd second this. We are currently mounting our local work directory into the docker container so we can do some validations before pushing them up to Jenkins. Unfortunately, there is 1 file that has to be different locally vs in the container (local.properties) and as such we have had to resort to using the entrypoint to symlink every file/directory once mounted then remove that entry. That was working fine until AGP (Android Gradle) 7 which no longer works correctly with symlinks. As such, I'm once again looking for a workaround to the missing exclude file option.

@pcjmfranken
Copy link

pcjmfranken commented Apr 12, 2022

There seem to be many posts and comments that are essentially requesting this exact same feature, describe a problem that this feature would solve, or they're running into problems with the commonly suggested 'submount' workaround. All these discussions are using different wording/terminology, and they're scattered across multiple Docker repositories, the legacy Docker forum, StackOverflow, and various other places.

Just a few in random order:

The ability to (dynamically) exclude specific bind mount contents (like any and all node_modules or cache directories) from existing on the host filesystem would be very nice for DevContainer workflows. There are too many caching/performance/permission problems with the common workaround, plus having to manage all the exclusions in the composefile is such a hassle and on top of that very error-prone.

Maybe someone from the Docker team could create a new Docker Roadmap item so we have a central place for reference and discussion?

@ndeloof
Copy link
Collaborator

ndeloof commented Apr 12, 2022

bind mount is managed by docker engine, not compose. And those are actually Linux features to define a mount for container-visible filesystem to access some parts of host's filesystem. We only expose the available lower-level features. Linux mounts does not offer anything like "exclusions" we could rely on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants