Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Should a docker-compose.override.yml file be standalone? #9

Open
ezzy1337 opened this issue Feb 21, 2018 · 2 comments

Comments

@ezzy1337
Copy link

@ezzy1337 ezzy1337 commented Feb 21, 2018

When using docker-compose.dev.yml file to override your base docker-compose.yml file. i.e.

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

Should the docker-compose.dev.yml file be a standalone compose file or is it preferred to only specify the values that need to be overridden?

example

Given these 2 compose files.

# docker-compose.yml
  redis:
    image: redis:4.0.8
    volumes:
      - ${REDIS_VOLUME}:/data
    expose:
      - "6379"
#docker-compose.dev.yml
  redis:
    user: "1000:50"
    ports:
      - "6379:6379"

The command below works just as you would expect.

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

The following commands will also work.

docker-compose up
docker-compose -f docker-compose.yml up

This command however fails because the redis service does not specify a value for context or image one of which is required so docker knows what to do in the event there isn't an image already built.

docker-compose -f docker-compose.dev.yml up

So should I write every compose file so it could be standalone or should I let override compose files be dependent on the base compose file it?

Considerations thus far

  1. If I write each compose file to be standalone I end up duplicating settings...and developers hate that.
  2. If override compose files should be dependent on their base compose file what should I do in the case where I set up a service specifically for dev but I don't want that running in production (Thinking of a situation like a database which in production may not be a docker container)? If I don't include that service in my docker-compose.yml file I can get in a state where docker-compose down does not stop all the containers it should stop. The only way I would be able to stop them all is remembering to specify '-f docker-compose.dev.yml' when calling docker-compose down. Seeing as how I have to specify -f docker-compose.dev.yml when spinning up the environment it seems reasonable I should have to specify -f docker-compse.dev.yml when tearing down the environment but I'd like your opinion on it.
@BretFisher

This comment has been minimized.

Copy link
Owner

@BretFisher BretFisher commented Feb 22, 2018

Hi Derek,

TL;DR

Q. Should a docker-compose.override.yml file be standalone?

A. No. Then it wouldn't be an override. 🥇


docker-compose up is optimized for local development by humans

Requiring anything else then that to fully setup a dev environment (even if it's 20 containers with seeded data and k/v stores) is likely unnecessary friction for developers, and is much worse in a team or open source where you don't want to add daily commands or extra typing for people. So that means your docker-compose.yml should have the goal of being just for dev.

docker-compose up always looks for docker-compose.override.yml

If you have a use-case where you can't get around custom compose settings per developer, then give a sample file like docker-compose.sample.override.yml that they can copy into a docker-compose.override.yml on their local machine which will automatically be used by docker-compose due to it's special name for local dev. Make sure docker-compose.override.yml is in .gitignore so their custom local settings aren't committed. That override should only have differences in it, and NOT be standalone, hence the term override.

test/prod are usually automated, so let them have manual overrides or dupes if needed

Now, for test and production, IMO they are secondary to this goal of better-dev-workflow. Almost all test environments are automated in CI/CD, and increasingly so in production... so if there's going to be duplication and complexity, let that be in those environments. It'll change far less than local dev.

overrides can't remove yaml values, and can't change block sequences (lists)

A good example is databases and local "cloud service emulators". Often a dev will need something like a k/v store or db running that are not needed in production compose files because those environments are using a cloud service. Compose files and overrides can't remove things from base compose files, so I know of no way to tell a "production override" file to not deploy the mysql from the dev compose file, or to remove a published port only needed for troubleshooting...

small projects/teams can keep it simple

For small simple solutions that are running everything in a Docker Swarm, you may be able to get away with CI/CD and prod using something like this one-liner to combine multiple compose files and pass the output to a stack deploy command for Swarm docker-compose -f docker-compose.yml -f docker-compose.prod.yml config | docker stack deploy -c - mystack

everyone is special

In the end every project I've worked on is different. Some always use a simple docker-compose.yml file, then the docker-compose.override.yml is always expected to be in use by developers (from a sample file provided) that has local db and k/v setups included. That override is in .gitignore to prevent committing. Then other environments are in the same repo as overrides with proper file names... but all use same docker-compose.yml.

Other projects are too complex and use a whole separate repo for storing CI/CD and prod yaml.

Some projects have many repo's used by different dev's for one large solution and with dozens of microservices they end up with various configs that can't be as easy as a single file/command.

no one is special

But I've never seen anyone get in trouble if they start with your goals and these in mind:

  1. K.I.S.S. for local dev. If something has to give way to complexity or duplication, let it be for CI/CD and prod, which are automated, and don't usually require typing the same commands over and over.
  2. Use the docker-compose.override.yml special naming to your advantage in docker-compose cli. Any file can be used to override, but that one file is always looked for on up/down commands.
  3. There are lots of hidden gems in docker-compose. It can use separate environment files per service, including the default of .env (same as Laravel) that is pulled into compose services for envvars. You can use YAML template features like variable subsutition and extension fields to make things more flexible in a compose file. Lastly, it can use envvars set on the host to set the default compose files and more.

Between all those options you can usually find ways to solve your problems so that local developers are always docker-compose up and still have the flexability you need for test/prod.

Does that answer your questions?

@ezzy1337

This comment has been minimized.

Copy link
Author

@ezzy1337 ezzy1337 commented Feb 22, 2018

As always the depth of your explanations is amazing. This absolutely answers my question. I think the key phrase for my situation was, "So that means your docker-compose.yml should have the goal of being just for dev". That ideology shift will simplify the whole process and address the situation I was talking about. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.