New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Simplify Docker Compose files #143
Comments
I agree 👍 Running the Docker environment was relatively easy because there is good documentation.. but figuring out how the Docker environment was configured to then customize it further was quite complicated. FWIW I started with this project template when building the farmOS Aggregator. We ended up consolidating the Docker Compose files similar to what you're describing (we also switched to an NGINX proxy & dropped some containers like celery & pgadmin):
This is working well so far. If you look closely I think they could be cleaned up a bit further... (notably the few extra |
Hi @tiangolo , I can first confirm that there is a mental step to cope with before getting the trick of the composition. I have 2 juniors in my team, and they still haven't fully mastered all the threads of the composition. But mostly because of the complexity of the docker-composition-world itself, when you think on all the layers you have to go through from the orchestrator / proxy / backend. Plus the scripts to build / push / deploy appropriately given environments variables. Hence my first feedback : it is not necessarily the number of docker-compose files which are an issue. My team actually feels they have a better grasp of the inner mechanisms with the current split layout (when they must dive into it). In order to simplify (help them), my choice has been to use a Makefile, which creates (yet another) abstraction layer, but that exposes the commands that they need for the release process. The makefile takes care of picking up the appropriate docker-compose files. We therefore have
but there are even more files than before 😇 And probably because of Makefile and .env files. We could have a python configuration along with python commands that would take care of this. I do not have a solution right now, but here is an idea for brainstorming : use some other tooling than |
For me, the deciding factor is that a range of people come to this project and some might be intimidated by the complexity. Anything that simplifies onboarding and makes the project more intuitive, without making things less useful, allows people to use these resources to do amazing things. At first, I liked the idea of keeping the compose files separate for the same reasons you mentioned, but onboarding to the project that way was definitely difficult, even with all the great documentation and support here. I ended up bringing the compose files together by environment and pipeline stage to get everything up and running the way I needed:
I could add a set of shared files too, but managing the duplication hasn't been a huge headache. That said, I want to manage frontend vs backend deployment and need to split the files again. Who knows where the splitting ends but it's definitely a lot easier to break the files out into separate ones vs putting them all together. I also wanted to comment on @ebreton because there are some excellent ideas there. I think a Makefile or a CLI to leverage the flexibility of separate or combined compose files would be amazing. Seems like a great place to field-test something like Click or better yet, the @tiangolo project Typer. A template file could auto-fill with the cookie-cutter fields and allow people to run something like Maybe a Makefile is the easiest way to do that for now, but a CLI or Makefile template as part of this project or as a demo for Typer would be a useful opportunity, regardless of the docker-compose file situation. |
Hi @tiangolo! Great job with FastAPI, thanks for that! I've just bootstrapped a project from this Cookiecutter and can't agree more that there are too many docker-compose files. How about using YAML anchors to avoid some of the duplication? |
Thanks for the feedback everyone! 🍰 Very interesting points here 🤓 ☕ I think I found a nice balance between simplicity, deduplication, flexibility, etc. There are now just 2 files:
The combination of these 2 files is normally the "default"/"standard" in Docker Compose, so, there's no need to add extra tricks defining which Docker Compose files to use and separator in the All the environment files are now in a single Some configs also read extra overrides or defaults from environment variables. Notably, the The tests are now run by the same container The generated README is also updated according to all the changes. Including the setup with Poetry, etc. Of course, the idea is just to keep it as simple and understandable as possible, so that it's easy to get started with, even for non-experts, but it should be easy to customize and add all the extra tools on top that each project might need. There's a new I finally found a way to simplify the local development of the project generator itself, so it should be easier to iterate on it. As a side note, I'll stop trying to keep the Full Stack FastAPI Couchbase project in sync and updated (more details on that project), which will allow me to focus the little time I have available to this generator for now. 🎉 |
Waaaoh. I didn't expect such an impact from the docker-compose optimisation. Dropping the backend-test container is a great news. I can't wait to use again the coverage tests 🎉 Thanks again @tiangolo , you do a tremendous job of maintening fastAPI and all its siblings. Bravo ! |
Thanks @ebreton ! 🚀 😄 I think my original concern is solved now, and there was no need for any duplication in the end 🎉 So I'm gonna close this issue now. ☕ |
Currently, there are several (I think too many) Docker Compose files.
I separate it that way to avoid all possible duplication of configurations for each environment (e.g. local development, testing, deployment to staging, deployment to production).
By reducing the config duplication that way it's possible to update a setting in a single file and then it works everywhere else.
That prevented having to keep all the configs in sync by hand. And it prevented spending time debugging "why it works on this environment but not on the other" just because one config was set in one file but not on the others.
But there are a lot of files.
I'm now thinking that there are just too many Docker Compose files.
I think it adds mental complexity, just trying to grasp the idea of what are the options and environments is already difficult.
It also means that there's not a single file that can be seen to have a general idea of the whole stack as it would be if each environment had its own single Docker Compose file with all the configurations.
Right now I'm considering merging all those Docker Compose files into a few, something like 3 or 4, at the expense of having duplication of configs in several of them and having to keep those configs in sync by hand.
What do you think?
@ebreton , @dmontagu , anyone else that sees this?
The text was updated successfully, but these errors were encountered: