New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: including external docker-compose.yml #318

Open
dnephin opened this Issue Jul 14, 2014 · 48 comments

Comments

Projects
None yet
@dnephin
Contributor

dnephin commented Jul 14, 2014

The Problem

I'd like to use fig for acceptance testing of services

In the case of a single service with a single level of dependencies this is easy (ex: a webapp with a database). As soon as the dependency tree grows to depth > 1, it gets more complicated (ex: service-a requires service-b which requires a database). Currently I'd have to specify the service dependency tree in each fig.yml. This is not ideal because as the tree grows, we end up with a lot of duplication, and having to update a few different fig.yml's in different projects when a service adds/changes a dependency is not great.

Proposed Solution

Support an include section in the fig.yml which contains url/paths to other fig.yml files. These files would be included in a Project so that the top-level fig.yaml can refer to them using <proejct>_<service>.

Example config

include:
    # link using http
    servicea:
        http: http://localhost:8080/projects/servicea/fig.yml
    # local paths
    serviceb:
        path: ../projects/serviceb/fig.yml

webapp:
    ...
    links:
        - servicea_webapp
        - serviceb_webapp
@dnephin

This comment has been minimized.

Contributor

dnephin commented Jul 14, 2014

I'm looking to implement this features myself, but I would like to upstream it eventually, so I'm interested to see how you feel about this idea.

I've looked over the code, and it seems to be relatively easy to make this work.

Some unresolved issues are:

  • including services which use image should be easy enough, but services with build would need to ensure that the naming remains consistent, and that the service was already built (and possibly pushed to a registry). See next two issues
  • in the case of remote repos, the config may need to support a repo name as well (as part of the includes section).
  • project naming. Instead of using the file path/directory for a project, it uses the key from the parent yaml config (the file with the include section). This poses a problem if the names don't match. I was considering adding a project_name key to the config to keep these consistent (this is not really critical)

For fetching the external configs. I was going to use requests since it's already a dependency, but I was also considering supporting git as another method, possibly using dulwich

@dnephin

This comment has been minimized.

Contributor

dnephin commented Aug 11, 2014

I have a working prototype with a single integration test in https://github.com/dnephin/fig/compare/include_remote_config

It still needs some cleanup and more tests, but I thought I'd link to the current progress.

@bfirsh

This comment has been minimized.

Contributor

bfirsh commented Aug 13, 2014

Nice!

@suzaku

This comment has been minimized.

Contributor

suzaku commented Aug 22, 2014

@dnephin @bfirsh +1 for the new feature.

I'm looking for a way to reuse fig.yml from another project,
before something like include is available,
I'll have to fallback to using copy/paste/edit ;(

@pirelenito

This comment has been minimized.

pirelenito commented Sep 12, 2014

That is awesome! I'd love to get this feature as well. Any updates on merging this on master @dnephin ?

@dnephin

This comment has been minimized.

Contributor

dnephin commented Sep 12, 2014

Cool, glad you are interested! I haven't had much time to finish this off yet. I still plan on doing it, but it might still be a week or two out. I've been thinking it actually probably requires #457 to deal with including any services that use build instead of image (because you wont be able to build them without the whole project).

The idea would be to use one of the tags as the image, and assume the other project has properly pushed the images and they are available to be pulled (or already cached locally).

@pirelenito

This comment has been minimized.

pirelenito commented Sep 12, 2014

Nice!

For a remote or standalone fig.yml file your solution of using tags seems pretty reasonable.

What are your thoughts if the fig.yml file is available locally inside its own project (meaning everything would be there)? Could the build be triggered? It seems to make sense.

@dnephin

This comment has been minimized.

Contributor

dnephin commented Sep 13, 2014

I haven't thought much about using it with locally available fig.yml. In that kind of setup would you have different services in subdirectories and you want to include them so that each individual one is smaller?

I think defaulting to use the build if no tags are provided is probably reasonable. It might need to do something with relative paths (append the path given in the include section to any build paths in the included fig.yml).

@dnephin

This comment has been minimized.

Contributor

dnephin commented Sep 30, 2014

https://github.com/dnephin/fig/compare/include_remote_config is mostly ready. There are two TODOs to resolve:

  • convert remote build services to use the first tag
  • support reading hdfs and s3 urls for remotes (currently supports http/https, and local files)
@ThomasSteinbach

This comment has been minimized.

ThomasSteinbach commented Oct 2, 2014

+1 - absolutely nescessary feature

It should be possible, to extend the imports. For instance a service included should get 'volumes_from' a data volume container.

Thus you can configure smaller orchestrations close to the services (e.g. gitlabService=gitlab+redis+postgres) and combine those smaller orchestration to a bigger one (whole orchestration = gitlabService + liferayService + jenkinsService). In the final fig.yml you can connect the imports to each other.

@rudibatt

This comment has been minimized.

rudibatt commented Dec 2, 2014

+1 for more reusable configuration

@clifton

This comment has been minimized.

clifton commented Dec 9, 2014

This would be extremely useful for my team and I 👍

@hadronzoo

This comment has been minimized.

hadronzoo commented Dec 9, 2014

👍 for include

@manuwell

This comment has been minimized.

manuwell commented Dec 15, 2014

+1

@rseymour

This comment has been minimized.

rseymour commented Dec 17, 2014

extremely useful +1 as well over here.

@rseymour

This comment has been minimized.

rseymour commented Dec 18, 2014

Playing with this, it seems to solve the initial problem, but not the exact problem I'm facing. My issue is wanting to fire up app X then at a later time link app Y (w/ it's own fig) to it. With this proposed solution, the second fig up will bring down the first app and restart it.

That fits with the initial problem, but ideally this could check to see if a set of containers from the included fig.yml was already up. In that case it could just do the link, /etc/hosts and ENV vars changes to link to the already up containers.

@dnephin

This comment has been minimized.

Contributor

dnephin commented Dec 18, 2014

@rseymour Thanks for trying it out!

I wonder if fig up --no-recreate will accomplish this?

@rseymour

This comment has been minimized.

rseymour commented Dec 18, 2014

🏆 --no-recreate totally worked @dnephin !

@dnephin

This comment has been minimized.

Contributor

dnephin commented Dec 26, 2014

PR is up at #758 is anyone is interesting in giving some feedback

@DCameronMauch

This comment has been minimized.

DCameronMauch commented Apr 12, 2016

Did anything ever happen with this? If not, will something happen? We are having to do some ugly things with multiple "-f" and relative path stuff.

@dnephin

This comment has been minimized.

Contributor

dnephin commented Apr 12, 2016

I build some things external to Compose in https://github.com/dnephin/compose-addons that act as pre/post processors for a Compose file. They probably need to be updated for the v2 format.

@proudcommerce

This comment has been minimized.

proudcommerce commented Jul 23, 2016

+1

@tjamet

This comment has been minimized.

tjamet commented Aug 11, 2016

@dnephin I like the idea of dcao include, do you think there is any chance to get it a part of compose itself?

@dnephin

This comment has been minimized.

Contributor

dnephin commented Aug 11, 2016

We have not made any progress integrating the ideas into Compose

@tjamet

This comment has been minimized.

tjamet commented Aug 17, 2016

Is there any way we can help doing so? or shall we use the add-ons?

@nharward-hd

This comment has been minimized.

nharward-hd commented Aug 19, 2016

Simplified example of a situation I've run into at multiple companies that I wish docker-compose could support and could with includes:

my_top_level_project_dir/docker-compose.yml
   db/docker-compose.yml
   middle-tier/docker-compose.yml
   front-end/docker-compose.yml
   another-front-end/docker-compose.yml

Each team/developer responsible for an area should be able to change to their appropriate subdir and type docker-compose up and be done with it. The middle-tier would include the db layer, and each front end would include the middle tier definitions. The top-level directory would start everything, by including each of the front-ends. We have a more complex setup with more components, but hopefully the idea gets across.

In this way each team can work [far] more efficiently by only using/defining the components they actually need. By allowing linking/depends_on for includes you can have a DRY system where each component can be effectively managed/versioned without the headache of managing multiple files, or multiple -f options on the command line.

@sgreenslade

This comment has been minimized.

sgreenslade commented Oct 10, 2016

+1 for Include

@ludekvodicka

This comment has been minimized.

ludekvodicka commented Jan 10, 2017

Any news about this?

@dnephin dnephin changed the title from feature: including external fig.yml to feature: including external docker-compose.yml Feb 16, 2017

@jvanbruegge

This comment has been minimized.

jvanbruegge commented Mar 9, 2017

I really like the way the syntax works with dcao-include. Sadly it does not work with V3.

@jahbini

This comment has been minimized.

jahbini commented Jul 19, 2017

Has anyone suggested just using the c-preprocessor parse a yml file just like a c.header file? It would allow simple (left-justified) includes. The ability to use it's macro capabilities might be useful too.

@pwFoo

This comment has been minimized.

pwFoo commented Nov 11, 2017

Also looking for a solution to use external docker-compose.yml by http/s url.

@rakshazi

This comment has been minimized.

rakshazi commented Apr 5, 2018

Hello!
Any updates on this?

@dosmanak

This comment has been minimized.

dosmanak commented Jun 4, 2018

This feature would really help us since we use a lot docker-compose -f docker-compose-common.yml -f docker-compose-local.yml and it sucks to write it down all the time.

@ludekvodicka

This comment has been minimized.

ludekvodicka commented Jun 4, 2018

As @dosmanak mentioned. This would be very appreciated feature.

@Papipo

This comment has been minimized.

Papipo commented Jun 4, 2018

This is easily fixed by creating single-line shell scripts.

@ludekvodicka

This comment has been minimized.

ludekvodicka commented Jun 4, 2018

@Papipo: Everything can be fixed by shell scripts.

We have Windows and Linux developer machines so we have to maintain two different scripts for each project. Instead of including one simple line in the docker-compose file.

@Papipo

This comment has been minimized.

Papipo commented Jun 6, 2018

@ludekvodicka that's a good point :)

@jvanbruegge

This comment has been minimized.

jvanbruegge commented Jun 6, 2018

I also want this for dependency reasons. I want to be able to have a service depend on a recursive service compound.
I have a reverse proxy that needs to be started after a few others, but not necessarily at last. All the services it depends on are defined in docker-compose files in subdirectories. It would be cool if I could do it like this:

services:
     reverse-proxy:
         #more fields here
         depends_on:
             - logger
             - ./my-recusive-service:service1

    logger:
        #more here

So when depends_on gets a path, it waits for service1 defined in ./my-recursive-service/docker-compose.yaml. This service1 may have other dependencies that get started first (e.g. mysql in case of wordpress)

When I have time I could try to implement that

@ET-CS

This comment has been minimized.

ET-CS commented Jul 8, 2018

@jvanbruegge Be aware that depends_on doe's not wait for mysql to be "ready" thus should not be used in order to control startup order.

https://docs.docker.com/compose/compose-file/#depends_on

https://docs.docker.com/compose/startup-order/

@sarink

This comment has been minimized.

sarink commented Sep 3, 2018

Having to type out

docker-compose -f docker-compose-common.yml -f docker-compose-local.yml <some command>

500 times a day is just the worst....

Why in the world is this still not implemented? ("include" is fundamentally different from yaml merges/anchors, so those are not a substitute)

@gaborcsardi

This comment has been minimized.

gaborcsardi commented Oct 17, 2018

Here is my workaround, in case anyone finds this useful, or can suggest a better way. I have a top level docker-compose.yml file with the base config, that is good for local development and deployment. I create subdirectories for the other deployment types, testing, prog, etc. E.g. I have a digitalocean type to deploy on a DO host. The digitalocean directory has a docker-compose-custom.yml file which contains the overrides I need:

project
+- docker-compose.yml
+- digitalocean
   +- docker-compose-custom.yml

E.g. digitalocean/docker-compose-custom.yml has

version: '3.3'

services:
   frontend:
     ports:
     - "80:3000"
     environment:
     - RHUB_BUILDER_EXTERNAL_URL=http://...

to specify the IP of the DO host, and port forwarding to the container.

I wrote a simple script called dc that wraps docker-compose and sets up the COMPOSE_FILE environment variable based on the current working directory. I use dc instead of docker-compose everywhere. E.g. if dc is invoked from within digitalocean, then it makes sure that both the base docker-compose.yml and the docker-compose-custom.yml file are used.

Here is the dc script:

#! /bin/bash

here=$(basename $(pwd))

# TODO: could do better here, e.g. find the root automagically
if [ -f docker-compose.yml ]; then
    root="."
else
    root=".."
fi

export COMPOSE_FILE="docker-compose.yml"
if [ -f docker-compose-custom.yml ]; then
    export COMPOSE_PATH_SEPARATOR=":"
    export COMPOSE_FILE="${COMPOSE_FILE}:${here}/docker-compose-custom.yml"
fi

(
    cd ${root};
    docker-compose "$@"
)

Btw. docker-compose config does not seem to use $COMPOSE_FILE, so unfortunately one cannot use that to double check the config. The script could be improved, but at least I have a reasonable way now to "import" a compose file in another one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment