New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: make project-name persistent. #745

Open
thaJeztah opened this Issue Dec 21, 2014 · 232 comments

Comments

Projects
None yet
@thaJeztah
Copy link
Member

thaJeztah commented Dec 21, 2014

By default, Compose bases the project name on basename of the directory compose
commands are run from. The project name can be overridden either by
passing a -p / --project-name option for each command or setting the
COMPOSE_PROJECT_NAME environment variable.

Using the --project-name option for each command is error-prone and for-
getting to pass the option when doing (for example) docker-compose up, will result in
another instance of the project being created under a different name or even
containers of a different project being replaced.

Using the COMPOSE_PROJECT_NAME environment variable is not useful when dealing
with multiple projects and can cause similar problems.

Proposal: make project-name persistent

To solve these problems, compose will save the name of a project to a file in the
build-context when the project is first started / built.

If a project-file is found, compose automatically uses the project-name from that
file and throw an exception if the user is trying to override the project-name
using the --project-name option.

File location

To allow further expansion of options, compose creates a hidden .docker-compose directory
in the "root" directory of the build-context. Inside that directory, a
project-name file is created, containing just the name of the project;

tree -a
.
├── .docker-compose
│   └── project-name
└── docker-compose.yml

Request for comment

There are a couple of things left to be discussed;

  • Should the configuration-file be created automatically or should this
    be an explicit action by the user (e.g. docker-compose init --project-name=foobar)
  • Should a command be added to remove the configuration file(s)? (e.g.
    docker-compose destroy)
  • Compose allows specifying an alternative compose-file (--file) should the
    project-name also be applied to those? Should each compose-file get its own
    configuration?

And, in a wider scope;

  • Is the project-name actually important? If not, compose could create a random
    project-name and store that inside the configuration. This would greatly reduce
    the risk of conflicting names with other projects running on the same server.

edit: renamed Fig to Compose

@bfirsh

This comment has been minimized.

Copy link
Contributor

bfirsh commented Dec 22, 2014

The project name isn't important. I like the idea of a random project name. Even better – could we generate a hash of the path to fig.yml and use that?

@dnephin

This comment has been minimized.

Copy link
Contributor

dnephin commented Dec 22, 2014

I actually think the project name is pretty important. If you want to be able to do anything with the images in a CI pipeline, you need to know the project name to be able to retag them as something else.

Being able to override project name from an environment variables makes it easy to keep these unique, and predictable. I think no matter what happens, we need to support the override from environment variables.

I do like the idea of making the project name configurable from a file. I think a similar discussion happened in (#45). Having the project name in.fig/project-name seems like it will lead to a lot of really small files. I think it would be easier to just put it into the fig.yml itself (like it is suggested in #45, and one of the docker compose proposals).

What are the advantages of putting it in a separate directory and file?

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Dec 22, 2014

The reason for putting it in a separate file has a number of reasons, I'll try to explains my thinking behind this;

  • To preserve the name that was used to start the project (i.e., --project-name=foobar), which can be different from the automatic project-name.
  • Having multiple instances or versions of a project running on the same server, without them conflicting each other
  • Or, in short, the .fig/project-name holds the name of that instance of the project.

Perhaps the file should be called instance-name or similar.

I do like the idea of making the project name configurable from a file

Although possible to do this with this proposal (by manually creating the project-name file), my primary use case was to have fig create the file in order to save the name that was used.

will lead to a lot of really small files.

True, the alternative is to skip the .fig directory, however, if additional things are needed in future, extra file(s) can be added without cluttering the root of your project. Git does it similar and I think it's a clean approach.

I think it would be easier to just put it into the fig.yml

I think both options can be complementary; use the name from fig.yml as the default if the name is not overridden by a flag or env-variable. Put the name that is actually used to start the project inside the .fig/project-name

Being able to override project name from an environment variables makes it easy to keep these unique, and predictable. I think no matter what happens, we need to support the override from environment variables.

Curious; how do you handle managing multiple projects? I.e. cd project-a && fig ps then cd project-b fig <something>?

Thanks for your feedback!

@dnephin

This comment has been minimized.

Copy link
Contributor

dnephin commented Dec 22, 2014

Curious; how do you handle managing multiple projects?

I usually run fig from python-tox, which lets me set the environment, so I'm never really setting it in my shell. I also tend to use tmux, so I have a different shells for every project.

I think both options can be complementary

Cool, I agree.

I don't know if I'm sold on .fig/instance-name vs say .fig-instance.yml that could contain any instance specific overrides, but I think this is more of a minor point.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Dec 22, 2014

Thinking of the project-name; If I understand your situation correctly, it's important to be able to have control over the images being produced, e.g. myapp:build12345, or also names of the containers? Just to get a "feel" for your situation.

My thoughts behind "random" project-names was that, as long as Fig is able to locate the containers for a project, nice-looking names aren't important. If I, as a user, can access a services' container via its service-name (e.g. fig stop web), the name of the actual container doesn't matter.

Even have been thinking that fig could keep track of containers by ID in a local storage inside the .fig directory (which could be a performance gain on machines with a lot of running containers). But that's really something for a separate issue.

Any thoughts on "implicitly" creating the 'project-name' file or "explicitly" via fig init? Perhaps if I have some time during the coming holidays I can experiment a bit (never coded anything in Python, so it will be really experimental :))

@dnephin

This comment has been minimized.

Copy link
Contributor

dnephin commented Dec 23, 2014

it's important to be able to have control over the images being produced, e.g. myapp:build12345, or also names of the containers?

I guess image is the really important one, but being able to ensure that container names don't conflict is also really important on shared hosts. I see this proposal actually aims to deal with that issue as well.

But I do think that predictable names for containers are still important. I can think of a couple tasks where it's important for external systems (not just fig) to be able to identify a container by name:

  • cleanup scripts to remove old containers, being able to identify what project created a container
  • debugging/inspecting containers

Both of these are pretty easy right now because I already know what the container name will be. If I have to lookup a name it's somewhat more involved.

Even have been thinking that fig could keep track of containers by ID in a local storage inside the .fig directory

It is true this could be a performance gain, but I worry this is the wrong approach. Adding any persistent state to fig introduces the possibility of lots of bugs (that come along with state). I would much rather see this being handled by a labeling system in dockerd. If fig is able to just label containers and then query for all containers with that label , it keeps all the state in a single place (dockerd). I know there was talk of this at some point, I'm not sure if it's on the roadmap.

Any thoughts on "implicitly" creating the 'project-name' file or "explicitly" via fig init?

I guess implicitly would be more backwards compatible. I would like to avoid a fig init unless absolutely necessary. Instead of using the basedir, I could see it implicitly creating a project name of <basename>-<4 digit random id>

@frank-dspeed

This comment has been minimized.

Copy link

frank-dspeed commented Jan 10, 2015

i can only say if fig starts generating random names now i will switch away from it because i run 1000+ containers on a single host and when i cant identify them by name any more its nothing for me.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 10, 2015

@frank-dspeed thanks for adding your use-case. If I understand correctly, you start your containers with fig, but "manage" them directly via Docker after that, making use of the naming convention that Fig uses?

Perhaps this is of interest to you as well: moby/moby#9882 - having meta-data would offer more ways to identify containers, not only their name.

@frank-dspeed

This comment has been minimized.

Copy link

frank-dspeed commented Jan 10, 2015

ah yes i did such proposals a lot for example simply adding new filds and that but i ended in that case with simply adding a ENV and then i have that as costum fild i can parse it 🎯

@funkyfuture

This comment has been minimized.

Copy link
Contributor

funkyfuture commented Jan 17, 2015

I think managing the unambiguity of the project name shouldn't be fig's responsibility.

when i understood your proposal correctly, the generation of a random name would render the FIG_PROJECT_VARIABLE and the --project-name-option useless. which is handy for 'templating' services.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Jan 17, 2015

I think managing the unambiguity of the project name shouldn't be fig's responsibility.

I'm not so sure; Fig "silently" overwriting another projects' containers because if happened to be in a directory with the same name is sometimes nasty.

when i understood your proposal correctly, the generation of a random name would render the FIG_PROJECT_VARIABLE and the --project-name-option useless. which is handy for 'templating' services.

Following the discussion above, I think something like this would work

  • If --project-name is provided, use that (and write/update the .project-name? Not sure)
  • No --project-name is provided, but FIG_PROJECT_NAME is, use FIG_PROJECT_NAME (and write/update to .project-name?)
  • None of the above, but .project-name is set, use .project-name
  • None of the above; create a random name and write it to .project-name
@funkyfuture

This comment has been minimized.

Copy link
Contributor

funkyfuture commented Jan 17, 2015

from someone's perspective who uses fig in scripts that take care of a name and pass it to fig, storing a name in the project (which is in my case rather a template) location makes no sense.

and still, i sensed it very intuitive that the directory name is used in the name of the resulting containers when i just ran fig up.
and let say, you look at docker ps -a, that won't be very helpful if it's filled up with random names.

would an option --random-name help in your situation?
beside that i think the ability to prepend some [a-zA-Z0-9_].* to reflect some namespacing would be useful for broader deployments.

@jderusse

This comment has been minimized.

Copy link

jderusse commented Feb 12, 2015

What's about using the fig.yml to store such informations ?

schema-version: 1.1
project_name: foo
containers:
  web:
    build: .
   command: python app.py
    links:
     - db
    ports:
     - "8000:8000"
  db:
    image: postgres

And to keep BC, in compose/project.py::from_config

@classmethod
    def from_config(cls, name, config, client):
        if 'schema-version' not in config:
            config = {
                'schema-version': '1.0',
                'containers': config
            }
        dicts = []
        for service_name, service in list(config.containers.items()):
            if not isinstance(service, dict):
                raise ConfigurationError('Service "%s" doesn\'t have any configuration options. All top level keys in your docker-compose.yml must map to a dictionary of configuration options.' % service_name)
            service['name'] = service_name
            dicts.append(service)
        return cls.from_dicts(name, dicts, client)
@mikehaertl

This comment has been minimized.

Copy link

mikehaertl commented Mar 31, 2015

@jderusse I'm very much for something like you suggested. Maybe you like to add your proposal to #45 as this is probably the better place to discuss it?

@michaeljs1990

This comment has been minimized.

Copy link

michaeljs1990 commented Sep 23, 2015

anyway to come to resolution on this ticket? It looks like the proposed #1233 gives you the best of both worlds by letting the user decide the intended action to take while not breaking any backwards compatibility.

@dnephin

This comment has been minimized.

Copy link
Contributor

dnephin commented Oct 9, 2015

Originally I wanted the name in the docker-compose.yml, but after thinking about it more, I really like the idea of a separate file.

A separate file give you the option to keep it out of version control if you want (for cases where you want each user to be able to have a different project name).

With the new networking support there is feature request to be able to configure the network name as well (it defaults to the project name). So with a separate config file we could include things like default network name, default scale etc. All the meta-configuration that isn't part of the application definition, but is still relevant to running the composition.

@mikehaertl

This comment has been minimized.

Copy link

mikehaertl commented Oct 9, 2015

I agree to what dnephin said. How about a file .docker-compose? We could list default options for all command line arguments there.

@thaJeztah

This comment has been minimized.

Copy link
Member

thaJeztah commented Oct 9, 2015

@dnephin sgtm

@mikehaertl I used a .docker-compose folder in my example, but a file could work as well, depending on how much information we want to store

@mikehaertl

This comment has been minimized.

Copy link

mikehaertl commented Oct 9, 2015

@thaJeztah Oh, right, sorry. I missed that.

lig referenced this issue in grammarly/rocker Mar 12, 2018

@askoretskiy

This comment has been minimized.

Copy link

askoretskiy commented Mar 19, 2018

Storing project name in docker-compose.yml file makes much sense when used with other tools, e.g. PyCharm.

It would be great to have store it in the same YAML file but possibly override if needed.

@flaviovs

This comment has been minimized.

Copy link

flaviovs commented Apr 8, 2018

I think that the core of this issue come from the fact that the default project name is simply (should I say naively?) taken as the base name of the current directory. This is effectively compressing the file system name space (hierarchical) to a flat one, which is problematic because of name conflicts. For example, this causes issues for people running services in ~/fooproject/master and ~/barproject/master.

I do think that project names are unimportant. So perhaps a way to make docker-compose robust is to generate project names bases on the full path (i.e. home_flaviovs_fooproject_master)?

I like the persistent idea, however. To be able to commit the project name to Git is such a common use case. Perhaps reading a .env.default file before .env is a simple solution for this (which BTW would take care of the long standing #210 as well)?

@helmus

This comment has been minimized.

Copy link

helmus commented Apr 18, 2018

I solved this by wrapping docker-compose binary with my own function so I can call the function from anywhere. This loads the .env file and uses the right docker-compose file anywhere.

Obviously you could extend this to have a function per project or something. I think there might be drawbacks to it but I haven't found them out.

DEFAULT_DOCKER_COMPOSE=~/my-project

function dcompose() {
    pushd $DEFAULT_DOCKER_COMPOSE &> /dev/null;
    /usr/local/bin/docker-compose $@;
    popd &> /dev/null;
}
## maintain auto complete
function _dc {
    pushd $DEFAULT_DOCKER_COMPOSE &> /dev/null;
    _docker_compose;
    popd &> /dev/null;
}

complete -F _dc dcompose
@vedmant

This comment has been minimized.

Copy link

vedmant commented May 10, 2018

I have a problem with images collision, and it looks really weird that I can not set project_name in the docker-compose.yml as it looks very logical and strait forward solution. Actually taking name from the project folder in the first place was a bad idea. In many cases projects folders can be like /home/project1/repository, /home/project2/repository and so on, in this case docker generated the same images names like repository_app.

@schmunk42

This comment has been minimized.

Copy link
Contributor

schmunk42 commented May 10, 2018

@vedmant You can set the image name in docker-compose.yml, just use image and build.

@bscheshirwork

This comment has been minimized.

Copy link

bscheshirwork commented May 29, 2018

So... How about separated by projects on same-named folder now?

docker-compose -f /foo/bar up -d
docker-compose -f /foo/foo/foo/bar down
@glensc

This comment has been minimized.

Copy link

glensc commented Jul 6, 2018

Perhaps this was already proposed. but why not make a breaking change in version: 4 and add some new keyword to allow setting project name from docker-compose.yml?

@MyJoiT

This comment has been minimized.

Copy link

MyJoiT commented Jul 31, 2018

my .docker-compose/project-name did not work ...
--project-name worked

@dazinator

This comment has been minimized.

Copy link

dazinator commented Aug 7, 2018

I would like to set project name in the docker-compose.yml also.

My use case is, I have a docker-compose.yml file which has a web app and a postgress database service.
I also want seperate docker-compose.yml files for adhoc tasks - basically describing adhoc tasks that I'd like to orchestarte with docker-compose. These adhoc tasks need to leverage the same same services / networks / volumes as used in the main project.

An example of an adhoc task that I'd like to orchestrate with docker-compose might be. migrating data into the postgress database. Its useful to orchestrate this with docker-compose because the task might involve spinning up additional services and volumes, networks, etc, alongside the current ones, in order to fulfill that task.

So in order to achieve this, I can create a seperate docker-compose.adhoctask.yml file to represent the task, so that I can run that task on demand only when required. Because it's in the same directory as the original docker-compose file, it gets the same project name by default and so it has access to the same networks, volumes etc? and stuff works.

However the problem comes when I don't want this adhoc task yml file to live in the same top level directory (or even the same repo) as the main docker-compose.yml. For example, perhaps I want to put the adhoc task docker-compose.adhoc.yml in a subfolder, with the DOCKERFILE and assets that the task needs nicely isolated / grouped together as it's own concern. As soon as I move it to a subfolder, it's project name no longer matches. This means it no longer has access to the same volumes / networks etc defined in the main project.

If I could specify the projectname in the docker-compose.yml and docker-compose.adhoc.yml - then I could structure them however I want (even put them in seperate repo's) and still execute them without having to constantly specify additional command line arguments or switch environment variables around.

@fprochazka

This comment has been minimized.

Copy link

fprochazka commented Aug 13, 2018

This is a real problem for me, because it causes naming conflicts

I have several projects in the structure

/company
    /ab   # project prefix
        /backend   # actual project
        /webapp
    /cd
        /backend
        /webapp

now when I start a docker-compose in one project, it creates network backend_default and prefixes containers that are not explicitly named with backend_ .. now If I want to start the other project, I have to go back to the right folder and first run docker-compose down to prevent conflicts - otherwise it would start new containers in the same network, and then when turning them off, it would complain about orhans, when there are none.

@simplesmiler

This comment has been minimized.

Copy link

simplesmiler commented Aug 24, 2018

Also an issue for me. I have multiple projects, and use docker-compose for dynamic demo stands on a single machine, a stand per branch.

/project-1
    /dev # docker-compose.yml inside
    /master # docker-compose.yml inside
    /feature
        /new-feature # docker-compose.yml inside
/project-2
    /dev # docker-compose.yml inside
    /master # docker-compose.yml inside

So "masters" and "devs" are conflicting between between projects. A .docker-compose file could solve that.

@shin-

This comment has been minimized.

Copy link
Contributor

shin- commented Aug 24, 2018

@simplesmiler At the risk of repeating myself, a .env solves that already.

@thedeeno

This comment has been minimized.

Copy link

thedeeno commented Aug 24, 2018

@shin- .env is often used for secrets not in version control.

What @simplesmiler wants is a way to commit the project name to version control while keeping .env outside version control.

As far as I can tell there is still no way to do this easily.

@shin-

This comment has been minimized.

Copy link
Contributor

shin- commented Aug 24, 2018

@thedeeno that's not what their post say. Are you working together? If not, we shouldn't presume what someone else's use case is beyond what's explicitly written.

@thedeeno

This comment has been minimized.

Copy link

thedeeno commented Aug 24, 2018

We're not working together. That's my interpretation of their problem. Totally fair point, I might be wrong and I might be presuming.

I wish WE worked together though so we could grab some coffee. I hope you don't think I'm being a jerk here.

Maybe I'm just remembering this thread incorrectly: Is there a way to commit the project name to version control while keeping .env ignored?

@shin-

This comment has been minimized.

Copy link
Contributor

shin- commented Aug 24, 2018

Is there a way to commit the project name to version control while keeping .env ignored? Or is this still a missing capability?

Not currently. As I mentioned a few months ago^1, the plan is to have docker-compose.env as a secondary env file that can be included in version control. It has slipped down the list of priorities somewhat but I hope to get to it soon.

1/ #745 (comment)

@thedeeno

This comment has been minimized.

Copy link

thedeeno commented Aug 24, 2018

That's great news! Accepting PR's?

@mikehaertl

This comment has been minimized.

Copy link

mikehaertl commented Aug 25, 2018

I gave up hope long ago but let's give it another try:

I think many here still fail to understand why we are forced to maintain a second file for this setting.

As I understand it, for some few here it breaks their project setup. But for the majority in this thread it would not be a problem. So how can it be that those few impose this restriction to all the others who are missing this feature? Why can't we have several options for how to configure the name and then leave the decision to us?

It's not even logically making any sense: The lack of this setting in docker-compose.yml is so obvious that I couldn't even believe that this option is missing when I looked for it in the manual. Almost each and every other aspect of your setup can be configured there. To me that's such a clear break in the configuration logic.

@schmunk42

This comment has been minimized.

Copy link
Contributor

schmunk42 commented Aug 28, 2018

I think that one could say, (almost) all settings in docker-compose.yml are "scoped" by the project name.

@roipoussiere

This comment has been minimized.

Copy link

roipoussiere commented Aug 28, 2018

As suggested in #4841, an option to specify which .env file to use (ie --env-file) should be very useful.

@schmunk42

This comment has been minimized.

Copy link
Contributor

schmunk42 commented Aug 28, 2018

@roipoussiere

This comment has been minimized.

Copy link

roipoussiere commented Aug 28, 2018

Thanks @schmunk42.
But that means I must create a hierarchy like:

.
├── project_a
│   ├── .env
├── project_b
│   ├── .env

instead of something easier like:

.
├── a.env
├── b.env
@schmunk42

This comment has been minimized.

Copy link
Contributor

schmunk42 commented Aug 28, 2018

Yes, see also #745 (comment) - you might need to open that in a new window, since it's sometimes hidden here 🤓

@nmaro

This comment has been minimized.

Copy link

nmaro commented Sep 6, 2018

I second the --env-file proposal. Made a ticket just for that. Go and like it if you want this option.

@arvera

This comment has been minimized.

Copy link

arvera commented Dec 19, 2018

So I only read a few comments on this bug/feature, and the fact that discussion continues and that it is open brings me to the conclusion that it has been 4 years since this was originally opened and there is still no solution? Can we make a partial decision and then move forward..?

@nickjj

This comment has been minimized.

Copy link

nickjj commented Dec 30, 2018

I'll admit, I haven't read this entire issue and all of the related issues but there are interesting problems around this topic that makes this more complicated than it seems at first glance. For example, as we all know .env is commonly used for secret values, so you typically ignore that from version control.

But let's say you're working with Stripe, which happens to allow you to have test keys and live keys, AKA. 2 sets of keys to work with. In development it would be very common for you to use the test keys, and in production you would end up using the live keys.

That automatically means you can't use .env by itself to deal with secret values because you'll have different keys in each environment, and you don't want to have to switch the values in your file between running your app in dev or prod (that would be crazy and super error prone).

What you likely end up doing is creating both a .env and .env.prod file and ignore both from version control. This way you can have your test / dev keys in development and then reference that with env_file in your development compose file.

But then in production you reference both .env and .env.prod in your production compose file.

But do you see where this is going? Right now Docker Compose doesn't let you have optional env_file files. If the file doesn't exist, then Docker Compose is going to blow up as soon as you try to run Docker Compose.

So you automatically need a .env file to exist in the end if you plan to use it with env_file and this isn't limited to Stripe either. A lot of web frameworks have certain settings that are environment specific but would change in both development and production (ie. SERVER_NAME with Flask).

So that means we're limited to having something like a .env_example file that you commit to version control and it's expected the end user renames that file to .env, and then inside of this file would contain our old friend COMPOSE_PROJECT_NAME.

Which now brings up the question. Do we actually need an alternative way of setting the project name? I'm not convinced that we do, but I am convinced setting a custom project name is a good idea in any project, or at least something other than the folder name since you can easily run into name conflicts.

@landsman

This comment has been minimized.

Copy link

landsman commented Jan 6, 2019

Is this issue on some road map please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment