New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make compose file allow to specify names for non-external volume #306

Merged
merged 2 commits into from Jul 27, 2017

Conversation

@lipingxue
Contributor

lipingxue commented Jul 6, 2017

- What I did
Fixes #274

- How I did it
I changed the config_schema.json to allow names for non-external volumes, and also made change in function convertVolumeToMount accordingly.
- How to verify it

  1. Try docker stack deploy with the following yaml file(specify customized volume name for non-external volume), and it works as expected
root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# cat /root/postgres2.yml 
version: "3.3"
 
services:
 
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    volumes:
      - postgres_vol:/var/lib/data
    environment:
      - "POSTGRES_PASSWORD=secretpass"
      - "PGDATA=/var/lib/data/db"
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.role == worker
volumes:
   postgres_vol:
      name: "postgres_vol@sharedVmfs-1"
      driver: "vsphere"
      driver_opts:
        size: "1GB"
root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# 


root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# ./docker-linux-amd64 stack deploy -c /root/postgres2.yml postgres
Creating network postgres_default
Creating service postgres_postgres
root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      postgres_vol@sharedVmfs-1

root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# docker service ps postgres_postgres 
ID            NAME                 IMAGE            NODE                       DESIRED STATE  CURRENT STATE           ERROR  PORTS
qx2nvqqwcqyu  postgres_postgres.1  postgres:latest  sc-rdops-vm02-dhcp-52-237  Running        Running 12 seconds ago        
  1. Try docker stack deploy with the following yaml file(specify volume name for external volume), and it returns with proper error
root@sc-rdops-vm02-dhcp-52-237:~# cat postgres_external.yml 
version: "3.3"
 
services:
 
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    volumes:
      - postgres_vol:/var/lib/data
    environment:
      - "POSTGRES_PASSWORD=secretpass"
      - "PGDATA=/var/lib/data/db"
    deploy:
      restart_policy:
        condition: on-failure
      placement:
        constraints:
          - node.role == worker
volumes:
   postgres_vol:
     external:
       name: "postgres@sharedVmfs-1"


root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# ./docker-linux-amd64 stack deploy -c /root/postgres_external.yml postgres
Creating network postgres_default
service postgres: cannot specify external volume name "postgres@sharedVmfs-1"

root@sc-rdops-vm02-dhcp-52-237:~/tmp_with_fix# docker stack ls
NAME  SERVICES

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)
image

@lipingxue lipingxue changed the title from New compose fix.liping to Make compose file allow to specify names for non-external volume Jul 6, 2017

@GordonTheTurtle GordonTheTurtle removed the dco/no label Jul 6, 2017

@dnephin

Thanks!

This looks like a good start. This change will need to be moved to a v3.4 version of the schema, but that doesn't exist yet, so I can take care of it later.

I think we should make this change to networks/configs/secrets as well, so they are all consistent. I can take care of that as well in another PR.

Show outdated Hide outdated cli/compose/convert/volume.go Outdated

@GordonTheTurtle GordonTheTurtle added dco/no and removed dco/no labels Jul 7, 2017

@codecov-io

This comment has been minimized.

Show comment
Hide comment
@codecov-io

codecov-io Jul 7, 2017

Codecov Report

Merging #306 into master will decrease coverage by 0.64%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #306      +/-   ##
==========================================
- Coverage   46.14%   45.49%   -0.65%     
==========================================
  Files         193      193              
  Lines       16073    16069       -4     
==========================================
- Hits         7417     7311     -106     
- Misses       8269     8380     +111     
+ Partials      387      378       -9

codecov-io commented Jul 7, 2017

Codecov Report

Merging #306 into master will decrease coverage by 0.64%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #306      +/-   ##
==========================================
- Coverage   46.14%   45.49%   -0.65%     
==========================================
  Files         193      193              
  Lines       16073    16069       -4     
==========================================
- Hits         7417     7311     -106     
- Misses       8269     8380     +111     
+ Partials      387      378       -9
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 7, 2017

Contributor

@dnephin I have addressed your comments and also have some questions for you. Please take a look. Thanks!

Contributor

lipingxue commented Jul 7, 2017

@dnephin I have addressed your comments and also have some questions for you. Please take a look. Thanks!

Show outdated Hide outdated cli/compose/loader/loader.go Outdated
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 14, 2017

Contributor

@dnephin I have addressed your comments, and please take a look. Thanks.

Contributor

lipingxue commented Jul 14, 2017

@dnephin I have addressed your comments, and please take a look. Thanks.

@dnephin

Thanks! I've pushed a commit to add a schema for 3.4 and rebased your commit on top of it.

I also adjusted the wording on the warning slightly, and made it an error to set both names.

I think now we just need to add to the full test in loader_test.go and a test in convert/volume_test.go.

@docker docker deleted a comment from GordonTheTurtle Jul 17, 2017

@vdemeester vdemeester self-requested a review Jul 17, 2017

@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 17, 2017

Contributor

@dnephin Thanks. I will look at loader_test.go and convert/volume_test.go to figure out how to add the test.

Contributor

lipingxue commented Jul 17, 2017

@dnephin Thanks. I will look at loader_test.go and convert/volume_test.go to figure out how to add the test.

@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 18, 2017

Contributor

@dnephin I have added unit test and repushed the commit. Please review it. Thanks!

Contributor

lipingxue commented Jul 18, 2017

@dnephin I have added unit test and repushed the commit. Please review it. Thanks!

@dnephin

Thanks! the tests look good, but we lost the changes I had made to move the schema to v3.4.

You can rebase on #358 to get the v3.4 changes and move the scheme change to that file.

Show outdated Hide outdated cli/compose/loader/loader.go Outdated
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 19, 2017

Contributor

@dnephin I cannot find your repo dnephin:compose-3.4. What is the URL for that repo? What is the easiest way to rebase the change introduced in #358? Thanks!

Contributor

lipingxue commented Jul 19, 2017

@dnephin I cannot find your repo dnephin:compose-3.4. What is the URL for that repo? What is the easiest way to rebase the change introduced in #358? Thanks!

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jul 19, 2017

Collaborator

https://github.com/dnephin/cli/tree/compose-3.4

You can do this (assuming you have docker/cli added as a remote called docker, otherwise change the word docker in the first line to the name of the remote):

# checkout my pr branch as the local branch called pr-compose-3.4
git fetch docker pull/358/head:pr-compose-3.4

# rebase
git rebase  pr-compose-3.4
Collaborator

dnephin commented Jul 19, 2017

https://github.com/dnephin/cli/tree/compose-3.4

You can do this (assuming you have docker/cli added as a remote called docker, otherwise change the word docker in the first line to the name of the remote):

# checkout my pr branch as the local branch called pr-compose-3.4
git fetch docker pull/358/head:pr-compose-3.4

# rebase
git rebase  pr-compose-3.4

@lipingxue lipingxue requested review from mistyhacks and thaJeztah as code owners Jul 19, 2017

Add Compose format 3.4
Signed-off-by: Daniel Nephin <dnephin@docker.com>
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 19, 2017

Contributor

@dnephin I have rebased to get v3.4 schema and addressed your comments. Please review it. Thanks!

Contributor

lipingxue commented Jul 19, 2017

@dnephin I have rebased to get v3.4 schema and addressed your comments. Please review it. Thanks!

@thaJeztah

Thanks @lipingxue - this is great! I left some comments

@dnephin for consistency; should the same change be applied to networks, configs, and secrets? (i.e., network.name instead of network.external.name)?

# Values can be strings or numbers
foo: "bar"
baz: 1
external-volume:
# Specifies that a pre-existing volume called "external-volume"
# can be referred to within this file as "external-volume"

This comment has been minimized.

@thaJeztah

thaJeztah Jul 20, 2017

Member

for other-external-volume below, should we update the comment to mention volume.external.name is deprecated? (can't comment on that line, so leaving the comment here 😄)

# This example uses the deprecated "volume.external.name" (replaced by "volume.name")
@thaJeztah

thaJeztah Jul 20, 2017

Member

for other-external-volume below, should we update the comment to mention volume.external.name is deprecated? (can't comment on that line, so leaving the comment here 😄)

# This example uses the deprecated "volume.external.name" (replaced by "volume.name")

This comment has been minimized.

@lipingxue

lipingxue Jul 21, 2017

Contributor

Done. Add the comment.

@lipingxue

lipingxue Jul 21, 2017

Contributor

Done. Add the comment.

# Values can be strings or numbers
foo: "bar"
baz: 1

This comment has been minimized.

@thaJeztah

thaJeztah Jul 20, 2017

Member

I'd like to see an example using a name and external, for example;

  external-volume3:
    name: this-is-volume3
    external: true
@thaJeztah

thaJeztah Jul 20, 2017

Member

I'd like to see an example using a name and external, for example;

  external-volume3:
    name: this-is-volume3
    external: true

This comment has been minimized.

@thaJeztah

thaJeztah Jul 20, 2017

Member

Also (not sure if that needs to be done in this file, or another one); can we have a test that tests if setting both volume.name and volume.external.name produces the "conflicting options" error? i.e.;

  external-volume4-fail:
    # name cannot be provided both as volume.name and volume.external.name
    name: custom-name
    external:
      name: conflicting-name
@thaJeztah

thaJeztah Jul 20, 2017

Member

Also (not sure if that needs to be done in this file, or another one); can we have a test that tests if setting both volume.name and volume.external.name produces the "conflicting options" error? i.e.;

  external-volume4-fail:
    # name cannot be provided both as volume.name and volume.external.name
    name: custom-name
    external:
      name: conflicting-name

This comment has been minimized.

@dnephin

dnephin Jul 21, 2017

Collaborator

A separate test case for testing the conflict would be great

@dnephin

dnephin Jul 21, 2017

Collaborator

A separate test case for testing the conflict would be great

This comment has been minimized.

@lipingxue

lipingxue Jul 21, 2017

Contributor

Done. Add a test case in for that.

@lipingxue

lipingxue Jul 21, 2017

Contributor

Done. Add a test case in for that.

@@ -305,6 +305,7 @@ type IPAMPool struct {
// VolumeConfig for a volume
type VolumeConfig struct {
Name string

This comment has been minimized.

@thaJeztah
@thaJeztah

thaJeztah Jul 20, 2017

Member

This comment has been minimized.

@dnephin

dnephin Jul 21, 2017

Collaborator

That link isn't going to any specific line. I don't think we should add it to the lists in types.go, those are for options being deprecated/removed from v2->v3.

In this case maybe just a comment in the struct would be fine

@dnephin

dnephin Jul 21, 2017

Collaborator

That link isn't going to any specific line. I don't think we should add it to the lists in types.go, those are for options being deprecated/removed from v2->v3.

In this case maybe just a comment in the struct would be fine

This comment has been minimized.

@thaJeztah

thaJeztah Jul 21, 2017

Member

Updating the docs, and mentioning that could be ok?

@thaJeztah

thaJeztah Jul 21, 2017

Member

Updating the docs, and mentioning that could be ok?

This comment has been minimized.

@lipingxue

lipingxue Jul 24, 2017

Contributor

Done. Add a comment in the struct.

@lipingxue

lipingxue Jul 24, 2017

Contributor

Done. Add a comment in the struct.

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jul 21, 2017

Collaborator

for consistency; should the same change be applied to networks, configs, and secrets?

Yes, but I was going to handle it after this PR merges. I think we need to refactor these objects to share more code.

Collaborator

dnephin commented Jul 21, 2017

for consistency; should the same change be applied to networks, configs, and secrets?

Yes, but I was going to handle it after this PR merges. I think we need to refactor these objects to share more code.

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Jul 21, 2017

Collaborator

hmm, webhooks didn't trigger on this PR either. This is becoming a problem

Collaborator

dnephin commented Jul 21, 2017

hmm, webhooks didn't trigger on this PR either. This is becoming a problem

@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 21, 2017

Contributor

@thaJeztah I have addressed your comments, and please review it.

Contributor

lipingxue commented Jul 21, 2017

@thaJeztah I have addressed your comments, and please review it.

Show outdated Hide outdated cli/compose/loader/loader.go Outdated
Show outdated Hide outdated cli/compose/loader/loader.go Outdated
Show outdated Hide outdated cli/compose/loader/loader_test.go Outdated
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 26, 2017

Contributor

@mstanleyjones I have addressed your comments, and please take a look.

Contributor

lipingxue commented Jul 26, 2017

@mstanleyjones I have addressed your comments, and please take a look.

@dnephin

LGTM

@thaJeztah

LGTM, but can you squash your commits ? Thinking of one commit for the compose-file 3.4 bump, and one commit for the other changes

Change to enable volume name can be customized.
Signed-off-by: Liping Xue <lipingxue@gmail.com>
Change to enable volume name can be customized.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Change to enable volume name can be customized.

Remove unused debug info.

Address comments from Daniel and solve the lint error.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address Daniel's comments to print warning message when name of external volume is set in loader code.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address Daniel's comments to return error when external volume is set in loader code.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address Daniel's comments to return error when external volume is set in loader code.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Remove the case that specifying external volume name in full-example.yml.

More fix.

Add unit test.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address comments from Daniel, move the schema change to v3.4.
Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address comments from Sebastiaan. Signed-off-by: Liping Xue <lipingxue@gmail.com>

Address comments from Misty.
Signed-off-by: Liping Xue <lipingxue@gmail.com>
@lipingxue

This comment has been minimized.

Show comment
Hide comment
@lipingxue

lipingxue Jul 27, 2017

Contributor

@thaJeztah I have squashed the comments, please review it. Thanks!

Contributor

lipingxue commented Jul 27, 2017

@thaJeztah I have squashed the comments, please review it. Thanks!

@thaJeztah

LGTM, thanks!

This change needs an update to the documentation; probably in the vnext-compose branch; https://github.com/docker/docker.github.io/tree/vnext-compose/compose/compose-file

but ping @londoncalling for confirmation

also ping @shin- for the implementation in docker-compose

@thaJeztah thaJeztah added this to the 17.08.0 milestone Jul 27, 2017

@thaJeztah thaJeztah merged commit 7524912 into docker:master Jul 27, 2017

8 checks passed

ci/circleci: cross Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: shellcheck Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
ci/circleci: validate Your tests passed on CircleCI!
Details
codecov/patch 100% of diff hit (target 50%)
Details
codecov/project Absolute coverage decreased by -0.64% but relative coverage increased by +53.85% compared to 2dac00b
Details
dco-signed All commits are signed

@shin- shin- referenced this pull request Jul 27, 2017

Closed

[v3.4] Volume name field #5059

@hairyhenderson

This comment has been minimized.

Show comment
Hide comment
@hairyhenderson

hairyhenderson Jul 27, 2017

Contributor

This is awesome! Thanks @lipingxue 🎉

Contributor

hairyhenderson commented Jul 27, 2017

This is awesome! Thanks @lipingxue 🎉

@kcrawley

This comment has been minimized.

Show comment
Hide comment
@kcrawley

kcrawley Sep 28, 2017

@dnephin is there an issue open regarding?

networks, configs, and secrets

kcrawley commented Sep 28, 2017

@dnephin is there an issue open regarding?

networks, configs, and secrets

@dnephin

This comment has been minimized.

Show comment
Hide comment
@dnephin

dnephin Sep 28, 2017

Collaborator

I don't think there are any open issues for the other objects

Collaborator

dnephin commented Sep 28, 2017

I don't think there are any open issues for the other objects

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment