Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set Default values correctly for docker variables #42641

Merged
merged 8 commits into from
Aug 29, 2018
Merged

Set Default values correctly for docker variables #42641

merged 8 commits into from
Aug 29, 2018

Conversation

denraf
Copy link
Contributor

@denraf denraf commented Jul 11, 2018

SUMMARY

Environment variables were no longer respected as the default is set twice.

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

docker common utils

ANSIBLE VERSION

ansible 2.6.0
config file = None
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible
python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

ADDITIONAL INFORMATION

env:
export DOCKER_TLS_VERIFY=1
export DOCKER_HOST=tcp://192.168.99.20:2376
export DOCKER_CERT_PATH=~/.docker/certs
export DOCKER_DEFAULT_IP=192.168.99.20
export DOCKER_TLS_HOSTNAME="docker-dev"

  docker_container:
    name: "{{ item.name }}"
    docker_host: "{{ item.docker_host | default('unix://var/run/docker.sock') }}"
    hostname: "{{ item.name }}"
    image: "molecule_local/{{ item.image }}"
    state: started
    recreate: false
    log_driver: json-file
    command: "{{ item.command | default('bash -c \"while true; do sleep 10000; done\"') }}"
    privileged: "{{ item.privileged | default(omit) }}"
    security_opts: "{{ item.security_opts | default(omit) }}"
    volumes: "{{ item.volumes | default(omit) }}"
    tmpfs: "{{ item.tmpfs | default(omit) }}"
    capabilities: "{{ item.capabilities | default(omit) }}"
    exposed_ports: "{{ item.exposed_ports | default(omit) }}"
    published_ports: "{{ item.published_ports | default(omit) }}"
    ulimits: "{{ item.ulimits | default(omit) }}"
    networks: "{{ item.networks | default(omit) }}"
    dns_servers: "{{ item.dns_servers | default(omit) }}"
  register: server
  with_items: "{{ molecule_yml.platforms }}"
  async: 7200
  poll: 0

Before:
"msg": "Error connecting: Error while fetching server API version: ('Connection aborted.', BadStatusLine('\x15\x03\x01\x00\x02\x02\n',))"

After:
changed: [localhost] => (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label': {'exposed_ports': [u'22/tcp', u'22/udp', u'80/tcp', u'443/tcp'], 'dns_servers': [u'10.86.2.1'], 'name': u'haproxy1', 'image': u'dev-redhat-base:latest', 'hostname': u'haproxy1', 'capabilities': [u'NET_ADMIN'], 'docker_host': u'tcp://192.168.99.20:2376', 'command': u'/sbin/init', 'registry': {'url': u'192.168.99.12:6559'}, 'groups': [u'haproxy'], 'volumes': [u'/sys/fs/cgroup:/sys/fs/cgroup:ro'], 'tmpfs': [u'/tmp', u'/run']}, u'ansible_job_id': u'525370450142.2904', 'failed': False, u'started': 1, 'changed': True, 'item': {'exposed_ports': [u'22/tcp', u'22/udp', u'80/tcp', u'443/tcp'], 'dns_servers': [u'192.168.99.1'], 'name': u'haproxy1', 'image': u'dev-redhat-base:latest', 'hostname': u'haproxy1', 'capabilities': [u'NET_ADMIN'], 'docker_host': u'tcp://192.168.99.20:2376', 'command': u'/sbin/init', 'registry': {'url': u'192.168.99.12:6559'}, 'groups': [u'haproxy'], 'volumes': [u'/sys/fs/cgroup:/sys/fs/cgroup:ro'], 'tmpfs': [u'/tmp', u'/run']}, u'finished': 0, u'results_file': u'/root/.ansible_async/525370450142.2904', '_ansible_ignore_errors': None, '_ansible_no_log': False}) => { }
changed: [localhost] => {
"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result",
"changed": true
}
META: ran handlers

@ansibot ansibot added affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. needs_triage Needs a first human triage before being processed. support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jul 11, 2018
@ansibot
Copy link
Contributor

ansibot commented Jul 11, 2018

The test ansible-test sanity --test validate-modules [explain] failed with 44 errors:

lib/ansible/modules/cloud/docker/docker_container.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_container.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_container.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_container.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_container.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_image.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_image.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_image.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_image.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_image.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_image_facts.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_image_facts.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_image_facts.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_image_facts.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_image_facts.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_login.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_login.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_login.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_login.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_login.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_network.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_network.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_network.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_network.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_network.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_secret.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_secret.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_secret.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_secret.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_secret.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_service.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_service.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_service.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_service.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_swarm.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_swarm.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_swarm.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_swarm.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_swarm.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')
lib/ansible/modules/cloud/docker/docker_volume.py:0:0: E324 Value for "default" from the argument_spec (None) for "api_version" does not match the documentation ('auto')
lib/ansible/modules/cloud/docker/docker_volume.py:0:0: E324 Value for "default" from the argument_spec (None) for "docker_host" does not match the documentation ('unix://var/run/docker.sock')
lib/ansible/modules/cloud/docker/docker_volume.py:0:0: E324 Value for "default" from the argument_spec (None) for "ssl_version" does not match the documentation ('1.0')
lib/ansible/modules/cloud/docker/docker_volume.py:0:0: E324 Value for "default" from the argument_spec (None) for "timeout" does not match the documentation (60)
lib/ansible/modules/cloud/docker/docker_volume.py:0:0: E324 Value for "default" from the argument_spec (None) for "tls_hostname" does not match the documentation ('localhost')

click here for bot help

@ansibot ansibot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Jul 11, 2018
@jborean93 jborean93 removed the needs_triage Needs a first human triage before being processed. label Jul 12, 2018
@mattclay mattclay added the ci_verified Changes made in this PR are causing tests to fail. label Jul 16, 2018
@ansibot ansibot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Jul 25, 2018
@felixfontein
Copy link
Contributor

Thanks for pointing this problem out! There is a much better solution, though:

from ansible.module_utils.basic import env_fallback

[...]
DOCKER_COMMON_ARGS = dict(
    docker_host=dict(type='str', aliases=['docker_url'], default=DEFAULT_DOCKER_HOST, fallback=(env_fallback, 'DOCKER_TLS_HOSTNAME')),
[...]

This will take care of extracting the value from the environment variable if it is defined and the parameter is not specified.

(To see more examples, grep for fallback= in lib/ansible/modules.)

Copy link
Contributor

@felixfontein felixfontein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost done, I think :)

docker_host=dict(type='str', aliases=['docker_url'], default=DEFAULT_DOCKER_HOST),
tls_hostname=dict(type='str', default=DEFAULT_TLS_HOSTNAME),
docker_host=dict(type='str', aliases=['docker_url'], default=DEFAULT_DOCKER_HOST, fallback=(env_fallback, 'DOCKER_HOST')),
tls_hostname=dict(type='str', default=DEFAULT_TLS_HOSTNAME, fallback=(env_fallback, 'DOCKER_TLS_HOSTNAME')),
api_version=dict(type='str', aliases=['docker_api_version'], default='auto'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a fallback with DOCKER_API_VERSION

tls=dict(type='bool', default=DEFAULT_TLS),
tls_verify=dict(type='bool', default=DEFAULT_TLS_VERIFY),
ssl_version=dict(type='str', default=DEFAULT_SSL_VERSION, fallback=(env_fallback, 'DOCKER_SSL_VERSION')),
tls=dict(type='bool', default=DEFAULT_TLS, fallback=(env_fallback, 'DOCKER_TLS_VERIFY')),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback env variable is called DOCKER_TLS, not DOCKER_TLS_VERIFY (that's the one below).

@ansibot
Copy link
Contributor

ansibot commented Aug 22, 2018

@denraf this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@ansibot ansibot added merge_commit This PR contains at least one merge commit. Please resolve! and removed ci_verified Changes made in this PR are causing tests to fail. stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Aug 22, 2018
@ansibot
Copy link
Contributor

ansibot commented Aug 22, 2018

@denraf this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@felixfontein
Copy link
Contributor

@denraf You need to remove the merge commits. Check the link for instructions.

@ansibot
Copy link
Contributor

ansibot commented Aug 22, 2018

@denraf this PR contains the following merge commits:

Please rebase your branch to remove these commits.

click here for bot help

@ansibot ansibot added needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI. stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Aug 22, 2018
api_version=dict(type='str', aliases=['docker_api_version'], default='auto'),
docker_host=dict(type='str', aliases=['docker_url'], default=DEFAULT_DOCKER_HOST, fallback=(env_fallback, 'DOCKER_HOST')),
tls_hostname=dict(type='str', default=DEFAULT_TLS_HOSTNAME, fallback=(env_fallback, 'DOCKER_TLS_HOSTNAME')),
api_version=dict(type='str', aliases=['docker_api_version'], default='auto', fallback=(env_fallback, 'DOCKER_API_VERSION')),
timeout=dict(type='int', default=DEFAULT_TIMEOUT_SECONDS),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, I forgot one more: timeout is missing the DOCKER_TIMEOUT fallback.

@ansibot ansibot removed merge_commit This PR contains at least one merge commit. Please resolve! needs_ci This PR requires CI testing to be performed. Please close and re-open this PR to trigger CI. needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html stale_ci This PR has been tested by CI more than one week ago. Close and re-open this PR to get it retested. labels Aug 22, 2018
@felixfontein
Copy link
Contributor

@kassiansun Can you take a look here?

@kassiansun
Copy link
Contributor

shipit

@felixfontein
Copy link
Contributor

bot_status

@ansibot
Copy link
Contributor

ansibot commented Aug 23, 2018

Components

lib/ansible/module_utils/docker_common.py
support: core
maintainers:

Metadata

waiting_on: ansible
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
too many files or commits: False
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 0
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 0
shipit_actors (maintainer or core team member): []
shipit_actors_other: felixfontein kassiansun
automerge: automerge shipit test failed

click here for bot help

@kassiansun
Copy link
Contributor

@felixfontein Seems we're not maintainers of docker_common

@felixfontein
Copy link
Contributor

Yep. Well, I guess we need help then :)

@resmo Can you take a look?

@resmo
Copy link
Contributor

resmo commented Aug 23, 2018

Hi all, is this a regression? Made by 4117b2d ?

@resmo
Copy link
Contributor

resmo commented Aug 23, 2018

AFAICS, there are more things to "fix" or cleanup. IMHO the source of the issue is in _get_value()
https://github.com/ansible/ansible/pull/42641/files#diff-1ba35f3163ddb36a0cfc16c50579434dR204

But for a "hotfix" this PR looks good to me. We should merge this and cleanup in 2.7 only.

shipit

@felixfontein
Copy link
Contributor

@resmo Ah, you found the source ;) Yes, it looks like it.

I think the best way is to remove the env variable parameter from _get_value(), and use the fallback mechanism for all variables. But that's something for 2.7 or maybe later (since Core Freeze is today).

@felixfontein
Copy link
Contributor

bot_status

@ansibot
Copy link
Contributor

ansibot commented Aug 23, 2018

Components

lib/ansible/module_utils/docker_common.py
support: core
maintainers:

Metadata

waiting_on: ansible
changes_requested_by: null
needs_info: False
needs_revision: False
needs_rebase: False
merge_commits: []
too many files or commits: False
mergeable_state: clean
shippable_status: success
maintainer_shipits (module maintainers): 0
community_shipits (namespace maintainers): 0
ansible_shipits (core team members): 1
shipit_actors (maintainers or core team members): resmo
shipit_actors_other: felixfontein kassiansun
automerge: automerge shipit test failed

click here for bot help

tls_hostname=dict(type='str', default=DEFAULT_TLS_HOSTNAME),
api_version=dict(type='str', aliases=['docker_api_version'], default='auto'),
timeout=dict(type='int', default=DEFAULT_TIMEOUT_SECONDS),
docker_host=dict(type='str', aliases=['docker_url'], default=DEFAULT_DOCKER_HOST, fallback=(env_fallback, ['DOCKER_HOST'])),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please update lib/ansible/utils/module_docs_fragments/docker.py to mention these variables, see lib/ansible/utils/module_docs_fragments/vmware.py to an example

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that would be a good idea. I guess we can get rid of the corresponding paragraph in the notes: section then.

@denraf Can you also update the documentation (in the docs fragment) for cacert_path, cert_path and key_path? They are a bit more tricky, since there the value of the env variable DOCKER_CERT_PATH is taken and modified.

@ansibot ansibot added the needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. label Aug 28, 2018
@gundalow
Copy link
Contributor

After discussion with @felixfontein I'll merge this and @felixfontein will update this in a follow up PR

@gundalow gundalow merged commit bbb57f5 into ansible:devel Aug 29, 2018
felixfontein pushed a commit to felixfontein/ansible that referenced this pull request Aug 29, 2018
@denraf denraf deleted the bugfix/docker-env-variables branch August 29, 2018 15:44
felixfontein added a commit to felixfontein/ansible that referenced this pull request Sep 1, 2018
mattclay pushed a commit that referenced this pull request Sep 4, 2018
* Set Default values correctly for docker variables (#42641)

* Set Default values correctly

* Properly documenting environment variables in documentation fragment. (#44812)

* Added changelog.
abadger pushed a commit that referenced this pull request Sep 6, 2018
* Add changelog entry for #16748.

* Changelog entry for #33579.

* Added changelog for #42380.

* Added changelog for #42641.

* Adding changelog entry for #42857.

* Added changelog for #44808.
ndswartz pushed a commit to ndswartz/ansible that referenced this pull request Nov 28, 2018
* Add changelog entry for ansible#16748.

* Changelog entry for ansible#33579.

* Added changelog for ansible#42380.

* Added changelog for ansible#42641.

* Adding changelog entry for ansible#42857.

* Added changelog for ansible#44808.
@ansible ansible locked and limited conversation to collaborators Jul 22, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. docker needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR. support:core This issue/PR relates to code supported by the Ansible Engineering Team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

9 participants