New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Created new conda module for managing conda packages #40455

Open
wants to merge 2 commits into
base: devel
from

Conversation

Projects
None yet
8 participants
@tmoschou

tmoschou commented May 20, 2018

SUMMARY

Created new conda module for managing conda packages.

ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

conda

ANSIBLE VERSION
ansible 2.6.0 (feature/conda-module d104f10b0e) last updated 2018/05/21 01:41:23 (GMT +1050)
  config file = None
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ******/ansible/lib/ansible
  executable location = ******/ansible/bin/ansible
  python version = 3.6.5 (default, Apr 19 2018, 10:56:22) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)]
ADDITIONAL INFORMATION

From https://conda.io/

Package, dependency and environment management for any language—Python, R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN

Conda is an open source package management system and environment management system that runs on Windows, macOS and Linux. Conda quickly installs, runs and updates packages and their dependencies. Conda easily creates, saves, loads and switches between environments on your local computer. It was created for Python programs, but it can package and distribute software for any language.

Module allows you to install, update or remove multiple packages in one transaction.
It is basically a wrapper around the conda command line utility. The command line utility supports writing output in machine readable json and supports a dry-run mode making implementing an ansible module wrapper relatively simple.

@ansibot

This comment has been minimized.

Contributor

ansibot commented May 20, 2018

The test ansible-test sanity --test validate-modules [explain] failed with 1 error:

lib/ansible/modules/packaging/language/conda.py:0:0: E307 version_added should be 2.6. Currently 2.5

The test ansible-test sanity --test yamllint [explain] failed with 4 errors:

test/integration/targets/conda/tasks/darwin.yml:14:1: empty-lines too many blank lines (1 > 0)
test/integration/targets/conda/tasks/linux.yml:14:1: empty-lines too many blank lines (1 > 0)
test/integration/targets/conda/tasks/main.yml:4:1: empty-lines too many blank lines (2 > 0)
test/integration/targets/conda/tasks/setup.yml:23:1: empty-lines too many blank lines (1 > 0)

click here for bot help

@ansibot ansibot added needs_revision and removed core_review labels May 20, 2018

@tmoschou

This comment has been minimized.

tmoschou commented May 21, 2018

ready_for_review

@ansibot ansibot added core_review and removed needs_revision labels May 21, 2018

@jborean93 jborean93 removed the needs_triage label May 24, 2018

@tmoschou tmoschou force-pushed the tmoschou:feature/conda-module branch May 25, 2018

@ansibot

This comment has been minimized.

Contributor

ansibot commented May 25, 2018

The test ansible-test sanity --test shebang [explain] failed with 1 error:

lib/ansible/modules/source_control/subversion.py:0:0: module should not be executable

click here for bot help

@ansibot ansibot added needs_revision and removed core_review labels May 25, 2018

Initial commit of conda module
Added integration tests

Lint errors

Skip freebsd integration tests, enable osx

@tmoschou tmoschou force-pushed the tmoschou:feature/conda-module branch to f8e30f7 May 25, 2018

@tmoschou

This comment has been minimized.

tmoschou commented May 25, 2018

ready_for_review

@ansibot ansibot added core_review and removed needs_revision labels May 25, 2018

@colin-nolan

This comment has been minimized.

Contributor

colin-nolan commented May 29, 2018

Have you written this module from scratch or is this code from another project, which has already been tried and tested? You posted a comment on UDST/ansible-conda, indicating you were aware of that codebase but this code seems to be entirely different.

We use UDST/ansible-conda in our production setups but that was only possible after fixing it to work with a lot of oddities of the conda CLI. I would be reluctant to change to this if it becomes core unless it can be shown to pass all the same tests as are in UDST/ansible-conda (there are many more test cases required to demonstrate that the module works than those in this PR). It would also be great to show that this code has also been tried and tested by users in the wild.

As it stands, I do not support this PR. However, I do think it would be good for there to be core support for Conda so I endorse the effort.

@tmoschou

This comment has been minimized.

tmoschou commented May 29, 2018

Hi @colin-nolan,

Firstly happy to add more tests.

This module has been developed internally at my work and now wishing to contribute upstream. We have tried and tested this module in our environments but of course welcome others testing too. I was only aware of the UDST code base after this PR was raised, and noticed that there is an open tick on UDST to contribute upstream, but the PR to was ultimately closed as permission to re-licence/redistribute had not been obtained/verified. And nearly wished to convey to any interested parties watching the issue this attempt was in the works.

Re: testing parity between UDST and this - I note that there are API differences. In particular this module allows installing, updating, removing multiple packages in one transaction. This was particularly important for us as installing packages over a loop, caused problems with dependency resolution where versions of packages were changed multiple times and not to the one intended (and much slower as multiple versions of the same package were downloaded and re-installed). As we pass in a list of packages, we hence don't have a version argument as this is specified using package spec notation (e.g. name: numpy>=1.8,<2|1.9). Contrary to UDST, we deliberately permit state: latest when specifying a version spec, in fact packages with a spec are always treated as if latest for the spec (regardless if state: present).

In regards to your concerns around the oddities of the conda CLI, this version handles various issues that I believe you may be referring to judging by UDST/ansible-conda#9, UDST/ansible-conda#14, etc - in particular a change in json format between 4.2 and 4.3 as well as various regressions between versions that we have encountered. E.g --dry-run mode returning different format, or --quiet being ignored if --json is also specified. This PR's integration suite loops over multiple versions of conda to test these. I have also reached out to the continuum conda discussion forum (@kalefranz) for review/feedback should there be any other oddities, that I may not be unaware of. I could not see any tests in UDST testing for odd behaviours of conda that I am not already testing for. If you have any specific examples please share.

@komailrk

This comment has been minimized.

komailrk commented Jun 3, 2018

I will be happy to try this out in our environment and give feedback.

Curious to see if the setup will still run if we rerun the playbook. From what I understood, Ansible will not modify the system if its not required.

@ansibot ansibot added the stale_ci label Jun 3, 2018

@kalefranz

This comment has been minimized.

Contributor

kalefranz commented Jun 3, 2018

Read through the whole PR top to bottom on my computer. On my phone now though, so we’ll see how far this goes. Probably just a first round of comments. These specifically pertaining to the API the module presents.

  • Conda’s core operations are create, install, update, and remove. Just like we don’t need create, I also don’t think we need update. I think the declared state—either present, latest, or absent—fully covers the conda operation type being invoked.

  • Speaking of the present state, conda 4.6 will have a new flag that makes this possible finally. See conda/conda#7291. The default behavior for conda install is really equivalent to what ansible’s latest is, this case coupled with the conda concept of constraints around a package spec. Ansible’s present is this new -S flag.

  • I strongly encourage changing env to env_name, and if necessary making env an alias. In general I prefer that people think of environments having locations rather than just names. The env key here would just be ambiguous and cause potential confusion.

  • The conda CLI quirks... Help me lock down the behavior, and fix through a deprecation path the things that need to change. If we get a robust set of tests here, I’d be eager to figure out how to add the conda ansible module as a test target on conda’s own CI that runs for each commit.

Ok those are initial drive-by thoughts. I’ll have more I’m sure when I can revisit this back on a proper computer screen.

@tmoschou tmoschou changed the title from Created new conda module for managing conda packages to [WIP] Created new conda module for managing conda packages Jun 4, 2018

@ansibot ansibot added the WIP label Jun 4, 2018

@ansibot ansibot removed the stale_ci label Jun 4, 2018

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 4, 2018

The test ansible-test sanity --test validate-modules [explain] failed with 1 error:

lib/ansible/modules/packaging/language/conda.py:0:0: E307 version_added should be 2.7. Currently 2.6

click here for bot help

- name: get miniconda3 installation script
get_url:
url: "https://repo.continuum.io/miniconda/Miniconda3-{{ conda_bootstrap.version }}-{{ conda_variant }}-x86_64.sh"

This comment has been minimized.

@nehaljwani

nehaljwani Jun 7, 2018

Contributor

s/repo.continuum.io/repo.anaconda.com :-)

@aldanor

This comment has been minimized.

aldanor commented Jun 12, 2018

Wonder if I'm missing something here; how do you install a package into a specific environment with this module? (Also ensuring that environment gets created if it didn't exist in the first place)

E.g., production box has two different conda environments that you want to maintain alongside.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment