Virtualenv by atarkowska · Pull Request #183 · ome/infrastructure

atarkowska · 2016-11-18T16:29:33Z

This PR removes PyPI dependences to virtualenv

This PR is for the discussion on #sysadmin and maybe some clean-up first

atarkowska · 2016-11-18T16:34:54Z

+      virtualenv_path: /home/{{ jenkinsuser }}/virtualenv
+      virtualenv_site_packages: "yes"
+      virtualenv_pip_rpm_dependencies:
+      virtualenv_pip_dependencies:


what to install by default in any node?

Should this be a variable that can be overriden by group?

atarkowska · 2016-11-18T16:35:06Z

+    virtualenv_path: /home/{{ jenkinsuser }}/virtualenv
+    virtualenv_site_packages: "yes"
+    virtualenv_pip_rpm_dependencies:
+    virtualenv_pip_dependencies:


which are doc dependences exactly?

I am not sure about the requirement for a virtualenv in ci-docs groups. As discussed in #158, the current nature of the documentation build stack (i.e. Ant + Sphinx) makes it not usable with a virtualenv in a Jenkins environment. Is there another use case?

@sbesson for autogen I had to add #129. Does it mean is no longer the case? We don't need venv at all?

This question is about https://github.com/openmicroscopy/infrastructure/pull/183/files#diff-30bd0d487ebfa61df9035699fe40e329R60

Briefly discussed with @aleksandra-tarkowska, I think the issue is that different jobs have different strategies. Some of them use Jenkins Ant + globally installed Sphinx (BIOFORMATS build jobs) others use globally installed Ant + virtualenv (OMERO docs autogen jobs). Rationalizing might be a long term goal. To maintain compatibility, my feeling is that installing Sphinx in the system packages via sphinx-build and creating a virtualenv with --system-site-packages should maintain compatibility for both types of documentation jobs.

atarkowska · 2016-11-18T16:38:59Z

- `omero_python_deps_profile`: Either `distribution` to use the distribution packages for Python modules where possible (default), or `pypi` to install recommended packages globally from PyPI using `pip`.
 - `omero_python_deps_recommended`: If `False` install a minimal set of Python dependencies, default `True`.

+If `omero_python_deps_recommended` False then use virtualenv role


@manics are you ok with removal of omero_python_deps_profile and handling that as described below?

In principle that's fine. What's the plan for specifying the list of pip packages- will the full list always have to be specified in the playbook?

manics

@openmicroscopy/devops A general issue that's arisen here: I think we should try and avoid uninstalling anything at the system level, as it causes problems with other packages that may depend on it.

manics · 2016-11-23T09:09:25Z

-  when: "omero_python_deps_profile not in ['distribution', 'pypi']"

-# TODO: Consider installing some of these with pip if the user requests it?
+- name: omero python deps | pypi pillow uninstall


Can you avoid uninstalling/removing packages:

If other packages depend on this it may break them (e.g. in a previous PR removing python-pillow also removed cloud-init)

In this particular case the global pip will also remove the files installed by yum, but the yum/rpm database still lists the package as installed, e.g.

[root@5a85874b4943 /]# yum install -q -y epel-release [root@5a85874b4943 /]# yum install -q -y python-pip [root@5a85874b4943 /]# pip freeze |grep Pillow [root@5a85874b4943 /]# yum install -q -y python-pillow [root@5a85874b4943 /]# pip freeze |grep Pillow Pillow==2.0.0 [root@5a85874b4943 /]# python -c 'import PIL' [root@5a85874b4943 /]# pip uninstall -y Pillow Uninstalling Pillow-2.0.0: Successfully uninstalled Pillow-2.0.0 [root@5a85874b4943 /]# pip freeze |grep Pillow [root@5a85874b4943 /]# rpm -q python-pillow python-pillow-2.0.0-19.gitd1c6db8.el7.x86_64 [root@5a85874b4943 /]# python -c 'import PIL' Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named PIL

manics · 2016-11-23T09:12:02Z

+
+- name: virtualenv | pypi {{ virtualenv_pip_dependencies }}
+  become: yes
+  become_user: "{{ virtualenv_user }}"


Can you make virtualenv_user default to root, since that's what you'd probably use in production.

manics · 2016-11-23T09:56:38Z

- `omero_python_deps_profile`: Either `distribution` to use the distribution packages for Python modules where possible (default), or `pypi` to install recommended packages globally from PyPI using `pip`.
 - `omero_python_deps_recommended`: If `False` install a minimal set of Python dependencies, default `True`.

+If `omero_python_deps_recommended` False then use virtualenv role


In principle that's fine. What's the plan for specifying the list of pip packages- will the full list always have to be specified in the playbook?

manics · 2016-12-07T12:36:14Z

A more general question: Is this role intended to cover all uses of virtualenv, including use-cases which aren't related to OMERO/BF and related applications?

atarkowska · 2016-12-07T13:11:30Z

@manics could you give the example of what you are thinking of?

manics · 2016-12-07T16:02:04Z

Suppose I want to setup a dedicated box for running infrastructure tasks (with jenkins, rundeck, manually, or some other method). This would require ansible, shade, python-openstackclients, and maybe some other python modules, but this has nothing to do with OMERO.

If you want this role to be focussed on OMERO you can setup things by default, if it's meant to work for every use case that's more difficult.

atarkowska · 2016-12-07T16:04:13Z

I am not sure I understand why setting

role: virtualenv

vars:
    virtualenv_pip_dependencies:
    - ansible
    - shade

won't work in that case?

maybe I will add a quick test...

manics · 2016-12-07T16:43:14Z

It should work. My question was about what your intentions are. If you keep this completely generic you'll have to bear this in mind if you need to make future changes to the role, since it could be used anywhere. If you limit the scope you've got more freedom.

atarkowska added 2 commits November 16, 2016 15:12

remove gcc from basedeps

61c0319

add virtualenv role

34c264d

atarkowska commented Nov 18, 2016

View reviewed changes

move to vars

8c7947c

manics suggested changes Nov 23, 2016

View reviewed changes

This was referenced Dec 9, 2016

copy snoopy keys #211

Closed

Install jinja2 PyYAML in omero-python-deps (IDR-0.3.1) #217

Merged

atarkowska closed this Dec 13, 2016

sbesson referenced this pull request in ome/devspace Jan 19, 2017

install sphinx in a job's virtualenv as required

dc2e370

Conversation

atarkowska commented Nov 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atarkowska Nov 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbesson Nov 18, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manics left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manics commented Dec 7, 2016

Uh oh!

atarkowska commented Dec 7, 2016

Uh oh!

manics commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atarkowska commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manics commented Dec 7, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

atarkowska commented Nov 18, 2016 •

edited

Loading

atarkowska Nov 18, 2016 •

edited

Loading

sbesson Nov 18, 2016 •

edited

Loading

manics commented Dec 7, 2016 •

edited

Loading

atarkowska commented Dec 7, 2016 •

edited

Loading

manics commented Dec 7, 2016 •

edited

Loading