Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install multiple requirements files via pip and virtualenv states #4648

Closed
OddBloke opened this issue Apr 30, 2013 · 26 comments
Closed

Install multiple requirements files via pip and virtualenv states #4648

OddBloke opened this issue Apr 30, 2013 · 26 comments
Labels
Documentation Relates to Salt documentation Feature new functionality including changes to functionality and code refactors, etc. P2 Priority 2 Platform Relates to OS, containers, platform-based utilities like FS, system based apps stale State-Module
Milestone

Comments

@OddBloke
Copy link

We have some projects where we have more than one requirements file (a core one which everywhere should install, and a site-specific one). At the moment, we have to write out a full state for each file:

main_requirements:
  pip.installed:
    - name:
    - requirements: ...

other_requirements:
  pip.installed:
    - name:
    - requirements: ...

It is also painful in a virtualenv definition:

my_virtualenv:
  virtualenv.managed:
    - name: ...
    - requirements: ...

extra_requirements:
  pip.installed:
    - name:
    - requirements: ...

And, of course, we then either have to introduce dependencies between the requirements states or depend on all of the requirements states where we need them.

Preferable would be a list of requirements files:

main_requirements:
  pip.installed:
    - name:
    - requirements:
      - main_requirements
      - other_requirements

This would boil down to pip install -r main_requirements -r other_requirements.

@thatch45
Copy link
Member

I am going to approve this for the future, thanks for the request

@boxy-robot
Copy link

It'd be pretty sweet if we could get a cwd on virtualenv.managed like there is on pip.installed.

I typically have my requirements files split out by environment like this:

Base file:

# base.txt
Django==1.5.1

Environment file:

# dev.txt
-r base.txt

django-debug-toolbar==0.9.4

The blank - name: is kind of weird, but works for now.

@copelco
Copy link

copelco commented Sep 7, 2013

@yellottyellott Agreed. Just ran into this issue too.

@gravyboat
Copy link
Contributor

I would like to see this as well.

@JasonSwindle
Copy link
Contributor

It looks like we are getting hit by this as well.

.......
  virtualenv.managed:
    - name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}-python-2.7
    - no_site_packages: True
    - user: {{ pillar['project_name'] }}
    - cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}
    - requirements: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/requirements.txt
.......
    State: - virtualenv
    Name:      /opt/apps/.virtualenvs/moo-python-2.7
    Function:  managed
        Result:    False
        Comment:   Created new virtualenv
Could not open requirements file: [Errno 2] No such file or directory: '/tmp/requirements/production.txt'
Storing complete log in /opt/apps/.pip/pip.log

        Changes:   new: Python 2.7.3

requirements.txt

# This file is here because many Platforms as a Service look for
#       requirements.txt in the root directory of a project.
-r requirements/production.txt

requirements/production.txt

# Pro-tip: Try not to put anything here. There should be no dependency in
#   production that isn't in development.
-r base.txt

psycopg2

requirements/base.txt

Django>=1.5.5
bpython>=0.12
django-braces>=1.2.2
django-model-utils>=1.5.0
logutils>=0.3.3
South>=0.8.2
django_auth_ldap>=1.1.3
ipython>=1.1.0
django-extensions>=1.2.5
pytz
django-crispy-forms>=1.4.0
python-memcached>=1.53
           Salt: 0.17.4
         Python: 2.7.3 (default, Apr 10 2013, 06:20:15)
         Jinja2: 2.6
       M2Crypto: 0.21.1
 msgpack-python: 0.1.10
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
         PyYAML: 3.10
          PyZMQ: 13.0.0
            ZMQ: 3.2.2

Even with -cwd: and chaining requirements.txt, it keeps using /tmp and not the correct path.

Jason Swindle

@garethgreenaway
Copy link
Contributor

Going to look into fixing this.

@garethgreenaway
Copy link
Contributor

Confirmed that the cwd option for the virtualenv state does work using the example state above with pillar data. Can someone include an example state where its not working?

@garethgreenaway
Copy link
Contributor

Multiple requirements appears to work as well. Simply separate them by a comma on in the state.

@gifflen
Copy link

gifflen commented Dec 31, 2013

I'm working with Jason Swindle above on the same project. When we attempt to run anything that chains requirement files with the -r option in a requirements file the cwd location is not being respected though it is for any file explicitly called from salt.

We have requirements.txt in the root of our project directory: /opt/apps/django-project/requirements.txt Its contents contains -r requirements/production.txt which in turn has -r base.txt. Using this method we are chaining the various requirements for different environments keeping common requirements in base.txt and secific environment requirements in each spcecific environment file.

When attempting to use virtualenv.managed as seen in Jason's comment above it finds /opt/apps/django-project/requirements.txt file appropriately and attempts to install the requirements in the file. Since that file is present to really only to act as a proxy for PAAS installs it should then hand it off to requirements/production.txt locatied in /opt/apps/django-project/requirements/production.txt. Instead the logs show it is searching /tmp/requirements/production.txt

When using virtualenv directly this functionality works as expected it is only when using salt that we encounter this problem.

If you need further information let me know. I can provide any amount of debug needed.

@garethgreenaway
Copy link
Contributor

That helps make the situation clearer :) I'll replicate that scenario and see what happens. Thanks.

@garethgreenaway
Copy link
Contributor

Can you provide the state file and pillar files you're using? I tested with the following and it worked without error:

State:
foo:
virtualenv.managed:
- name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}-python-2.7
- no_site_packages: True
- cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}
- requirements: requirements.txt

Pillar:
document_root: /opt/apps
project_name: moo

/opt/apps/moo/requirements:
-rw-r--r-- 1 root root 19 Dec 31 09:03 base.txt
-rw-r--r-- 1 root root 12 Dec 31 09:20 production.txt

/opt/apps/moo/requirements.txt:
-r requirements/production.txt

/opt/apps/moo/requirements/production.txt:
-r base.txt

/opt/apps/moo/requirements/base.txt:
Django>=1.5.5
pytz

@JasonSwindle
Copy link
Contributor

Here you go, I hope this helps. :)

pillar

## Project name
project_name: 'appname'

## Application's home
document_root: '/opt/apps'

sls

app_install:
  file.directory:
    - name: {{ pillar['document_root'] }}/.virtualenvs/
    - owner: {{ pillar['project_name'] }}
    - group: www-data
    - require:
      - user: app_user
  git.latest:
    - name: {{ pillar['git_url'] }}
    - rev: {{ pillar['branch_or_tag'] }}
    - target: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo
    - user: {{ pillar['project_name'] }}
    - force: True
    - require:
      - file: app_install
      - pkg: git_install
  virtualenv.managed:
    - name: {{ pillar['document_root'] }}/.virtualenvs/{{ pillar['project_name'] }}
    - no_site_packages: True
    - user: {{ pillar['project_name'] }}
    - cwd: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo
    - requirements: {{ pillar['document_root'] }}/{{ pillar['project_name'] }}/{{ pillar['project_name'] }}-repo/requirements.txt
    - require:
      - git: app_install
      - pip: python-virtualenv_install
      - pkg: python-dev_install
      - pkg: libldap2-dev_install
      - pkg: libsasl2-dev_install
      - pkg: libssl-dev_install
      - pkg: python-pip_install
      - pkg: python-ldap_install
      - pkg: libpq-dev_install
      - file: newrelic_app_config
      - file: ldap_conf
    - require_in:
      - pip: uwsgi_install

Error

----------
    State: - virtualenv
    Name:      /opt/apps/.virtualenvs/appname
    Function:  managed
        Result:    False
        Comment:   virtualenv exists
Could not open requirements file: [Errno 2] No such file or directory: '/tmp/requirements/production.txt'
Storing complete log in /opt/apps/appname/.pip/pip.log

        Changes:
----------
    State: - pip
    Name:      uwsgi
    Function:  installed
        Result:    False
        Comment:   One or more requisite failed
        Changes:

@basepi
Copy link
Contributor

basepi commented Jan 3, 2014

So unless I'm mistaken, this thread has gone fairly off-topic. Can one of you create a new thread for the cwd stuff? This thread should only be used for the multiple requirements files feature request.

@garethgreenaway
Copy link
Contributor

From my testing with both scenarios, both the cwd option and being able to specify multiple requirement files either by chaining them or specifying multiple requirement files inside the state file, works.

@basepi
Copy link
Contributor

basepi commented Jan 6, 2014

Can someone else verify on the latest develop branch?

@JasonSwindle
Copy link
Contributor

Howdy,

I am trying to but #9587 sadly is stopping me.

Jason Swindle

@rallytime
Copy link
Contributor

Looks like #9587 has been resolved. Can anyone else reproduce these issues? Looks like this has been fixed on develop.

@rallytime
Copy link
Contributor

Since this has been fixed on develop, I am closing this one. If this issue comes up again, leave a comment and we can definitely re-open it and address any problems.

@chris-martin
Copy link
Contributor

@garethgreenaway:

being able to specify multiple requirement files either by chaining them or specifying multiple requirement files inside the state file, works

Can you please clarify what this means?

  • What does it means to "chain" requirements files?
  • "specifying multiple requirement files" - How?

The documentation for this state still seems to indicate that only one requirements file may be given.

@gravyboat
Copy link
Contributor

@chris-martin For specifying multiple requirements files the code over here: https://github.com/saltstack/salt/blob/develop/salt/modules/pip.py#L200 seems to indicate you'd want to use:

- requirements:
  - requirement_one
  - requirement_two

For the chaining of requirements files I believe @garethgreenaway is referring to something like this:

https://stackoverflow.com/questions/11704287/split-requirements-files-in-pip

Where a file contains two additional file references. (which is disgusting by the way @garethgreenaway ;) )

Docs probably need to be updated @jfindlay. This issue should either be re-opened and noted as docs, or a new one created to confirm functionality and document it.

@jfindlay jfindlay added Documentation Relates to Salt documentation State-Module P2 Priority 2 Platform Relates to OS, containers, platform-based utilities like FS, system based apps and removed fixed-pls-verify fix is linked, bug author to confirm fix labels Sep 16, 2015
@jfindlay jfindlay reopened this Sep 16, 2015
@bashu
Copy link

bashu commented Oct 24, 2015

+1 still an issue...

@gladiatr72
Copy link
Contributor

For those that are watching this thread, I've got a patch to modules/pip.py to address the requirements chaining bug. My work is against develop--I haven't verified that the multiple requirements feature has been taken care of, but it looks like it! I ran into the issue with the chained requirements file and decided to fix it. After I give it another once-over (post-holiday) I'll submit it against this issue.

Summary: With the default method of copy + chown for handling requirements files and a non-root user argument, the existing code takes the referenced requirements file, copies it to a temp file, does its user-switch and then runs against the temp file.

The patched module, instead, uses a temp directory filtering on -r lines with the state provided requirements file as a base. The end result is a list of requirements files that are copied to the temp directory, chown'd and then given to the dropped-privilege pip install process.

@BaseApi, I know this is an ancient thread. If you'd prefer a new issue (and you are available to even see this), I'd be happy to make the pull req against a new issue.

@basepi
Copy link
Contributor

basepi commented Jan 4, 2016

Looks like your pull request was submitted and accepted! No need for a new issue as far as I'm concerned. Thanks @gladiatr72 !

(I should also note to be careful of username typos..... @basepi vs @BaseApi, or I might miss something in the future. :) )

@anlutro
Copy link
Contributor

anlutro commented Jun 8, 2016

Just going to chime in and say that while the pip module now supports multiple requirements file, the virtualenv.managed state function still insists that the requirements argument be a string here.

Would a fix for this be appropriate against current release branches, or would it have to go into develop?

@thatch45
Copy link
Member

thatch45 commented Jun 8, 2016

This is typically something that we would put in develop, since it changes the behavior of a state instead of being a bugfix. Thanks for asking :)

@stale
Copy link

stale bot commented May 19, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

@stale stale bot added the stale label May 19, 2018
@stale stale bot closed this as completed May 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation Relates to Salt documentation Feature new functionality including changes to functionality and code refactors, etc. P2 Priority 2 Platform Relates to OS, containers, platform-based utilities like FS, system based apps stale State-Module
Projects
None yet
Development

No branches or pull requests