Allow pip requirements.txt syntax in environment files #3969

sdvillal · 2016-11-30T07:37:55Z

resolves #3763
resolves #4378

Currently the declaration of pip dependencies in conda environment.yml files is limited to a list of strings that would be accepted in a pip command line. This disallows using pip requirement files or installing pip dependencies in editable mode (unless using lesser known syntax tricks like "--editable=my_project"). Both are use cases I have missed recently and have also been requested elsewhere (e.g. #3763 #4378). This PR enables the full pip requirement file syntax and features to be used in environment.yml files. For example:

name: advanced-pip-example

dependencies:
  - pip
  - pip:

    # Global options can be tweaked.
    # For example, if you want to use a pypi mirror first:
    - --index-url https://pypi.doubanio.com/simple
    - --extra-index-url https://pypi.python.org/simple
    # (check https://www.pypi-mirrors.org/)

    # Current syntax still works
    - xmltodict==0.10.2

    # Install in editable mode.
    # More natural than - "--editable=git+https://github.com/neithere/argh.git#egg=argh"
    - -e git+https://github.com/neithere/argh.git#egg=argh

    # You could also specify a package in a directory.
    # The directory can be relative to this environment file.
    - -e ./module_to_install_in_editable_mode

    # Use another requirements file.
    # Note that here also we can use relative paths.
    # pip will be run from the environment file directory, if provided.
    - -r another-project-requirements.txt

    # Anything else that pip requirement files allows should work seamlessly...

As far as I can tell, these simple changes should not break any existing environment specification.

Also this PR fixes conda not listing pip egg-link dependencies (for example, dependencies installed in editable mode), and adds some integration tests to make sure pip packages are installed correctly (these tests depend on pypi to be up).

If merged, something like this could go to the changelog:

### Improvements
* accept all valid pip syntax in environment files (#3969)

### Bug Fixes
* Follow egg links when listing pip packages. Fixes, for
  example, `conda list` on the presence of packages
  installed on editable mode. (#3969)

sdvillal · 2016-11-30T12:22:42Z

I think these failing tests have nothing to do with the changes in this PR.

sdvillal · 2016-12-26T12:59:07Z

I have polished the PR a bit: better description, fixed paths on windows tests, squashed to one commit and changed the target branch to upstream 4.3.x.

The tests still fail for reasons that have nothing to do with this commit (I think because of recent changes to the packages in anaconda.org). I will trigger again test execution when these problems get fixed (e.g. if #4145 is merged).

tdhopper · 2017-01-03T22:28:15Z

@sdvillal Looks like #4145 is merged. I'm eager to get this feature!

sdvillal · 2017-01-03T23:59:57Z

@tdhopper We also need #4169 merged to get the green tick. Will keep an eye and trigger when it gets merged. Even then this PR might not get reviewed anytime soon. I think the admins face a pile of work.

What could be useful is to actually check that --extra-index also works when directly used in the environment yaml (I do not see why it would not). Do you have an example of a public url I could try, so I can add it to the PR description after making sure it does work?

tdhopper · 2017-01-04T16:06:21Z

@sdvillal: I put an index up at http://pythonextraindextest.com.s3-website-us-east-1.amazonaws.com with a package called exception that I maintain. Can you try that?

sdvillal · 2017-01-09T07:56:24Z

Thanks @tdhopper, that was useful. I have now updated the PR description and added integration tests to the pull request after confirming that everything seems to work as expected. On the way I have fixed conda not listing pip packages installed on editable mode. Hopefully this makes everything better and easier to review. And hopefully we finally earn that green tick.

In the example and the tests (which are just running that example) I'm using a official pip mirror to show how to change pip index priorities. You can delete your test index if you want to.

sdvillal · 2017-01-09T11:02:17Z

Green tick, ready to merge.

If merged, something like this could go to the changelog:

### Improvements
* accept all valid pip syntax in environment files (#3969)

### Bug Fixes
* Follow egg links when listing pip packages. Fixes, for
  example, `conda list` on the presence of packages
  installed on editable mode. (#3969)

- runs pip from the same directory where the environment file resides - adds reference to pip requirements file format - adds some tests

codecov · 2017-02-24T21:18:13Z

Codecov Report

Merging #3969 into 4.3.x will decrease coverage by -0.71%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            4.3.x    #3969      +/-   ##
==========================================
- Coverage   86.05%   85.35%   -0.71%     
==========================================
  Files          86       82       -4     
  Lines       10124     9595     -529     
==========================================
- Hits         8712     8189     -523     
+ Misses       1412     1406       -6

Impacted Files	Coverage Δ
conda/egg_info.py	`95.52% <100%> (+0.28%)`	✅
conda/base/exceptions.py	`0% <ø> (-100%)`	❌
conda/common/signals.py	`80% <ø> (-20%)`	❌
conda/gateways/disk/test.py	`68% <ø> (-19.88%)`	❌
conda/common/url.py	`78.05% <ø> (-14.97%)`	❌
conda/gateways/disk/permissions.py	`65.22% <ø> (-13.95%)`	❌
conda/core/index.py	`87.65% <ø> (-9.75%)`	❌
conda/gateways/disk/init.py	`83.87% <ø> (-9.68%)`	❌
conda/gateways/disk/create.py	`63.13% <ø> (-7.17%)`	❌
conda/cli/find_commands.py	`91.11% <ø> (-5.04%)`	❌
... and 46 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddd12b9...996a450. Read the comment docs.

includes working example of advanced pip use

codecov · 2017-02-24T22:09:57Z

Codecov Report

Merging #3969 into 4.3.x will decrease coverage by -0.71%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            4.3.x    #3969      +/-   ##
==========================================
- Coverage   86.05%   85.35%   -0.71%     
==========================================
  Files          86       82       -4     
  Lines       10124     9595     -529     
==========================================
- Hits         8712     8189     -523     
+ Misses       1412     1406       -6

Impacted Files	Coverage Δ
conda/egg_info.py	`95.52% <100%> (+0.28%)`	✅
conda/base/exceptions.py	`0% <ø> (-100%)`	❌
conda/common/signals.py	`80% <ø> (-20%)`	❌
conda/gateways/disk/test.py	`68% <ø> (-19.88%)`	❌
conda/common/url.py	`78.05% <ø> (-14.97%)`	❌
conda/gateways/disk/permissions.py	`65.22% <ø> (-13.95%)`	❌
conda/core/index.py	`87.65% <ø> (-9.75%)`	❌
conda/gateways/disk/init.py	`83.87% <ø> (-9.68%)`	❌
conda/gateways/disk/create.py	`63.13% <ø> (-7.17%)`	❌
conda/cli/find_commands.py	`91.11% <ø> (-5.04%)`	❌
... and 46 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddd12b9...0ce1f87. Read the comment docs.

codecov · 2017-03-08T21:53:14Z

Codecov Report

Merging #3969 into 4.3.x will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            4.3.x    #3969      +/-   ##
==========================================
+ Coverage   86.05%   86.06%   +0.01%     
==========================================
  Files          86       86              
  Lines       10124    10128       +4     
==========================================
+ Hits         8712     8716       +4     
  Misses       1412     1412

Impacted Files	Coverage Δ
conda/egg_info.py	`95.52% <100%> (+0.28%)`	✅

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddd12b9...0ce1f87. Read the comment docs.

codecov · 2017-04-04T06:32:18Z

Codecov Report

Merging #3969 into 4.3.x will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##            4.3.x    #3969      +/-   ##
==========================================
+ Coverage   86.05%   86.06%   +0.01%     
==========================================
  Files          86       86              
  Lines       10124    10128       +4     
==========================================
+ Hits         8712     8716       +4     
  Misses       1412     1412

Impacted Files	Coverage Δ
conda/egg_info.py	`95.52% <100%> (+0.28%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddd12b9...0ce1f87. Read the comment docs.

kalefranz

LGTM. Preparing for 4.4.0 soon. This will be in it.

sdvillal · 2017-04-27T07:40:24Z

Thanks Kale, looking forward to the new release.

github-actions · 2021-09-14T06:02:42Z

Hi there, thank you for your contribution to Conda!

This pull request has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue or pull request if needed.

sdvillal force-pushed the env-pip-via-requirements branch from 79f981f to b486223 Compare December 13, 2016 08:38

sdvillal force-pushed the env-pip-via-requirements branch 2 times, most recently from cbfd52c to a27318e Compare December 26, 2016 10:26

sdvillal changed the base branch from master to 4.3.x December 26, 2016 10:53

sdvillal force-pushed the env-pip-via-requirements branch 2 times, most recently from dd0bdea to 4ccaaf1 Compare December 26, 2016 12:07

sdvillal force-pushed the env-pip-via-requirements branch from 4ccaaf1 to 3a40fdf Compare January 3, 2017 16:49

sdvillal force-pushed the env-pip-via-requirements branch 2 times, most recently from f0e5809 to a29b5cc Compare January 9, 2017 07:41

sdvillal force-pushed the env-pip-via-requirements branch from 530dc81 to 4047934 Compare January 9, 2017 10:01

sdvillal mentioned this pull request Jan 27, 2017

Invalid requirement while trying to use pip options #4378

Closed

sdvillal added 2 commits February 24, 2017 21:29

use requirements files to install pip dependencies

ebd0f1a

- runs pip from the same directory where the environment file resides - adds reference to pip requirements file format - adds some tests

follow egg links when listing pip dependencies

1068c5b

sdvillal force-pushed the env-pip-via-requirements branch from 4047934 to 996a450 Compare February 24, 2017 21:18

add pip installation tests

0ce1f87

includes working example of advanced pip use

sdvillal force-pushed the env-pip-via-requirements branch from 996a450 to 0ce1f87 Compare February 24, 2017 22:09

sdvillal mentioned this pull request Feb 25, 2017

Adding pip flags in environment.yml #3763

Closed

TomAugspurger mentioned this pull request Mar 11, 2017

DOC: Use nbsphinx pandas-dev/pandas#15581

Merged

kalefranz changed the base branch from 4.3.x to 4.4.x April 27, 2017 03:01

kalefranz added this to the 4.4.0 milestone Apr 27, 2017

Merge branch '4.4.x' into env-pip-via-requirements

f49fce0

kalefranz approved these changes Apr 27, 2017

View reviewed changes

kalefranz merged commit 901bb38 into conda:4.4.x Apr 27, 2017

kalefranz changed the title ~~Allow to use pip requirements.txt syntax in environment files~~ Allow pip requirements.txt syntax in environment files Apr 27, 2017

kalefranz mentioned this pull request May 12, 2017

Custom pypi repo in conda environment files #3372

Closed

vxgmichel mentioned this pull request Jun 23, 2017

[doc] Get rid of mocks for docs taurus-org/taurus#408

Closed

kalefranz mentioned this pull request Oct 12, 2017

Package Request: uwsgi ContinuumIO/anaconda-issues#6594

Open

kalefranz mentioned this pull request Jan 29, 2018

Allow pip install flags to be recognized in environment files #6805

Open

mkobit mentioned this pull request May 14, 2018

Creating environment with some pip arguments (which are not supported in requirements.txt) broken in 4.4+ #7290

Open

2 tasks

hackalog mentioned this pull request Jun 20, 2018

Use environment.yml mechanism for specifying conda dependencies drivendata/cookiecutter-data-science#124

Closed

matthewfeickert mentioned this pull request Jan 11, 2019

Possibility of making MLS a PyPI package? dkirkby/MachineLearningStatistics#3

Closed

sanzoghenzo mentioned this pull request Nov 2, 2020

convert poetry sources to pip --index-url dojeda/poetry2conda#12

Open

mfansler mentioned this pull request Dec 15, 2020

no such option: --extra-index-url http://localhost:8080 / CondaValueError: pip returned an error #10415

Closed

loikein mentioned this pull request Jun 10, 2021

Debugging workflow: rearrange dependencies; bump rpy2 version; use mamba OpenSourceEconomics/econsa#85

Merged

github-actions bot added the locked [bot] locked due to inactivity label Sep 14, 2021

github-actions bot locked as resolved and limited conversation to collaborators Sep 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow pip requirements.txt syntax in environment files #3969

Allow pip requirements.txt syntax in environment files #3969

sdvillal commented Nov 30, 2016 •

edited by kalefranz

sdvillal commented Nov 30, 2016

sdvillal commented Dec 26, 2016 •

edited

tdhopper commented Jan 3, 2017

sdvillal commented Jan 3, 2017 •

edited

tdhopper commented Jan 4, 2017

sdvillal commented Jan 9, 2017

sdvillal commented Jan 9, 2017

codecov bot commented Feb 24, 2017

codecov bot commented Feb 24, 2017

codecov bot commented Mar 8, 2017

codecov bot commented Apr 4, 2017

kalefranz left a comment

sdvillal commented Apr 27, 2017

github-actions bot commented Sep 14, 2021

Allow pip requirements.txt syntax in environment files #3969

Allow pip requirements.txt syntax in environment files #3969

Conversation

sdvillal commented Nov 30, 2016 • edited by kalefranz

sdvillal commented Nov 30, 2016

sdvillal commented Dec 26, 2016 • edited

tdhopper commented Jan 3, 2017

sdvillal commented Jan 3, 2017 • edited

tdhopper commented Jan 4, 2017

sdvillal commented Jan 9, 2017

sdvillal commented Jan 9, 2017

codecov bot commented Feb 24, 2017

Codecov Report

codecov bot commented Feb 24, 2017

Codecov Report

codecov bot commented Mar 8, 2017

Codecov Report

codecov bot commented Apr 4, 2017

Codecov Report

kalefranz left a comment

Choose a reason for hiding this comment

sdvillal commented Apr 27, 2017

github-actions bot commented Sep 14, 2021

sdvillal commented Nov 30, 2016 •

edited by kalefranz

sdvillal commented Dec 26, 2016 •

edited

sdvillal commented Jan 3, 2017 •

edited