Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`conda env create --file` errors one package at a time #6073

Open
mivade opened this issue Oct 4, 2017 · 22 comments

Comments

Projects
None yet
@mivade
Copy link

commented Oct 4, 2017

When trying to create a new environment from a environment YAML file that was created on one operating system, I get errors such as the following when running on another OS:

ResolvePackageNotFound: 
  - pango 1.39.0 1

The problem is that as soon as I comment the offending package out (since it is likely not a hard requirement, but rather just something that happened to be installed by whomever created the environment file), I just get another similar error for another package.

Considering it can take 10s of seconds before getting this error, it would be especially helpful to be presented with a list of all packages that can't be found instead of having to hunt them one at a time.

@MichaelJendryke

This comment has been minimized.

Copy link

commented Nov 29, 2017

+1

@matthiasroder

This comment has been minimized.

Copy link

commented Dec 9, 2017

I run into the same issue.

conda env create -f environment.yml -v

gives me:

Traceback (most recent call last): File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/exceptions.py", line 640, in conda_exception_handler return_value = func(*args, **kwargs) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda_env/cli/main_create.py", line 108, in execute installer.install(prefix, pkg_specs, args, env) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda_env/installers/conda.py", line 32, in install channel_priority_map=_channel_priority_map) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 475, in install_actions_list dists_for_envs = determine_all_envs(r, specs, channel_priority_map=channel_priority_map) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 540, in determine_all_envs spec_for_envs = tuple(SpecForEnv(env=p.preferred_env, spec=p.name) for p in best_pkgs) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 540, in <genexpr> spec_for_envs = tuple(SpecForEnv(env=p.preferred_env, spec=p.name) for p in best_pkgs) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/plan.py", line 539, in <genexpr> best_pkgs = (r.index[r.get_dists_for_spec(s, emptyok=False)[-1]] for s in specs) File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/conda/resolve.py", line 563, in get_dists_for_spec raise ResolvePackageNotFound([(ms,)]) ResolvePackageNotFound: - backports 1.0 py27h6492d98_1

I suspect that this has something to do with "channels" and how conda sorts dependencies. I therefore wanted to see whether the situation would be solved by using the old "sorting algorithm":

conda config --set channel_priority false

and then again:

conda env create -f environment.yml -v

to no avail.

@colllin

This comment has been minimized.

Copy link

commented Jan 11, 2018

Is anyone looking into this? I am experiencing the same problem after exporting a conda environment on macOS and then trying to reproduce the environment on Ubuntu 16.04. Actually this is literally the only thing I did.

  • I started on macOS,
  • installed some dependencies,
  • realized the data was too big,
  • exported the environment,
  • spun up an Ubuntu instance on EC2,
  • cloned the repo,
  • tried to re-create the conda environment.

I exported using conda env export > environment.yml, and tried to re-create using conda env create --name my-project --file environment.yml. I receive the same error as reported by OP:

ResolvePackageNotFound: 
  - ca-certificates 2017.08.26 ha1e5d58_0

I also tried to first create a conda environment without the file, then install from the file, with a different error:

$ conda create --name my-project `grep -o 'python=[0-9.]\+' environment.yml`
$ conda install --file environment.yml
CondaValueError: could not parse 'name: my-project' in: environment.yml

This is the same error I receive if I try to create the environment from the file using conda create rather than conda env create.

@kalefranz

This comment has been minimized.

Copy link
Member

commented Jan 11, 2018

Oftentimes build numbers don't match across platforms. Use the --no-builds flag with conda env export.

conda env export --no-builds > environment.yml
@mivade

This comment has been minimized.

Copy link
Author

commented Jan 11, 2018

What I have generally been doing to work around this is to hand craft environment files rather than using conda env export and only including packages that I'm explicitly depending on. This is not always ideal (especially for things that might have a lot of dependencies), but it has been working well enough so far.

@colllin

This comment has been minimized.

Copy link

commented Jan 11, 2018

Thanks for the --no-builds tip, @kalefranz. That seemed promising. I believe this changes the problem, but I still have similar looking errors, albeit a much smaller list of failed packages:

ResolvePackageNotFound: 
  - appnope=0.1.0
  - libcxx=4.0.1
  - libgfortran=3.0.1
  - libcxxabi=4.0.1

Note: I also ran conda update conda, which updated me to 4.4.7 (miniconda and anaconda don't ship with latest conda?).

Is there a simple explanation here? The linux channels are not in-sync with the macOS channels? Or... I'm fairly new to python... Can package dependencies vary depending on the destination platform?

@mivade, good idea... Glad to hear it's a working path forward. It kind of defeats the purpose of using an environment manager, though — even if I am open to it, it would be ridiculous to try to get my team on-board, and expect them to learn how to do it and get it right, with complexities such as channels, etc. But I'm not sure what other option I have right now!

For reference, I believe npm install --save and yarn add behave similarly to what @mivade describes. When you explicitly install/add a package, it only writes that package to your manifest, but will also install any dependencies. The dependencies don't need to be written, because they are just that — dependencies — and can be determined dynamically at install-time. I believe this is a better pattern than what is currently followed by conda env export, which appears to be more equivalent to a lock-file, which works great in node-land, but apparently doesn't work so well in python-land.

I have already wasted too much time on these issues to dig any further on my own. If you have other ideas, I will definitely give them a shot.

@kalefranz

This comment has been minimized.

Copy link
Member

commented Jan 11, 2018

The reason this issue is tagged as a feature request and with the “environment-spec” tag is because we know there’s quite a bit of work we need to do here to improve the user experience.

Indeed conda env export should only include, or have a mode that only includes, “user-provided” specs, and not the equivalent of a lock file that lists every package in the environment.

@maxwellmckinnon

This comment has been minimized.

Copy link

commented May 20, 2018

I also am blocked by the libcxx issue

Using Anaconda API: https://api.anaconda.org
Fetching package metadata .............

ResolvePackageNotFound:
  - libcxx 4.0.1*`

Like the original poster said, very aggravating that it takes 10s to see the failure, then fixing it just brings on more.

Can't there be a switch to just install everything possible from the environment, then alert which ones failed afterward? This defeats the #1 promise of anaconda being a solution to the "runs on my computer" problem. The biggest/only thing I want from anaconda is the export reqs and import reqs to work flawlessly.

Is there another workflow I'm missing that is the intended path of deploying code to another computer? I thought it was supposed to be conda env export > env.yml on one, conda create env -f env.yml on the other.

@gabegm

This comment has been minimized.

Copy link

commented May 30, 2018

Have the same problem, had to manually remove the dependencies causing the error from the env file.

ResolvePackageNotFound:

  • libcxxabi=4.0.1
  • libcxx=4.0.1

Exported the env from macOS and tried to import on Linux

@pattersongp

This comment has been minimized.

Copy link

commented Jul 8, 2018

+1

@leemengtaiwan

This comment has been minimized.

Copy link

commented Jul 18, 2018

Exactly same problem like @gabegm even if I used

conda env export --no-builds > environment.yaml

Had to manually remote the dependencies

libcxxabi=4.0.1
libcxx=4.0.1

which should be handle more gracefully by conda.

@ryanrunchey

This comment has been minimized.

Copy link

commented Jul 26, 2018

Same problem

conda env create --force -f=/Users/ryanrunchey/Downloads/gcloud_ryanrunchey_starling_conda_env_base_2018-07-25/environment.yml -n gcloud_starling_base

Using Anaconda API: https://api.anaconda.org
Fetching package metadata .............

ResolvePackageNotFound:

  • gensim 3.5.0 py36_0
@howie

This comment has been minimized.

Copy link

commented Aug 13, 2018

Same problem on ubuntu

conda.exceptions.ResolvePackageNotFound:

  • libcxxabi=4.0.1
  • libgfortran=3.0.1
  • libcxx=4.0.1
@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2018

@howie, you cannot share env files between OSes, your env file has come from macOS here.

@leemengtaiwan, you said:

which should be handle more gracefully by conda.

There's nothing conda can do about this, environment files are there to explicitly lock the exact package set. Now that's going to differ across OSes for obvious reasons.

@mingwandroid

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2018

My suggestion is that you generate an env file on each OS and then update your tooling to use the appropriate one given the current OS.

@howie

This comment has been minimized.

Copy link

commented Aug 13, 2018

@mingwandroid thanks for your information 😂

@aakashns

This comment has been minimized.

Copy link

commented Feb 12, 2019

My suggestion is that you generate an env file on each OS and then update your tooling to use the appropriate one given the current OS.

@mingwandroid Is there a way to get conda to ignore the unresolved packages, and proceed with the installation?

@Arslan-Zaidi

This comment has been minimized.

Copy link

commented Feb 19, 2019

Same problem here.

ResolvePackageNotFound:

  • appnope 0.1.0
  • libcxxabi 4.0.1
  • libgfortran 3.0.1

Things I've tried:

  1. --no-builds
  2. added all channels that were present on macOS (where the env was created)
  3. manually removed the above mentioned packages from the .yml file

These steps seemed to do it, but I think it was only no. 3 that fixed the issue as I tried the first two individually to no avail.

@aakashns

This comment has been minimized.

Copy link

commented Feb 20, 2019

manually removed the above mentioned packages from the .yml file

It appears this is what most people (including me) end up doing most of the time. conda should probably add a --ignore-missing flag to do this automatically

@sheegl

This comment has been minimized.

Copy link

commented Feb 28, 2019

I came across this issue when trying to export my environment.yml and run on other systems. I was mainly interested in getting version numbers for my dependencies. So I wrote a quick dirty script that exports the environment, picks the version numbers and adds them to the environment.yml. This doesn't add any unexpected dependencies that cause inconsistencies on other systems.

#!/usr/bin/env python

import subprocess
import re
import sys

env_filepath = sys.argv[1]

with open(env_filepath, 'r') as f:
    env_yml = f.read()

env_name = re.search('^name: (.+)', env_yml, flags=re.MULTILINE).group(1)
env_export = subprocess.check_output(
    'conda env export --name ' + env_name + ' --no-builds', shell=True
).strip().decode('utf-8')

for line in env_yml.split('\n'):
    if line:
        m = re.search(line.strip() + '(==?.+)', env_export, flags=re.MULTILINE)
        if m:
            print(line + m.group(1))
        else:
            print(line)
    else:
        print()

Just pass your environment.yml path to the script. Hope this helps someone.

@johnprakashgithub

This comment has been minimized.

Copy link

commented Mar 13, 2019

Getting same error while creating conda environment

ResolvePackageNotFound:

  • libcxx
  • libcxxabi
@cognitiaclaeves

This comment has been minimized.

Copy link

commented Mar 20, 2019

+1 on the --ignore-missing suggestion. Please implement this. I can manually drop the incorrect libraries out of generated file, but this doesn't protect me from changes that can happen in future packaging in conda and conda-forge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.