Add conda-forge based CI #133

jaimergp · 2023-02-10T14:08:39Z

Description

Checklist - did you ...

Add a file to the news directory (using the template) for the next release's release notes?
Add / update necessary tests?
Add / update outdated documentation?

Comes from #120, using conda/conda#12335

…nda-forge-ci

jaimergp · 2023-02-14T23:54:58Z

Needs changes in conda/conda#12350

jaimergp · 2023-02-22T12:16:49Z

@costrouc I'll need some help debugging the errors in tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package. Maybe we can pair-debug this one? I cannot reproduce locally (it passes), so I wonder if it has to do with this change.

costrouc · 2023-02-27T02:54:43Z

I can reproduce this test with the following locally (am running on a x86 machine in the unlikely case this would be why @jaimergp could not reproduce it).

docker run --rm -it \
       -v /home/costrouc/p/conda/conda:/opt/conda-src \
       -v /home/costrouc/p/conda/conda-libmamba-solver:/opt/conda-libmamba-solver-src \
       -e TEST_SPLITS=3 \
       -e TEST_GROUP=1 \
       -e CONDA_SOLVER=libmamba \
       ghcr.io/conda/conda-ci:main-linux-python3.10-conda-forge \
       bash

Then running

sudo env CONDA_SOLVER=classic /opt/conda/condabin/conda install -p /opt/conda --file /opt/conda-libmamba-solver-src/dev/requirements.txt
/opt/conda/bin/python -m pip install /opt/conda-libmamba-solver-src --no-deps -vvv
source /opt/conda-src/dev/linux/integration.sh

I am next going to try to narrow this down.

costrouc · 2023-02-27T03:23:43Z

Okay can reproduce with a single test.

sudo env CONDA_SOLVER=classic /opt/conda/condabin/conda install -p /opt/conda --file /opt/conda-libmamba-solver-src/dev/requirements.txt
/opt/conda/bin/python -m pip install /opt/conda-libmamba-solver-src --no-deps -vvv

# taken from dev/linux/integration.sh
git config --global --add safe.directory /opt/conda-src
sudo su root -c "/opt/conda/bin/conda install -yq conda-build"
sudo /opt/conda/bin/conda install --quiet -y --file tests/requirements.txt
eval "$(sudo /opt/conda/bin/python -m conda init --dev bash)"
conda-build tests/test-recipes/activate_deactivate_package tests/test-recipes/pre_link_messages_package
conda info
export TMP=$HOME/pytesttmp
mkdir -p $TMP

# run single test
pytest tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package

From the docker image. FAILED tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package - RuntimeError: Cannot unlink unmanageable packages:

The logs are attached logs.txt

costrouc · 2023-02-27T03:45:34Z

When using pdb I was able to extract some of the state around the failure.

(Pdb) unlink_precs
IndexedSet([PrefixRecord(_hash=919153346818182340, name='urllib3', version='1.19.1', build='dev_0', build_number=0, channel=Channel("<develop>"), subdir='pypi', depends=('python 2.7.*',), constrains=('cryptography >=1.3.4', 'idna >=2.0.0', 'pyopenssl >=0.14', 'pysocks !=1.5.7,<2.0,>=1.5.6'), package_type='virtual_python_egg_link', files=(), paths_data=PathsData(paths_version=1, paths=()))])
(Pdb) link_precs
IndexedSet([PackageRecord(_hash=6144791784308069146, name='urllib3', version='1.20', build='py27_1', build_number=1, channel=Channel("conda-forge/linux-64"), subdir='linux-64', fn='urllib3-1.20-py27_1.tar.bz2', md5='0507cc609938ad8fc81c18f818b80f52', url='https://conda.anaconda.org/conda-forge/linux-64/urllib3-1.20-py27_1.tar.bz2', sha256='139f41180aebfbb75161e313daecc29bd34c5b914ac27948e6ba62d7ae8ab2ca', depends=('python 2.7*',), constrains=('python_abi * *_cp27mu',), track_features=(), license='MIT', timestamp=0, size=130718)])
(Pdb) unlink_precs[0]
PrefixRecord(_hash=919153346818182340, name='urllib3', version='1.19.1', build='dev_0', build_number=0, channel=Channel("<develop>"), subdir='pypi', depends=('python 2.7.*',), constrains=('cryptography >=1.3.4', 'idna >=2.0.0', 'pyopenssl >=0.14', 'pysocks !=1.5.7,<2.0,>=1.5.6'), package_type='virtual_python_egg_link', files=(), paths_data=PathsData(paths_version=1, paths=()))
(Pdb) unlink_precs[0].is_unmanageable
True
(Pdb) unlink_precs[0].package_type
<PackageType.VIRTUAL_PYTHON_EGG_LINK: 'virtual_python_egg_link'>
(Pdb) from conda.models.enums import PackageType
(Pdb) unlink_precs[0].package_type in PackageType.unmanageable_package_types()
True
(Pdb) self.specs_to_add
IndexedSet([MatchSpec("urllib3=1.20")])
(Pdb) self.prefix
'/home/test_user/pytesttmp/0b0a abdecf'
(Pdb) os.listdir(self.prefix + '/lib/python2.7/site-packages/')
['certifi-2020.6.20-py3.6.egg-info', 'README', 'urllib3.egg-link', 'wheel-0.37.1-py3.9.egg-info', 'setuptools-44.0.0.post20200106-py2.7.egg-info', 'wheel', 'pip-10.0.1-py2.7.egg-info', 'setuptools', 'easy-install.pth', 'pkg_resources', 'pip', 'certifi', 'easy_install.py', 'easy_install.pyc']
(Pdb) subprocess.check_output('ls -la "/home/test_user/pytesttmp/0b0a abdecf/lib/python2.7/site-packages/urllib3.egg-link"', shell=True)
b'-rw-r--r-- 1 test_user test_user 51 Feb 27 03:34 /home/test_user/pytesttmp/0b0a abdecf/lib/python2.7/site-packages/urllib3.egg-link\n'

jaimergp · 2023-02-27T10:19:14Z

Awesome! Thanks @costrouc. Look how fun this is getting now:

The line that fails is 1688:

# should raise an error
with pytest.raises(PackagesNotFoundError):
    # TODO: This raises PackagesNotFoundError, but the error should really explain
    #       that we can't install urllib3 because it's already installed and
    #       unmanageable. The error should suggest trying to use pip to uninstall it.
    stdout, stderr, _ = run_command(Commands.INSTALL, prefix, "urllib3=1.20", "--dry-run")  # <<< THIS ONE

The TODO note says that we should actually see a different exception... which is the one we are getting 😂 :

________ IntegrationTests.test_conda_pip_interop_conda_editable_package ________
Traceback (most recent call last):
  File "/opt/conda-src/tests/test_create.py", line 1688, in test_conda_pip_interop_conda_editable_package
    stdout, stderr, _ = run_command(Commands.INSTALL, prefix, "urllib3=1.20", "--dry-run")
  File "/opt/conda-src/conda/testing/integration.py", line 287, in run_command
    result = do_call(args, p)
  File "/opt/conda-src/conda/cli/conda_argparse.py", line 92, in do_call
    return getattr(module, func_name)(args, parser)
  File "/opt/conda-src/conda/notices/core.py", line 121, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda-src/conda/cli/main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "/opt/conda-src/conda/cli/install.py", line 264, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/opt/conda-src/conda/core/solve.py", line 130, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/opt/conda-src/conda/core/solve.py", line 182, in solve_for_diff
    raise RuntimeError("Cannot unlink unmanageable packages:%s"
RuntimeError: Cannot unlink unmanageable packages:
  - <develop>/pypi::urllib3-1.19.1-dev_0

However, at no point I see the conda_libmamba_solver involved in that traceback. Did you set up CONDA_SOLVER=libmamba before running the test?

Which branches are checked out in conda/conda and conda/conda-libmamba-solver, locally?

costrouc · 2023-02-27T13:50:33Z

However, at no point I see the conda_libmamba_solver involved in that traceback. Did you set up CONDA_SOLVER=libmamba before running the test?

Yeah if I look at the context around the sovler being used in the test

self = <conda_libmamba_solver.solver.LibMambaSolver object at 0x7f484b7dbd60>
update_modifier = <UpdateModifier.FREEZE_INSTALLED: 'freeze_installed'>
deps_modifier = <DepsModifier.NOT_SET: 'not_set'>
prune = <conda.auxlib._Null object at 0x7f486ddd3eb0>
ignore_pinned = <conda.auxlib._Null object at 0x7f486ddd3eb0>
force_remove = <conda.auxlib._Null object at 0x7f486ddd3eb0>
force_reinstall = False
should_retry_solve = True

Which branches are checked out in conda/conda and conda/conda-libmamba-solver, locally?

conda conda/conda@598bc26e4 (main)

conda-libmamba-solver 02423c3 (conda-forge-ci)

…able_package`

costrouc

@jaimergp looks like all test are passing and I can confirm that all packages from the base environment do come from conda-forge which was the goal of this PR. I've created an issue linked above which details the test that was disabled.

jezdez

With conda/conda#12297 being merged, it might be useful to adopt the same pattern to build the test matrix via the exclude statement.

jaimergp · 2023-02-28T18:08:16Z

We still need conda/conda#12389

jaimergp · 2023-02-28T18:16:57Z

Added #143 to track deselected upstream tests.

…er into conda-forge-ci

jaimergp · 2023-03-06T16:00:13Z

Well, we won't need to worry about that py37 error anymore because we are not supporting it anyway. See conda/conda#12436 for details.

jaimergp · 2023-03-06T17:18:13Z

See remove libxml2-2.10.3 bn 2 conda-forge/admin-requests#687 for unrelated osx errors.

jaimergp · 2023-03-06T23:32:37Z

Re-running failed jobs to see if the packaging issue was fixed.

jaimergp · 2023-03-07T08:21:39Z

@jezdez - can you take another look? This is ready for review / merge on my end, I think.

jezdez

I think this needs slight tweaks to the test matrix still, right?

.github/workflows/upstream_tests.yml

jezdez · 2023-03-07T08:59:40Z

.github/workflows/upstream_tests.yml

+          - default-channel: 'conda-forge'
+            python-version: '3.9'
+            test-type: 'unit'
+            test-group: '1'
+          - default-channel: 'conda-forge'
+            python-version: '3.9'
+            test-type: 'unit'
+            test-group: '2'
+          - default-channel: 'conda-forge'
+            python-version: '3.9'
+            test-type: 'unit'
+            test-group: '3'
+          - default-channel: 'defaults'
+            python-version: '3.9'
+            test-type: 'integration'
+            test-group: '1'
+          - default-channel: 'defaults'
+            python-version: '3.9'
+            test-type: 'integration'
+            test-group: '2'
+          - default-channel: 'defaults'
+            python-version: '3.9'
+            test-type: 'integration'
+            test-group: '3'


This should follow the same pattern as upstream, since this isn't testing the integration on conda-forge and not testing the unit tests on defaults.

Oh god it changed again in conda/conda#12413 🤦

I know :( I thought you knew about this already, I'm sorry!

Bad timing 😂 Sync'ed again, let's see how it goes.

add conda-forge based CI

1511108

conda-bot added the cla-signed [bot] added once the contributor has signed the CLA label Feb 10, 2023

jaimergp added 4 commits February 10, 2023 15:12

conda-forge images only for 3.10; adjust

9465863

fix name

5319f16

add news

6617216

Merge branch 'main' of github.com:conda/conda-libmamba-solver into co…

90b9e97

…nda-forge-ci

jaimergp added 9 commits February 21, 2023 21:28

update workflows

3d8b428

trim trim trim

c09179a

conda-forge images only on 3.10

8572131

do not parametrize this one

ed9ac10

do not split unit tests upstream

a4d6809

fix priority on windows?

998d7ff

use conda fork temporarily

1fc8324

use checkout@v3

7d52aaf

change channel & versions a bit

c48da3a

change priority for defaults

02423c3

costrouc mentioned this pull request Feb 27, 2023

Conda forge ci costrouc #139

Closed

costrouc mentioned this pull request Feb 27, 2023

Upstream conda tests/test_create.py::IntegrationTests::test_conda_pip_interop_conda_editable_package fails #141

Closed

2 tasks

Deselecting test in upstream conda `test_conda_pip_interop_conda_edit…

41fc847

…able_package`

costrouc marked this pull request as ready for review February 27, 2023 18:57

costrouc approved these changes Feb 27, 2023

View reviewed changes

jezdez requested changes Feb 28, 2023

View reviewed changes

use upstream CI as much as possible (WIP)

26b98a5

jaimergp added 2 commits February 28, 2023 18:56

deselect failing conda/conda unit tests

b205b9a

sort / categorize a bit

a00cde9

jaimergp mentioned this pull request Feb 28, 2023

Deselected upstream tests #143

Closed

43 tasks

jaimergp added 4 commits March 1, 2023 11:20

Merge branch 'main' into conda-forge-ci

dbca190

dir on windows

92c1ffb

Merge branch 'conda-forge-ci' of github.com:conda/conda-libmamba-solv…

6c511d3

…er into conda-forge-ci

move conda info to the end

31cb432

jaimergp mentioned this pull request Mar 1, 2023

Calling conda command on cmd exits without return code in CI if not called via call conda/conda#12418

Closed

costrouc mentioned this pull request Mar 6, 2023

Simplify exception parsing #103

Merged

3 tasks

jaimergp added 3 commits March 6, 2023 12:52

use branch on windows for more debugging

b4e5112

Merge branch 'main' into conda-forge-ci

fbab16b

remove py37 from matrix

aa037ee

jaimergp added 4 commits March 7, 2023 08:41

update docs

529e3e8

undo temp changes

a46e975

editable

0f1c7be

fix repo

c98e238

jezdez requested changes Mar 7, 2023

View reviewed changes

sync with upstream

9195943

jezdez approved these changes Mar 7, 2023

View reviewed changes

jezdez changed the title ~~add conda-forge based CI~~ Add conda-forge based CI Mar 7, 2023

jezdez merged commit 511cc6a into main Mar 7, 2023

jezdez deleted the conda-forge-ci branch March 7, 2023 12:35

github-actions bot added the locked [bot] locked due to inactivity label Mar 7, 2024

github-actions bot locked as resolved and limited conversation to collaborators Mar 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add conda-forge based CI #133

Add conda-forge based CI #133

jaimergp commented Feb 10, 2023 •

edited

jaimergp commented Feb 14, 2023

jaimergp commented Feb 22, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

jaimergp commented Feb 27, 2023

costrouc commented Feb 27, 2023 •

edited

costrouc left a comment •

edited

jezdez left a comment

jaimergp commented Feb 28, 2023

jaimergp commented Feb 28, 2023

jaimergp commented Mar 6, 2023

jaimergp commented Mar 6, 2023 •

edited

jaimergp commented Mar 6, 2023

jaimergp commented Mar 7, 2023

jezdez left a comment

jezdez Mar 7, 2023

jaimergp Mar 7, 2023

jezdez Mar 7, 2023

jaimergp Mar 7, 2023

Add conda-forge based CI #133

Add conda-forge based CI #133

Conversation

jaimergp commented Feb 10, 2023 • edited

Description

Checklist - did you ...

jaimergp commented Feb 14, 2023

jaimergp commented Feb 22, 2023 • edited

costrouc commented Feb 27, 2023 • edited

costrouc commented Feb 27, 2023 • edited

costrouc commented Feb 27, 2023 • edited

jaimergp commented Feb 27, 2023

costrouc commented Feb 27, 2023 • edited

costrouc left a comment • edited

Choose a reason for hiding this comment

jezdez left a comment

Choose a reason for hiding this comment

jaimergp commented Feb 28, 2023

jaimergp commented Feb 28, 2023

jaimergp commented Mar 6, 2023

jaimergp commented Mar 6, 2023 • edited

jaimergp commented Mar 6, 2023

jaimergp commented Mar 7, 2023

jezdez left a comment

Choose a reason for hiding this comment

jezdez Mar 7, 2023

Choose a reason for hiding this comment

jaimergp Mar 7, 2023

Choose a reason for hiding this comment

jezdez Mar 7, 2023

Choose a reason for hiding this comment

jaimergp Mar 7, 2023

Choose a reason for hiding this comment

jaimergp commented Feb 10, 2023 •

edited

jaimergp commented Feb 22, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc commented Feb 27, 2023 •

edited

costrouc left a comment •

edited

jaimergp commented Mar 6, 2023 •

edited