Tests failing after libmamba 1.4.2 release #186
Assignees
Labels
severity::2
critical; broken functionality with an unacceptably complex workaround
source::contributor
created by a frequent contributor
type::bug
describes erroneous operation, use severity::* to classify the type
Milestone
Comments
costrouc
added
the
type::bug
costrouc
changed the title
jaimergp
changed the title
|
Summarizing findings in #187:
We are skipping 1.4.2 in CI now to make things pass but it's very likely this will break again in an eventual 1.4.3 or 1.5.0. We need to establish what's exactly broken and communicate with the mamba team. |
This comment was marked as duplicate.
This comment was marked as duplicate.
2 tasks
|
We're investigating this issue and once we understand the implications we will consider patching repodata. Currently we don't think this is a serious breakage in older releases. |
|
I'm using #191 (comment) workflow to git bisect the issue. Looking between libmamba 1.4.1 <-> 1.4.2 release. My bisection log git bisect start 'libmambapy-1.4.2' 'libmambapy-1.4.1'
30ca2e2d bad release libmamba 1.4.2, libmambapy 1.4.2, mamba 1.4.2, micromamba 1.4.2
6ed35f84 good release libmamba 1.4.1, libmambapy 1.4.1, mamba 1.4.1, micromamba 1.4.1
git bisect good a3597e33ff4a7a184f97790a926094f1512f2a1e
a3597e33 good fix: Improve message after the env creating with micromamba (#2425)
git bisect bad d12ea0179558682bbb0b1aec96ab689aea6889b2
d12ea017 bad Fix segfault in add_pin/all_problems_structured (#2428)
git bisect bad 0345b9d7f8d65b0795c4ba73a255cc06c7b23b13
0345b9d7 bad Use custom function to properly parse matchspec (#2433)
git bisect good e5451c94f9031792341074b7d0742b2f455217dc
e5451c94 good Replaced libtool 2.4.6_9 with libtool 2.4.7-3 in vcpkg builds (#2439)
0345b9d7f8d65b0795c4ba73a255cc06c7b23b13 is the first bad commitAll tests pass on All tests fail on So @jaimergp for your meeting tomorrow the failing commit is mamba-org/mamba@0345b9d |
|
Excellent job @costrouc! I updated the mamba team as well. It's a pity that the PR is a single commit so we can't bisect further, but maybe we can start reverting single blocks in the diff and see if the test starts passing. For example, this line modifies the job_flag a bit, if you compare it to its equivalent predecessor (note the extra |
jezdez
added
severity::2
|
Appologies ahead of time for the long post and some of it may/may not be helpful. I am showing the steps I ran to reproduce and at the end I am investigating the different integers supplied to libsolv. Research into libmamba breakage. Check the git branches:
$ docker run -it --rm --platform=linux/amd64 \
-v /home/costrouc/p/conda/conda:/opt/conda-src \
-v /home/costrouc/p/conda/conda-libmamba-solver:/opt/conda-libmamba-solver-src \
-v /home/costrouc/p/mamba-org/mamba:/opt/mamba-src \
ghcr.io/conda/conda-ci:main-linux-python3.10-conda-forge bash
$ source /opt/conda-libmamba-solver-src/dev/linux/bashrc.shNot run the test that will fail. cd /opt/conda-src
pytest tests/core/test_solve.py::test_force_remove_1 # PASS
CONDA_SOLVER=libmamba pytest tests/core/test_solve.py::test_force_remove_1 # FAIL
pytest tests/core/test_solve.py::test_aggressive_update_packages # PASS
CONDA_SOLVER=libmamba pytest tests/core/test_solve.py::test_aggressive_update_packages # FAILThey both PASS when CONDA_SOLVER=classic and FAIL when CONDA_SOLVER=libmamba. Both are quite convoluted tests and it is hard to identify exactly what causes the failure. So my goal is to find an approach that in the future we could also use to help debug issues. Testing with the latest on mamba and I get the same results:
Now testing where I know the exact breaking changes are and can still reproduce the error.
I had an idea of being able to detect changes by looking at On line ~480 I am directly looking at the information being passed to libsolv in the try_solve function. He we can directly inspect what is being passed to libsolv in #include <iostream>
#include <list>
bool MSolver::try_solve()
{
...
// DEBUG
std::cout << "\n >>> DEBUG SOLVER STATE <<<" << std::endl;
std::cout << "M_FLAGS: ";
for (auto v : m_flags)
std::cout << "(" << v.first << " " << v.second << ") ";
std::cout << std::endl;
std::cout << "M_JOBS: ";
const auto l = std::list<::Id>(m_jobs->begin(), m_jobs->end());
for (auto v : l)
std::cout << v << " ";
std::cout << std::endl;
...
}I've checked this is reproducible in that I can run the same test multiple times with same integers being produced. First the output when test pass using the commit $ CONDA_SOLVER=libmamba pytest tests/core/test_solve.py::test_force_remove_1 -s
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 259 -2147482764
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 2563 -2147483603 2563 -2147482889 262659 214
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 259 -2147482764 768 1145 259 -2147482763 768 1147 259 -2147482762 768 1149 259 -2147482761 768 1151 259 -2147482760 768 1153 259 -2147482759 768 1155 2560 1157 131843 218When it is passing. Also I see the following warning when the test was passing libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.
error libmamba Selected channel specific (or force-reinstall) job, but package is not available from channel. Solve job will fail.Now for the failing test when I set the right git commit $ CONDA_SOLVER=libmamba pytest tests/core/test_solve.py::test_force_remove_1 -s
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 259 -2147482764
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 2563 -2147483603 2563 -2147482889 262659 214
...
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 259 1145 771 1145 259 1147 771 1147 259 1149 771 1149 259 1151 771 1151 259 1153 771 1153 259 1155 771 1155 2563 1157 131843 218
>>> DEBUG SOLVER STATE <<<
M_FLAGS: (1 1) (4 1) (26 1) (24 1) (12 1)
M_JOBS: 3331 216 3331 220 3331 222 3331 224 3331 226 3331 228 2819 216 2819 220 2819 222 2819 224 2819 226 2819 228 771 216 771 220 771 222 771 224 771 226 771 228 2563 1157 131843 218 |
|
The different is in the third stage of the test:
vs.
Many of the numbers are the same. |
|
Continuing on with the work. Here I have checked in two branches where I can compare the libsolve state for a passing and failing verison of the test with only the changes from the commit mamba-org/mamba@0345b9d. The test I run for each is (base) test_user@3c85ea9b1da7:/opt/conda-src$ CONDA_SOLVER=libmamba pytest tests/core/test_solve.py::test_force_remove_1 -s | python /opt/mamba-src/parse_libsolve_state.py For the failing test I see costrouc/mamba@99ac04e Spent quite awhile trying to find the definition of The passing build costrouc/mamba@68c19e5 |
costrouc
added a commit
to costrouc/mamba
that referenced
this issue
May 27, 2023
costrouc
added a commit
to costrouc/mamba
that referenced
this issue
May 27, 2023
|
Looking at this in detail Stage 1Identical Stage 2Identical Stage 3Looking line by line how they are different
|
|
So to me it looks like the issue is with With this info I made a small change costrouc/mamba@f72a63d which causes the test to pass. The libsolve state is now identical to the working "good" branch where the test passes. Both tests now pass. So the "fix" is to throw away the channel when using the |
|
Also wanted to note here the scope of the issue. Looking at costrouc/mamba@f72a63d which "fixes" these failing tests. This breakage only occurs when performing an update and "complex" matchspecs are specified e.g. not just package name and version constraint (example |
3 tasks
|
All tests broken by 1.4.2 now work with #270 (and libmamba >=1.5.1). Closing here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
What happened?
The following tests are failing in CI right after we released the 23.3.0 conda-libmamba-solver. It is hard to assign a definitive issue title since I still don't understand the core issue. https://github.com/conda/conda-libmamba-solver/actions/runs/4703354936/jobs/8341750886.
These errors happen on all OSs. We are only seeing this on conda-forge but this is due to 1.4.2 not being in defaults yet.
The following unit-tests fail on conda-libmamba-solver
The text was updated successfully, but these errors were encountered: