Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI Conda: Add to known-test-failures #36937

Merged
merged 9 commits into from
Feb 2, 2024

Conversation

mkoeppe
Copy link
Member

@mkoeppe mkoeppe commented Dec 20, 2023

Not all of the added failures are conda-specific, but the 6 instances of CI Conda amplify the error probability (#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure is reported) to the failures and arrange for it to be printed by the doctester as part of the [failed in baseline] message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise from failures of the Conda CI that PR authors and reviewers find distracting.

📝 Checklist

  • The title is concise, informative, and self-explanatory.
  • The description explains in detail what this PR is about.
  • I have linked a relevant issue or discussion.
  • I have created tests covering the changes.
  • I have updated the documentation accordingly.

⌛ Dependencies

@tobiasdiez
Copy link
Contributor

Ignoring the whole file just because there is a single failing test runs against the goal of making the conda tests more stable. Hence motion to close. For the same reason this is not suitable as blocker.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 21, 2023

Ignoring the whole file just because there is a single failing test

That's not at all what it does, as we have explained to you at length in #36678.

We have also explained to you at length why your approach of #36372 is not suitable.
It simply has no support by any reviewer.
It is highly inappropriate to use the blocker label to have it take effect in every PR.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 21, 2023

As I said in #36372 (comment) and before, Tobias, these continued abuses of your Triage team privileges have to stop.

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

I think that for each record of a file in .github/workflows/ci-conda-known-test-failures.json, there should be a github issue associated with it so that we track the history of the failures in the file.

Is there an issue for

    "sage.parallel.map_reduce": {
        "failed": true
    },

?

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

and for

    "sage.structure.coerce_actions": {
        "failed": true
    },

?

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

Since json does not support comments, we may add an "issue" field so that "issue: 12345" will point to the github issue with the number.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 21, 2023

fine with me, but I think i'd prefer a free-form text that can contain an URL or whatever for flexibility.

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

Yes, URL would work better.

@mkoeppe
Copy link
Member Author

mkoeppe commented Dec 21, 2023

i added this info where available to the commit messages, so you can see it via "git blame"

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

fine with me, but I think i'd prefer a free-form text that can contain an URL or whatever for flexibility.

Then

    "sage.structure.coerce_actions": {
        "failed": true
        "comment": "some text, preferably an URL"
    },

for example.

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

i added this info where available to the commit messages, so you can see it via "git blame"

My git blame only shows the sha and the committer's name and the date. But I can see them in the git log (just for this PR).

@kwankyu
Copy link
Collaborator

kwankyu commented Dec 21, 2023

@tobiasdiez Let's go this way. You may want to add those comments.

Matthias is working in #36558 to make these failures in baseline more visible to developers (without failing checks).

@tobiasdiez
Copy link
Contributor

Yeah sure, why wouldn't we want to exclude even more files from the conda tests? For consistency, you should also add a similar file-wide exclusion mechanism for the tests that are known to fail sometimes in the Build & Test.

So stupid, instead of viewing the conda environment as a means to harden the test suit, we now massively swap the failures under the carpet. And that in a way that doesn't even allow to answer the simple question "is this the only example that fails, or are other examples failing as well"!

I also don't understand why I should add these comments. My solution keeps very well track of what example fails. It's one of the major shortcomings of Mathias solution, as I've pointed out multiple times. So at the very least he should do the work himself.

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 4, 2024

Completeness is not part of the scope of this PR.

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 4, 2024

I now went through the logs and added a few more exceptions to #36960.

Thanks for doing this work. Do you have a script for this - retrieving and parsing logs?

You may be interested in:

@mkoeppe mkoeppe force-pushed the ci_conda_more_known_failures branch from 25bc49f to 3ec3e8f Compare January 14, 2024 21:55
@mkoeppe mkoeppe force-pushed the ci_conda_more_known_failures branch from 3ec3e8f to 8f4bc66 Compare January 22, 2024 01:04
Copy link

Documentation preview for this PR (built with commit 8f4bc66; changes) is ready! 🎉

@mkoeppe
Copy link
Member Author

mkoeppe commented Jan 23, 2024

Let's merge this please.

Copy link
Collaborator

@kwankyu kwankyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. Hopefully save us from chronic conda failures in CI.

vbraun pushed a commit to vbraun/sage that referenced this pull request Jan 24, 2024
    
Not all of the added failures are conda-specific, but the 6 instances of
CI Conda amplify the error probability
(sagemath#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure
is reported) to the failures and arrange for it to be printed by the
doctester as part of the `[failed in baseline]` message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise
from failures of the Conda CI that PR authors and reviewers find
distracting.

<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36936 (which refactors the `[failed in baseline]`
printing)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36937
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez
vbraun pushed a commit to vbraun/sage that referenced this pull request Jan 27, 2024
    
Not all of the added failures are conda-specific, but the 6 instances of
CI Conda amplify the error probability
(sagemath#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure
is reported) to the failures and arrange for it to be printed by the
doctester as part of the `[failed in baseline]` message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise
from failures of the Conda CI that PR authors and reviewers find
distracting.

<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36936 (which refactors the `[failed in baseline]`
printing)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36937
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez
vbraun pushed a commit to vbraun/sage that referenced this pull request Jan 29, 2024
    
Not all of the added failures are conda-specific, but the 6 instances of
CI Conda amplify the error probability
(sagemath#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure
is reported) to the failures and arrange for it to be printed by the
doctester as part of the `[failed in baseline]` message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise
from failures of the Conda CI that PR authors and reviewers find
distracting.

<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36936 (which refactors the `[failed in baseline]`
printing)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36937
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez
vbraun pushed a commit to vbraun/sage that referenced this pull request Jan 30, 2024
    
Not all of the added failures are conda-specific, but the 6 instances of
CI Conda amplify the error probability
(sagemath#36694 (comment))

We also add info (ideally a link to the GitHub Issue where the failure
is reported) to the failures and arrange for it to be printed by the
doctester as part of the `[failed in baseline]` message.

Marked as blocker so it takes effect in CI runs -- to reduce the noise
from failures of the Conda CI that PR authors and reviewers find
distracting.

<!-- ^^^^^
Please provide a concise, informative and self-explanatory title.
Don't put issue numbers in there, do this in the PR body below.
For example, instead of "Fixes sagemath#1234" use "Introduce new method to
calculate 1+1"
-->
<!-- Describe your changes here in detail -->

<!-- Why is this change required? What problem does it solve? -->
<!-- If this PR resolves an open issue, please link to it here. For
example "Fixes sagemath#12345". -->
<!-- If your change requires a documentation PR, please link it
appropriately. -->

### 📝 Checklist

<!-- Put an `x` in all the boxes that apply. -->
<!-- If your change requires a documentation PR, please link it
appropriately -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
<!-- Feel free to remove irrelevant items. -->

- [x] The title is concise, informative, and self-explanatory.
- [x] The description explains in detail what this PR is about.
- [x] I have linked a relevant issue or discussion.
- [ ] I have created tests covering the changes.
- [ ] I have updated the documentation accordingly.

### ⌛ Dependencies

<!-- List all open PRs that this PR logically depends on
- sagemath#12345: short description why this is a dependency
- sagemath#34567: ...
-->
- Depends on sagemath#36936 (which refactors the `[failed in baseline]`
printing)

<!-- If you're unsure about any of these, don't hesitate to ask. We're
here to help! -->
    
URL: sagemath#36937
Reported by: Matthias Köppe
Reviewer(s): Kwankyu Lee, Matthias Köppe, Tobias Diez
@vbraun vbraun merged commit caf7cf8 into sagemath:develop Feb 2, 2024
17 of 18 checks passed
@mkoeppe mkoeppe added this to the sage-10.3 milestone Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants