Revert test matrix back to manual #111

lilyminium · 2024-04-07T12:43:36Z

Relates to #110

Changes made in this Pull Request:

PR Checklist

lilyminium · 2024-04-22T10:27:03Z

Annoyingly, I had to add a line allowing coverage failures due to the current issues -- perhaps this can be taken out if/when those get resolved.

lilyminium · 2024-04-22T10:28:58Z

@IAlibay this is ready for review if you have a chance to have a look! I verified that tests failed before pushing the fix, and tests seem to be passing now that the coverage step is allowed to fail. Apologies it took longer than expected.

IAlibay

Apologies for the briefness, sending this on the go from a phone.

A few initial review comments / questions.

IAlibay · 2024-04-22T10:41:09Z

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

@@ -120,6 +130,10 @@ jobs:
      if: github.repository == '{{ cookiecutter.github_host_account }}/{{ cookiecutter.repo_name }}' && github.event_name != 'schedule'
      uses: codecov/codecov-action@v3
      with:
+        # codecov can be flaky and result in spurious failures


tbd - api token entry

The main failure here is a lack of API token, not having it often cause failures even when not on forks. How do we best ensure that folks do this?

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

lilyminium · 2024-05-26T11:15:17Z

Ok, I've set up a CODECOV token for the https://github.com/MDAnalysis/mdakit-cookie repository and upgrading the act action appears to get CI working again. I've also added some comments to set fail_ci_if_error: false for downstream kits that may need it for forks.

Edit: nevermind now CI isn't running for some reason. -_-

This reverts commit 57050d2.

lilyminium · 2024-05-27T01:53:53Z

Unless I'm missing something, the codecov upload action may be forever doomed to failure as it's not a valid GitHub run. The CI now gets around it by on-the-fly editing the CI to accept a failed codecov action.

This reverts commit 2601768.

This reverts commit 51bdcff.

* tmp let CI run without codecov * check directory exists * make package and repo name markedly different * tmp add assert False * fix env value? * fix path? * fix var name * add more listing * even more listing * am doofus * rm some temporary changes * add list back in * tmp try fixes to check ci * tmp simplify matrix * allow os * allow mdanalysis versions * use manual python matrix * Revert "tmp try fixes to check ci" This reverts commit 613debe. * Revert "Revert "tmp try fixes to check ci"" This reverts commit 5133e40. * Revert "Revert "Revert "tmp try fixes to check ci""" This reverts commit 38cf157.

See MDAnalysis/cookiecutter-mdakit#111

IAlibay

Looks good, just a couple of really quick things - thanks so much for all your work on this @lilyminium !

I'm going to have a brief look at what's been happening in codecov land re: tokens, if there's anything interesting I'll add a separate review, if not then the current solution is imho as good as it'll get.

IAlibay · 2024-05-27T08:41:14Z

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

@@ -123,11 +113,22 @@ jobs:
        file: coverage.xml
        name: codecov-{{ '${{ matrix.os }}' }}-py{{ '${{ matrix.python-version }}' }}
        verbose: True
+        # to upload coverage reports, set a secret called CODECOV_TOKEN


Do we have a set of setup instruction docs somewhere where we can put this? (not in this PR but let's raise an issue to get that done at some point :) )

IAlibay · 2024-05-27T08:52:37Z

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

+        # to upload coverage reports, set a secret called CODECOV_TOKEN
+        # in the repository settings
+        # (Obtain this from the Codecov website after setting up the repository there)
+        token: {{ '${{ secrets.CODECOV_TOKEN }}' }}


I think there's a thing with codecov v4 about this - not sure if we should/want to bump up the version (it might make things break even more!)

IAlibay · 2024-05-27T08:52:53Z

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

+        fail_ci_if_error: true
+        # use OpenID Connect (OIDC) if you have this enabled -- this will ignore
+        # the `token` argument above
+        use_oidc: true


Are the cookiecutter tests currently failing because this is set to true?

Follow up: should this be true by default?

Probably related; TIL use_oidc is v4 only.

They're currently failing because of the issues solved in #115! I did test that the fix in #115 resolves this. The codecov action currently gets edited on the fly so failure is ignored.

Probably related; TIL use_oidc is v4 only.

Interesting, the logs seemed to indicate that codecov was picking up the OIDC token and using it to connect.

Follow up: should this be true by default?

I would assume that OIDC is a better means of validation than the plain CODECOV token, if users can do so -- that being said I'm not quite sure how it well it works for forks. I gave it a try just in case it magically helped the upload. Happy to remove and look at it more in detail later for a simpler PR.

.github/workflows/gh-ci.yaml

IAlibay · 2024-05-27T08:58:43Z

.github/workflows/gh-ci.yaml

If you can be bothered, there's a few action versions that could be bumped up here (upload, download, and checkout can all go to v4, setup-miniconda I think would be good to bump to v3 [I know it doesn't play nice with macos-latest osx-arm64 runners])

IAlibay · 2024-05-27T09:00:12Z

{{cookiecutter.repo_name}}/.github/workflows/{{cookiecutter._ci_name}}.yaml

Something I had to do with the broken kits was bump up the conda-incubator/setup-miniconda version.

I found a lot of times that v2 would fail with osx-arm64 for some reason or another.

IAlibay · 2024-05-27T09:30:52Z

@lilyminium - just had a brief look at the codecov action history, this came up: codecov/codecov-action#1404

I've not had time to test it out (or read up on all the history related to it 😅), but it looks like the latest version of v4 might fix all the problems? Is it worth trying here?

lilyminium · 2024-05-27T14:11:55Z

Upgrading to codecov v4 kicked off a whole bunch of things (needing to upgrade act, leading to a bunch of issues where the most recent is libmamba failing to install MDAnalysis into the image due to lack of disk space). I think I might do that in a separate PR so we can firstly get Lawson's fix in and not overly complicate this one.

This reverts commit 2ecd7b0.

lilyminium · 2024-05-27T14:26:38Z

Apologies @IAlibay -- as mentioend in a previous comment IMO it might be easier to address the version concerns in a new PR, as this one unblocks #115 slightly it'd be good to get it in!

IAlibay

Sounds good, probably lots of smaller PRs for each bump is better anyways, sorry for the wild goose chase!

lilyminium · 2024-05-27T21:12:51Z

Thanks @IAlibay for all the work you put into reviewing!

check tests fail

1e50018

lilyminium mentioned this pull request Apr 7, 2024

Decouple CI MDAnalysis/mdahole2#58

Draft

4 tasks

lilyminium added 6 commits April 18, 2024 15:06

update job name

51d4010

reset name back to main-tests

0d44293

rm dev

8fb7da0

change python matrix to manual

d744ba6

is codecov the problem here?

102549c

dont fail ci for codecov

a62d745

lilyminium changed the title ~~Check tests fail~~ Revert test matrix back to manual Apr 22, 2024

lilyminium marked this pull request as ready for review April 22, 2024 10:27

lilyminium requested a review from IAlibay April 22, 2024 10:28

IAlibay requested changes Apr 24, 2024

View reviewed changes

lilyminium mentioned this pull request Apr 27, 2024

[CookieCutter CI] Decouple "latest" and "develop" MDAnalysis compatibility checks #110

Open

lilyminium added 6 commits April 27, 2024 15:28

remove all dynamic python versions

c9de494

add codecov token?

16324a5

escape braces

d449d2d

ignore codecov failure?

bfa4b66

upgrade act and allow codecov failure again

5186c18

add more comment to CI

007e0a1

lilyminium added 4 commits May 26, 2024 21:20

try changing the name of the package

57050d2

Revert "try changing the name of the package"

956b25d

This reverts commit 57050d2.

assert False

66a4245

tmp list directory

6b36c22

lilyminium marked this pull request as draft May 26, 2024 11:39

lilyminium added 2 commits May 26, 2024 21:43

downgrade act?

b860ccf

rm assert False

2601768

lilyminium closed this May 26, 2024

lilyminium added 2 commits May 27, 2024 11:05

try oidc?

9c7a1dd

give up and try editing file to ignore upload failure

ceec4a3

lilyminium marked this pull request as ready for review May 27, 2024 01:52

lilyminium and others added 3 commits May 27, 2024 11:54

Revert "rm assert False"

51bdcff

This reverts commit 2601768.

Revert "Revert "rm assert False""

537f817

This reverts commit 51bdcff.

orbeckst added a commit to MDAnalysis/mdgeomkit that referenced this pull request May 27, 2024

revert test matrix to manual

75d63ec

See MDAnalysis/cookiecutter-mdakit#111

orbeckst added a commit to MDAnalysis/mdgeomkit that referenced this pull request May 27, 2024

revert test matrix to manual

0f5f5e6

See MDAnalysis/cookiecutter-mdakit#111

lilyminium mentioned this pull request May 27, 2024

[WIP] ModuleNotFound Error Fix #115

Merged

5 tasks

IAlibay reviewed May 27, 2024

View reviewed changes

lilyminium added 8 commits May 27, 2024 23:07

try v4

fcb4c12

bump act action?

3dd02ed

upgrade versions

2ecd7b0

why isnt this running?

c2109f4

reorder os?

38b8320

upgrade act all the way

ca96eaf

its called act-latest now?

ba6795f

oops wrong place

e879faf

lilyminium force-pushed the decouple-latest-and-dev-testing branch from dc547e2 to e879faf Compare May 27, 2024 13:43

add p=false

c871ad8

lilyminium added 2 commits May 28, 2024 00:12

downgrade codecov back to v4, act back to 0.2.40, see how that goes

e1d47d5

Revert "upgrade versions"

a685a34

This reverts commit 2ecd7b0.

IAlibay approved these changes May 27, 2024

View reviewed changes

lilyminium merged commit 961a9f8 into main May 27, 2024
9 of 12 checks passed

lilyminium deleted the decouple-latest-and-dev-testing branch May 27, 2024 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert test matrix back to manual #111

Revert test matrix back to manual #111

lilyminium commented Apr 7, 2024

lilyminium commented Apr 22, 2024

lilyminium commented Apr 22, 2024

IAlibay left a comment

IAlibay Apr 22, 2024

lilyminium commented May 26, 2024 •

edited

lilyminium commented May 27, 2024

IAlibay left a comment

IAlibay May 27, 2024

IAlibay May 27, 2024

IAlibay May 27, 2024

IAlibay May 27, 2024

IAlibay May 27, 2024

lilyminium May 27, 2024 •

edited

IAlibay May 27, 2024

IAlibay May 27, 2024

IAlibay commented May 27, 2024

lilyminium commented May 27, 2024

lilyminium commented May 27, 2024

IAlibay left a comment

lilyminium commented May 27, 2024

Revert test matrix back to manual #111

Revert test matrix back to manual #111

Conversation

lilyminium commented Apr 7, 2024

Changes made in this Pull Request:

PR Checklist

lilyminium commented Apr 22, 2024

lilyminium commented Apr 22, 2024

IAlibay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilyminium commented May 26, 2024 • edited

lilyminium commented May 27, 2024

IAlibay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lilyminium May 27, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IAlibay commented May 27, 2024

lilyminium commented May 27, 2024

lilyminium commented May 27, 2024

IAlibay left a comment

Choose a reason for hiding this comment

lilyminium commented May 27, 2024

lilyminium commented May 26, 2024 •

edited

lilyminium May 27, 2024 •

edited