CI: Add GHA setup #702

amotl · 2022-10-15T22:15:19Z

Dear Chris,

after learning that CI builds regularly exhaust credits on Travis CI (I think that also currently happened at #700), this patch adds a corresponding configuration for GHA. It may save some credits, and aims to mitigate the hassle having to renew them regularly, and manually.

The builds just succeeded for the first time at [1-4], looking good so far. I think GHA performs well, with a total runtime of ~30 minutes, where it takes ~40 minutes on Travis CI.

With kind regards,
Andreas.

[1] https://github.com/panodata/apprise/actions/runs/3257301254/usage (33 minutes runtime)
[2] https://github.com/caronc/apprise/actions/runs/3257334731/usage (34 minutes runtime)
[3] https://github.com/caronc/apprise/actions/runs/3257397087/usage (29 minutes runtime)
[4] https://github.com/caronc/apprise/actions/runs/3257501205/usage (30 minutes runtime)

codecov-commenter · 2022-10-15T22:17:13Z

Codecov Report

Base: 100.00% // Head: 99.46% // Decreases project coverage by -0.53% ⚠️

Coverage data is based on head (cdc5bfd) compared to base (ed876d1).
Patch coverage: 50.00% of modified lines in pull request are covered.

Additional details and impacted files

@@             Coverage Diff             @@
##            master     #702      +/-   ##
===========================================
- Coverage   100.00%   99.46%   -0.54%     
===========================================
  Files          113      113              
  Lines        14532    14532              
  Branches      2959     2787     -172     
===========================================
- Hits         14532    14454      -78     
- Misses           0       68      +68     
- Partials         0       10      +10

Flag	Coverage Δ
unittests	`99.45% <50.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
apprise/plugins/NotifyMacOSX.py	`100.00% <ø> (ø)`
apprise/plugins/NotifyDBus.py	`41.66% <16.66%> (-58.34%)`	⬇️
apprise/plugins/NotifyGnome.py	`100.00% <100.00%> (ø)`
apprise/plugins/NotifyMQTT.py	`99.37% <100.00%> (-0.63%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

amotl · 2022-10-15T22:59:01Z

Unfortunately, the build lacks a few bits of coverage on NotifyDBus, see [1]. ~~Maybe ab6f0d5 from #689 will improve it.~~ Edit: This has been resolved now.

[1] https://app.codecov.io/gh/caronc/apprise/commit/d2c868ee5141ac88216b8a70235353b9939dbe34

.github/workflows/tests.yml

caronc · 2022-10-16T14:52:36Z

It's a really amazing idea idea. I love the idea of dropping 3rd party dependencies. But would it still support coverage, or would we drop that too and just make sure tests fail if coverage isn't met?

caronc · 2022-10-16T14:55:34Z

Just another thought (thinking out-loud here). The build status, coverage status and the ability to just kick a runner if it fails to try again is one positive with the current setup. I'd be curious how that changes under the new?

amotl · 2022-10-16T20:11:20Z

.github/workflows/tests.yml

+    - name: Run tests
+      run: |
+        coverage run -m pytest
+
+    - name: Process coverage data
+      run: |
+        coverage combine
+        coverage xml
+        coverage report
+
+    - name: Upload coverage data
+      uses: codecov/codecov-action@v3
+      with:
+        files: ./coverage.xml
+        flags: unittests
+        env_vars: OS,PYTHON,BARE
+        name: codecov-umbrella
+        fail_ci_if_error: false


Would [the new setup] still support coverage, or would we drop that too and just make sure tests fail if coverage isn't met?

Code coverage tracking and reporting to Codecov is fully supported and already works on the GHA builds on your repository. The semantics of all program invocations did not change at all, and the GHA recipe is merely a translation of the Travis CI + Tox recipe.

Codecov already sent a report back on this PR at #702 (comment), while Travis CI was out of service.

amotl · 2022-10-16T20:12:46Z

.github/workflows/tests.yml

+        flags: unittests
+        env_vars: OS,PYTHON,BARE
+        name: codecov-umbrella
+        fail_ci_if_error: false


We will probably want to set fail_ci_if_error: true here.

There is a minor flaw with Codecov we also experienced recently. We reported about it at crate/crate-python#451. The verified solution is to send the right CODECOV_TOKEN along, even if the repository is public and would actually not need it.

Unless the repository is configured that way, I will keep fail_ci_if_error: false.

[2022-10-18T08:51:11.864Z] ['error'] There was an error running the uploader: Error uploading to [https://codecov.io:](https://codecov.io/) Error: There was an error fetching the storage URL during POST: 404 - {'detail': ErrorDetail(string='Unable to locate build via Github Actions API. Please upload with the Codecov repository upload token to resolve issue.', code='not_found')}

-- https://github.com/caronc/apprise/actions/runs/3271932392/jobs/5382320635#step:12:43

amotl · 2022-10-16T20:20:48Z

[A few] positive things with the current setup. I'd be curious how that changes under the new?

The build status and coverage status.

Are you referring to the badges? GHA's will be ¹, and Codecov stays the same.

The ability to just kick a runner if it fails to try again.

On failed builds, you have the possibility to re-run all jobs of your workflow, or only the ones that failed.

Other than this, if you add a workflow_dispatch to your recipe, you can interactively run the workflow on any branch you choose.

 on: 
   workflow_dispatch:

There is also a feature to run nightly or weekly checks, or any other cron-like schedule. But I think you know this already, it is also used within codeql-analysis.yml.

 on: 
   schedule: 
     - cron: '0 2 * * *'

https://github.com/caronc/apprise/actions/workflows/tests.yml/badge.svg ↩

amotl · 2022-10-16T21:22:18Z

.github/workflows/tests.yml

+      # Run all jobs to completion (false), or cancel
+      # all jobs once the first one fails (true).
+      fail-fast: false


I think a good default is to use fail-fast: true. I usually only disable failing fast, if I want to get the whole picture at once about all failing tests on all environments. It is very useful if the test suite works in general, but croaks on specific environments, and you want to find out about them in one run.

In daily business, it will save a bunch of resources when the jobs on the other environments get canceled, if one environment fails.

So, I am proposing to fail-fast by default. Do you agree?

I don't see harm in the idea

Thanks, I've updated the patch accordingly.

amotl · 2022-10-16T21:27:14Z

.github/workflows/tests.yml

+      matrix:
+        os: [
+          "ubuntu-latest",
+          # "macos-latest",


After bringing in #700, I think it will probably be safe to enable testing on macOS on CI right away, if desired.

I think it is equally important to add Windows to CI, to protect against basic errors like 601f9af. Further efforts are made with #707 in this area.

On the other hand, I believe it will be crazy to run macOS- and Windows-tests on the whole test matrix. I think it will be good enough to restrain them to Python 3.10 only, for example.

It may save some credits on Travis CI.

amotl · 2022-10-18T10:28:17Z

I think this patch will be ready to go, if you are happy with it.

caronc · 2022-10-18T11:19:16Z

Done! 🚀👍

amotl force-pushed the amo/gha branch 3 times, most recently from d843b13 to d2c868e Compare October 15, 2022 22:40

amotl marked this pull request as draft October 15, 2022 22:55

amotl commented Oct 15, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

amotl force-pushed the amo/gha branch from 39946df to e0ed61d Compare October 15, 2022 23:18

amotl marked this pull request as ready for review October 15, 2022 23:24

amotl commented Oct 15, 2022

View reviewed changes

.github/workflows/tests.yml Show resolved Hide resolved

amotl requested a review from caronc October 16, 2022 00:27

amotl force-pushed the amo/gha branch from e0ed61d to e963763 Compare October 16, 2022 18:08

amotl commented Oct 16, 2022

View reviewed changes

amotl force-pushed the amo/gha branch from e963763 to cdc5bfd Compare October 16, 2022 20:58

amotl mentioned this pull request Oct 16, 2022

CI: Remove Travis CI, and tox #705

Merged

amotl commented Oct 16, 2022

View reviewed changes

amotl mentioned this pull request Oct 17, 2022

CI: Enable testing on macOS and Windows #707

Merged

amotl force-pushed the amo/gha branch 2 times, most recently from eb34249 to 5e3da19 Compare October 18, 2022 08:49

amotl mentioned this pull request Oct 18, 2022

Tests: Collection of flukes on CI #692

Open

amotl added 2 commits October 18, 2022 10:53

CI: Disable tests not working or flaky on PyPy 3.6

3688918

CI: Add GHA setup

10710e3

It may save some credits on Travis CI.

amotl force-pushed the amo/gha branch from 5e3da19 to 10710e3 Compare October 18, 2022 08:53

caronc merged commit 7c993fb into caronc:master Oct 18, 2022

caronc deleted the amo/gha branch October 18, 2022 11:18

amotl added a commit to panodata/apprise that referenced this pull request Oct 18, 2022

Switched from Travis-CI to GHA (caronc#702)

34b3998

amotl added a commit to panodata/apprise that referenced this pull request Oct 18, 2022

Switched from Travis-CI to GHA (caronc#702)

720512a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Add GHA setup #702

CI: Add GHA setup #702

amotl commented Oct 15, 2022 •

edited

codecov-commenter commented Oct 15, 2022 •

edited

amotl commented Oct 15, 2022 •

edited

caronc commented Oct 16, 2022

caronc commented Oct 16, 2022

amotl Oct 16, 2022 •

edited

amotl Oct 16, 2022

amotl Oct 18, 2022 •

edited

amotl commented Oct 16, 2022 •

edited

amotl Oct 16, 2022 •

edited

caronc Oct 17, 2022

amotl Oct 18, 2022

amotl Oct 16, 2022 •

edited

amotl Oct 17, 2022 •

edited

amotl commented Oct 18, 2022

caronc commented Oct 18, 2022

CI: Add GHA setup #702

CI: Add GHA setup #702

Conversation

amotl commented Oct 15, 2022 • edited

codecov-commenter commented Oct 15, 2022 • edited

Codecov Report

amotl commented Oct 15, 2022 • edited

caronc commented Oct 16, 2022

caronc commented Oct 16, 2022

amotl Oct 16, 2022 • edited

Choose a reason for hiding this comment

amotl Oct 16, 2022

Choose a reason for hiding this comment

amotl Oct 18, 2022 • edited

Choose a reason for hiding this comment

amotl commented Oct 16, 2022 • edited

Footnotes

amotl Oct 16, 2022 • edited

Choose a reason for hiding this comment

caronc Oct 17, 2022

Choose a reason for hiding this comment

amotl Oct 18, 2022

Choose a reason for hiding this comment

amotl Oct 16, 2022 • edited

Choose a reason for hiding this comment

amotl Oct 17, 2022 • edited

Choose a reason for hiding this comment

amotl commented Oct 18, 2022

caronc commented Oct 18, 2022

amotl commented Oct 15, 2022 •

edited

codecov-commenter commented Oct 15, 2022 •

edited

amotl commented Oct 15, 2022 •

edited

amotl Oct 16, 2022 •

edited

amotl Oct 18, 2022 •

edited

amotl commented Oct 16, 2022 •

edited

amotl Oct 16, 2022 •

edited

amotl Oct 16, 2022 •

edited

amotl Oct 17, 2022 •

edited