ci: tell codecov to wait_for_ci to avoid flappy reports #1160

moul · 2023-09-21T12:50:24Z

I've conducted tests on my fork, which you can find at https://github.com/moul/gno.

The 'if' condition draws inspiration from @ajnavarro's contributions, so a shoutout to him for that.

Additionally, @ajnavarro pointed out gaps in our testing for certain packages. We need to strategize on expanding our test coverage to address these overlooked areas.

I suggest waiting for his alternative work before considering merging mine.

codecov · 2023-09-21T13:13:30Z

Codecov Report

Patch coverage has no change and project coverage change: -0.05% ⚠️

Comparison is base (61f3ada) 47.08% compared to head (968c04c) 47.03%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1160      +/-   ##
==========================================
- Coverage   47.08%   47.03%   -0.05%     
==========================================
  Files         365      365              
  Lines       61156    61156              
==========================================
- Hits        28793    28765      -28     
- Misses      30011    30039      +28     
  Partials     2352     2352

Flag	Coverage Δ
gno.land-_test.gnokey	`0.00% <ø> (ø)`
gno.land-_test.gnoland	`88.14% <ø> (ø)`
gno.land-_test.pkgs	`27.88% <ø> (ø)`
gnovm-_test.cmd	`45.89% <ø> (ø)`
gnovm-_test.gnolang.native	`63.09% <ø> (ø)`
gnovm-_test.gnolang.other	`16.63% <ø> (ø)`
gnovm-_test.gnolang.pkg0	`17.98% <ø> (ø)`
gnovm-_test.gnolang.pkg1	`8.21% <ø> (ø)`
gnovm-_test.gnolang.pkg2	`9.87% <ø> (ø)`
gnovm-_test.gnolang.realm	`41.68% <ø> (ø)`
gnovm-_test.gnolang.stdlibs	`53.53% <ø> (ø)`
gnovm-_test.pkg	`25.96% <ø> (ø)`
tm2-_test.flappy	`∅ <ø> (?)`
tm2-_test.pkg.amino	`58.32% <ø> (ø)`
tm2-_test.pkg.bft	`63.69% <ø> (-0.24%)`	⬇️
tm2-_test.pkg.others	`59.23% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul · 2023-09-21T16:49:07Z

.github/codecov.yml

@@ -1,7 +1,7 @@
 codecov:
  require_ci_to_pass: false
  notify:
-    wait_for_ci: false
+    wait_for_ci: true


this is the trick; making codecov wait for all jobs to finish before generating a report.

moul · 2023-09-21T16:49:28Z

.github/codecov.yml

+    project:
+      default:
+        target: auto
+        threshold: 0.5%


i think this one should stay low like 0.5, or even smaller.

moul · 2023-09-21T16:49:53Z

.github/codecov.yml

+    patch:
+      default:
+        target: auto
+        threshold: 1%


while i think this one could potentially be increased, because I think it works per file, and sometimes, adding a unique line may be seen as 5% change.

Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul · 2023-09-21T16:55:05Z

.github/workflows/db-tests.yml

@@ -20,7 +20,7 @@ jobs:
    strategy:
      fail-fast: false
      matrix:
-        go-version:
+        goversion:


removing the dash to allow using the variable if if conditions

moul · 2023-09-21T16:55:52Z

.github/workflows/examples.yml

          - "1.20.x"
          - "1.21.x"
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
+      - uses: actions/checkout@v4


moving the actions/checkout before setup-go, because setup-go is trying to cache dependencies based on go.mod and go.sum

moul · 2023-09-21T16:56:32Z

.github/workflows/gnoland.yml

      - name: test
        working-directory: gno.land
        run: |
          export GOPATH=$HOME/go
          export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic"
          make ${{ matrix.args }}
-      - if: runner.os == 'Linux'
+      - if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }}


to prevent inconsistent results, let's only upload the coverage of the latest supported version, while still unit testing with the previous version.

moul · 2023-09-21T16:57:06Z

.github/workflows/gnoland.yml

        uses: codecov/codecov-action@v3
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          name: gno.land
          flags: gno.land-${{matrix.args}}
          files: ./gno.land/coverage.out
-          #fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}
-          fail_ci_if_error: false # temporarily
+          fail_ci_if_error: ${{ github.repository == 'gnolang/gno' }}


i think this is what generated failed CI because of codecov 502 errors; would be nice to setup an auto-retry mechanism

Signed-off-by: moul <94029+moul@users.noreply.github.com>

ajnavarro · 2023-09-21T17:03:48Z

.github/workflows/gnovm.yml

      - name: test
        working-directory: gnovm
        run: |
          export GOPATH=$HOME/go
          export GOTEST_FLAGS="-v -p 1 -timeout=30m -coverprofile=coverage.out -covermode=atomic"
          make ${{ matrix.args }}
-      - if: runner.os == 'Linux'
+      - if: ${{ runner.os == 'Linux' && matrix.goversion == '1.21.x' }}


this might be a problem when we update go versions and 1.21 is not supported anymore, we can lose coverage execution. Maybe add go version to the flags?

yeah, I'm the one who made the 3 last go version bumps and I basically made sure to update all the references I can find.

go1.21: chore: bump golang at go1.21 #1093

go1.20: chore: switch to go1.20 #518

go1.19: chore: bump go1.19 #310

at some point, I was considering adding a variable.yml file with "latest_go" and "previous_go" variable; or maybe we can add a quick misc/check-go-versions.sh script to assist us or to check in the CI.

Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul · 2023-09-21T17:12:02Z

The flakiness issue appears to be resolved, as seen in my fork:

Edit: I might have been a bit hasty in my assessment. I've just initiated new builds. I'll keep you updated.

Edit 2: Upon review, I believe this PR effectively makes Codecov visible while minimizing false positives. I recommend we proceed with this to resolve the current impasse. Subsequently, we can dedicate more time to refining QA and DX.

Signed-off-by: moul <94029+moul@users.noreply.github.com>

@ajnavarro

I've conducted tests on my fork, which you can find at https://github.com/moul/gno. The 'if' condition draws inspiration from @ajnavarro's contributions, so a shoutout to him for that. Additionally, @ajnavarro pointed out gaps in our testing for certain packages. We need to strategize on expanding our test coverage to address these overlooked areas. I suggest waiting for his alternative work before considering merging mine. --------- Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul requested a review from a team as a code owner September 21, 2023 12:50

github-actions bot assigned moul Sep 21, 2023

moul requested a review from jaekwon as a code owner September 21, 2023 13:18

moul force-pushed the master branch from a89a30a to 90d7f70 Compare September 21, 2023 16:00

Update codecov.yml

a37ba6a

moul force-pushed the master branch from 90d7f70 to a37ba6a Compare September 21, 2023 16:00

moul added 4 commits September 21, 2023 18:05

chore: fixup

4416a4b

Signed-off-by: moul <94029+moul@users.noreply.github.com>

chore: fixup

f2f708c

Signed-off-by: moul <94029+moul@users.noreply.github.com>

Create tmp.go

cfa1fae

chore: fixup

db978bb

Signed-off-by: moul <94029+moul@users.noreply.github.com>

github-actions bot added the 📦 🌐 tendermint v2 Issues or PRs tm2 related label Sep 21, 2023

moul commented Sep 21, 2023

View reviewed changes

chore: fixup

f749046

Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul commented Sep 21, 2023

View reviewed changes

chore: fixup

1e60225

Signed-off-by: moul <94029+moul@users.noreply.github.com>

github-actions bot removed the 📦 🌐 tendermint v2 Issues or PRs tm2 related label Sep 21, 2023

moul added the 🐞 bug Something isn't working label Sep 21, 2023

ajnavarro reviewed Sep 21, 2023

View reviewed changes

chore: fixup

1ccf42d

Signed-off-by: moul <94029+moul@users.noreply.github.com>

moul added 4 commits September 21, 2023 20:39

chore: fixup

aa344d6

Signed-off-by: moul <94029+moul@users.noreply.github.com>

Update tm2.yml

cd08a31

Update codecov.yml

c3dbc92

Update codecov.yml

968c04c

ajnavarro approved these changes Sep 22, 2023

View reviewed changes

moul merged commit cfefb3b into gnolang:master Sep 22, 2023
187 checks passed

moul added this to the 🚀 main.gno.land (required) milestone Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

moul commented Sep 21, 2023

codecov bot commented Sep 21, 2023 •

edited

moul Sep 21, 2023

moul Sep 21, 2023

moul Sep 21, 2023

moul Sep 21, 2023

moul Sep 21, 2023

moul Sep 21, 2023

moul Sep 21, 2023

ajnavarro Sep 21, 2023 •

edited

moul Sep 21, 2023 •

edited

moul commented Sep 21, 2023 •

edited

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

ci: tell codecov to wait_for_ci to avoid flappy reports #1160

Conversation

moul commented Sep 21, 2023

codecov bot commented Sep 21, 2023 • edited

Codecov Report

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

moul Sep 21, 2023

Choose a reason for hiding this comment

ajnavarro Sep 21, 2023 • edited

Choose a reason for hiding this comment

moul Sep 21, 2023 • edited

Choose a reason for hiding this comment

moul commented Sep 21, 2023 • edited

codecov bot commented Sep 21, 2023 •

edited

ajnavarro Sep 21, 2023 •

edited

moul Sep 21, 2023 •

edited

moul commented Sep 21, 2023 •

edited