Migrate release CI back to github #9437

hasufell · 2023-11-13T09:43:50Z

No description provided.

hasufell · 2023-11-13T15:42:45Z

Example pipeline: https://github.com/haskell/cabal/actions/runs/6851573157/job/18629453051?pr=9437
Example draft release: https://github.com/haskell/cabal/releases/tag/untagged-5a72917f7c0a5a2480ec
Example cirrus CI pipeline: https://cirrus-ci.com/task/6646855723909120?logs=main#L11

Also includes:

script to download artifacts
script to create yaml snippet for ghcup

hasufell · 2023-11-13T15:55:01Z

Workflow:

push tag cabal-install-v4.0.0.0
wait for github and cirrus builds to finish
run scripts/release/download-gh-artifacts.sh cabal-install-v4.0.0.0 <your-gpg-email>
- upload the contents of the folder gh-release-artifacts/cabal-4.0.0.0 to https://downloads.haskell.org/~cabal/cabal-install-4.0.0.0/
run scripts/release/create-yaml-snippet.sh cabal-install-v4.0.0.0
- based on the output create a ghcup-metadata PR

Kleidukos

From my perspective this is quite appealing and I would like to give it a try for the next release.

need more explaining

Kleidukos · 2023-11-13T16:57:54Z

EDIT: I have been a bit quick to approve without giving much background, so here it is: This is appealing because:

There is a FreeBSD job configured. When I released cabal-3.10.2.0, I had to manually create the FreeBSD bindists on my own private box.
We offload macOS and Windows runners administration to GitHub. These runners being unstable was not the end of the world but certainly did not help.

As such I am inclined to give this pipeline a chance for these two cases. The linux jobs of the Gitlab pipeline have never been a problem when I took care of 3.10.2.0, so I don't think we need to get rid of them.

hasufell · 2023-11-13T17:03:24Z

The linux jobs of the Gitlab pipeline have never been a problem when I took care of 3.10.2.0, so I don't think we need to get rid of them.

We get rid of gitlab CI so we don't have to maintain two entirely separate CIs and then figure out how to mangle everything together in a big mess. That's dead code and unnecessary redundancy. Adding new linux distro/distro-versions to the current github CI release pipeline is very easy.

hasufell · 2023-11-14T05:08:23Z

Time comparison:

Gitlab CI: 98m 15s
- https://gitlab.haskell.org/haskell/cabal/-/pipelines/86388
GitHub CI: 32m 35s
- https://github.com/haskell/cabal/actions/runs/6851573157

andreabedini

Happy to give this a go!

chreekat

Since I am not a Cabal release manager, I don't think my comments about the new workflow should be binding, but I hope they are helpful.

I do think the GitLab stuff should be kept as a backup as part of good change management, however, so I will request that change.

I think it's a bit funny that the pipeline is still using multiple sources (GitHub and Cirrus). You could just as easily use GitLab as the backend. Or GitLab could use Cirrus to pick up FreeBSD. Maybe I'll try that.

The final thing I'd recommend is documentation. I assume the release workflow is documented in this repo?

Overall lgtm and I'll defer to the actual release managers for the actual approvals!

chreekat · 2023-11-15T08:16:22Z

.github/scripts/build.sh

@@ -0,0 +1,66 @@
+#!/bin/bash
+
+set -eux


I prefer explicit tracing instead of increasing default verbosity everywhere because it adds a permanent tax on reading and storing logs. I feel like a lot of my career has been spent figuring out how to filter useless log messages. But maybe that's just me.

chreekat · 2023-11-15T08:20:06Z

.github/workflows/release.yaml

+    tags:
+      - 'cabal-install-*'
+  schedule:
+    - cron: '0 2 * * *'


Who is alerted when scheduled jobs fail on GitHub Actions? Assuming it's more than one person, this will be an improvement over GitLab. But fwiw, the release pipeline runs on every push to master on GitLab, rather than on a schedule. I think either is fine, but I want to highlight the change to other reviewers.

https://docs.github.com/en/github-ae@latest/actions/monitoring-and-troubleshooting-workflows/notifications-for-workflow-runs

Notifications for scheduled workflows are sent to the user who initially created the workflow. If a different user updates the cron syntax in the workflow file, subsequent notifications will be sent to that user instead.

chreekat · 2023-11-15T08:22:42Z

.github/workflows/release.yaml

+          DISTRO: ${{ matrix.platform.DISTRO }}
+          ADD_CABAL_ARGS: ${{ matrix.platform.ADD_CABAL_ARGS }}
+
+      - if: always()


Why always()?

It's possible the build finished, but the script errored anyway. It's good to have the binary for investigation

.github/workflows/release.yaml

chreekat · 2023-11-15T09:04:08Z

.github/workflows/release.yaml

+      - if: matrix.ARCH == 'ARM'
+        uses: docker://hasufell/arm32v7-ubuntu-haskell:focal
+        name: Run build (armv7 linux)
+        with:
+          args: bash .github/scripts/build.sh
+        env:
+          ARTIFACT: ${{ matrix.ARTIFACT }}
+
+      - if: matrix.ARCH == 'ARM64'
+        uses: docker://hasufell/arm64v8-ubuntu-haskell:focal
+        name: Run build (aarch64 linux)
+        with:
+          args: bash .github/scripts/build.sh
+        env:
+          ARTIFACT: ${{ matrix.ARTIFACT }}


Why are these two steps instead of one parameterized step? The name doesn't seem important; it could just be "Run build", and the docker image could be a matrix parameter. Right?

Afair the uses doesn't allow variables.

I <3 json-as-code

.github/workflows/release.yaml

chreekat · 2023-11-15T09:09:33Z

.github/workflows/release.yaml

+        shell: pwsh
+        run: |
+          C:\msys64\usr\bin\bash -lc "pacman --disable-download-timeout --noconfirm -Syuu"
+          C:\msys64\usr\bin\bash -lc "pacman --disable-download-timeout --noconfirm -Syuu"


Is this duplication intentional?

Could use a comment then.

chreekat · 2023-11-15T09:22:24Z

I think I messed up by marking my review as "needs changes". The actual change I want is the restoration of GitLab files. I don't think my other suggestions should be binding.

chreekat · 2023-11-15T09:27:07Z

Time comparison:

Gitlab CI: 98m 15s
    https://gitlab.haskell.org/haskell/cabal/-/pipelines/86388
GitHub CI: 32m 35s
    https://github.com/haskell/cabal/actions/runs/6851573157

Ironically, the GitLab pipeline took forever because the Darwin job had to queue
for an hour for an open spot. The GitHub job runs on the same server and
therefore competes for the same resource. It just gets to jump the queue since
the two job runners don't know about each other.

hasufell · 2023-11-15T09:42:21Z

I do think the GitLab stuff should be kept as a backup as part of good change management, however, so I will request that change.

That would mean we just increased complexity, which defeats the purpose of this PR.

If anything goes extremely wrong during the next release, you can restore the gitlab code from the git history.

Co-authored-by: Bryan Richter <bryan@haskell.foundation>

chreekat · 2023-11-15T10:09:17Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice. And basically free in this case.

Mikolaj · 2023-11-15T10:19:10Z

keep it around for the next release and then dropping it

Regarding this one point, I'd make it two releases and I'd keep the gitlab CI running all the time to ensure we spot problems in it ASAP, just as we do today.

While I'm talking, @hasufell, thank you for the PR. I remember the big appeal of your proposal long ago, now materialised, was that we'd test what we release. Does this PR bring us closer to the point where we test daily the same binaries that will be released? In particular, are our normal GHA tests run on these artifacts, and when not applicable, on the exact commits from which the artifacts are created? What's the roadmap to unify CI and release CI, to a reasonable extent?

hasufell · 2023-11-16T01:35:28Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice.

I don't agree with this at all. There's no indication this is necessary. In HLS we also didn't need it and it has a much more complex CI.

chreekat · 2023-11-16T08:20:27Z

That would mean we just increased complexity, which defeats the purpose of this PR.

If it wasn't clear, I meant to keep it around for the next release and then dropping it. I hope that doesn't defeat the purpose. This change is bigger than just reimplementing some code ,and keeping two systems running in parallel is good practice.

I don't agree with this at all. There's no indication this is necessary. In HLS we also didn't need it and it has a much more complex CI.

To avoid a deadlock I've created https://github.com/chreekat/cabal/tree/b/add-github-release-ci, which is an edited version of this branch that keeps GitLab CI. I hope it won't be necessary.

hasufell · 2023-11-16T09:16:55Z

I can split the PR in two commits, so that Gitlab removal is easier to revert in case it's necessary.

But I won't subscribe to "let's keep code in master, just in case". That's bad engineering practice.

Mikolaj · 2023-11-16T11:28:41Z

But I won't subscribe to "let's keep code in master, just in case". That's bad engineering practice.

I don't think anybody is proposing that. Gitlab CI would be running continuously so that we can spot any problems and fix them ASAP and during a release, if needed, the release manager would be able to compare the results of gitlab and github, test behaviour of the artifacts from both systems, etc.

hasufell · 2023-11-16T11:30:59Z

I have decided to build bindists myself for the next cabal release, so cabal's CI isn't relevant for GHCup directly anymore.

You can use this PR and the contained patch as you please.

andreabedini · 2023-11-17T01:28:34Z

I have reopened this to prevent us losing valuable work.

andreabedini · 2023-11-18T04:02:59Z

And thank you @hasufell for working on this!

andreabedini

Just leaving some comments as I had the time to read this properly. It looks quite comprehensive! good stuff.

PS: @hasufell you have been clear that you don't intend to continue working on this so my comments are not to be taken as request for more work from you :-) They more like notes to my future self.

andreabedini · 2023-12-01T16:51:25Z

.github/scripts/build.sh

+
+# https://github.com/haskell/cabal/issues/7313#issuecomment-811851884
+if [ "$(getconf LONG_BIT)" == "32" ] || [ "${DISTRO}" == "CentOS" ] ; then
+    echo 'constraints: lukko -ofd-locking' >> cabal.project.release.local


I believe this should just be included in cabal.project.release under a if arch(I386) conditional. I am not sure if there is something specific to CentOS, it does not seem to be mentioned in the thread above

andreabedini · 2023-12-01T16:57:06Z

.github/scripts/build.sh

+args=(
+    -w "ghc-$GHC_VERSION"
+    --disable-profiling
+    --enable-executable-stripping


other things that should be in cabal.project.release

andreabedini · 2023-12-01T16:58:28Z

.cirrus.yml

+  timeout_in: 120m
+  only_if: $CIRRUS_TAG != ''
+  env:
+    ADD_CABAL_ARGS: "--enable-split-sections"


this can be implemented in cabal.project.release under a conditional if os(FreeBSD)

andreabedini · 2023-12-01T17:03:55Z

.github/workflows/release.yaml

+        uses: actions/checkout@v3
+
+      - if: matrix.ARCH == 'ARM'
+        uses: docker://hasufell/arm32v7-ubuntu-haskell:focal


Are there officially maintaned images we can use?

andreabedini · 2023-12-01T17:04:03Z

.github/workflows/release.yaml

+          ARTIFACT: ${{ matrix.ARTIFACT }}
+
+      - if: matrix.ARCH == 'ARM64'
+        uses: docker://hasufell/arm64v8-ubuntu-haskell:focal


same as above

andreabedini · 2023-12-01T17:08:37Z

.github/workflows/release.yaml

+
+      - name: git config
+        run: |
+          git config --global --get-all safe.directory | grep '^\*$' || git config --global --add safe.directory "*"


in the current ci we just do

git config --global protocol.file.allow always

and we have a comment point to https://bugs.launchpad.net/ubuntu/+source/git/+bug/1993586 (cabal PR #8546)

hasufell marked this pull request as draft November 13, 2023 09:43

hasufell force-pushed the purge-gitlab branch 16 times, most recently from 001d0e1 to bb01a96 Compare November 13, 2023 14:57

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 5c37718 to 34a7c9c Compare November 13, 2023 15:45

hasufell marked this pull request as ready for review November 13, 2023 15:45

hasufell mentioned this pull request Nov 13, 2023

[Rel-Eng]: Adds script to automatically generate GHCup metadata for Cabal releases #9411

Merged

3 tasks

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 3187a1a to 8781d59 Compare November 13, 2023 15:53

hasufell requested review from andreabedini and Kleidukos November 13, 2023 15:56

hasufell force-pushed the purge-gitlab branch 2 times, most recently from 81397df to 0800602 Compare November 13, 2023 16:09

Kleidukos previously approved these changes Nov 13, 2023

View reviewed changes

Mikolaj requested a review from gbaz November 13, 2023 18:24

Migrate release CI back to github

a88308c

hasufell force-pushed the purge-gitlab branch from 0800602 to a88308c Compare November 14, 2023 12:20

andreabedini approved these changes Nov 14, 2023

View reviewed changes

chreekat requested changes Nov 15, 2023

View reviewed changes

Apply suggestions from code review

8edddae

Co-authored-by: Bryan Richter <bryan@haskell.foundation>

hasufell closed this Nov 16, 2023

andreabedini reopened this Nov 17, 2023

Merge branch 'master' into purge-gitlab

fbf1f85

wz1000 mentioned this pull request Nov 28, 2023

HLS 2.5.0.0 haskell/ghcup-metadata#159

Merged

andreabedini reviewed Dec 1, 2023

View reviewed changes

hasufell mentioned this pull request Mar 21, 2024

Formalising our stance regarding binary distribution of cabal-install #9827

Open

Migrate release CI back to github #9437

Are you sure you want to change the base?

Migrate release CI back to github #9437

Conversation

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

hasufell commented Nov 13, 2023

Kleidukos left a comment

Choose a reason for hiding this comment

Kleidukos commented Nov 13, 2023

hasufell commented Nov 13, 2023 • edited

hasufell commented Nov 14, 2023

andreabedini left a comment

Choose a reason for hiding this comment

chreekat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chreekat commented Nov 15, 2023

chreekat commented Nov 15, 2023 • edited

hasufell commented Nov 15, 2023

chreekat commented Nov 15, 2023

Mikolaj commented Nov 15, 2023 • edited

hasufell commented Nov 16, 2023

chreekat commented Nov 16, 2023

hasufell commented Nov 16, 2023

Mikolaj commented Nov 16, 2023

hasufell commented Nov 16, 2023

andreabedini commented Nov 17, 2023

andreabedini commented Nov 18, 2023

andreabedini left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hasufell commented Nov 13, 2023 •

edited

chreekat commented Nov 15, 2023 •

edited

Mikolaj commented Nov 15, 2023 •

edited

andreabedini left a comment •

edited