Autobuild: Create Debian repository on release #3013

ann0see · 2023-02-14T21:06:56Z

Short description of changes

This PR creates files in the autobuild process to allow apt to use a GitHub release as apt repository. Based on work by @hoffie from https://github.com/jamulussoftware/repos and this idea: Automate update of deb repo on a new release repos#2 (comment)
Tested and working based on this release: https://github.com/ann0see/jamulus/releases/tag/r3_9_1_repotest2
For latest release test see: https://github.com/ann0see/jamulus/releases/tag/r3_9_1devrepo (tested and working)

I consider this approach easier as it is more automated and there's no need for a separate repo.

CHANGELOG: Linux: Debian users can now use the official PPA to get automatic updates. See the Linux install page on jamulus.io for more information.

Context: Fixes an issue?

Related to: #2122

Does this change need documentation? What needs to be documented and how?
Yes. We need to supply the link to the repository, probably using the redirect to the latest release by GitHub and a link to the GPG key on jamulus.io

Status of this Pull Request

Ready for review. Before this can be merged, we need to add a GPG key as secret.

What is missing until this pull request can be merged?

Add GPG key as secret.

Checklist

I've verified that this Pull Request follows the general code principles
I tested my code and it does what I want (pushed a full release after opening this PR, tested headless build)
My code follows the style guide
I waited some time after this Pull Request was opened and all GitHub checks completed without errors.
I've filled all the content above

ann0see · 2023-02-15T17:26:30Z

.github/workflows/autobuild.yml

+        run:                        |
+          set -eu
+
+          [[ "${GPG_PRIVATE_KEY:-}" ]] || {


Forks which don't have a GPG key set, will have a failed build.
I'm not sure if a silent failure would be better. Skipping the repo build should be possible.

There are multiple options:

Remove the whole job (easy)

Introduce a variable enabling/disabling the repo creation

If GPG_PRIVATE_KEY is not present, skip the job. We still need to spin up the container, I'm afraid. GitHub dosn't support checking for present secrets in the .yaml file directly as far as I understand

Will needs.create_release.outputs.publish_to_release == 'true' be true? Is that what you mean by "failed build"?

Yes. If they publish a release it will be true and the job will fail.

Ok. Maybe my new approach works.

Seems to work: https://github.com/jamulussoftware/jamulus/actions/runs/4494259255/jobs/7906934988

That run demonstrates that skipping the job works if the GPG_PRIVATE_KEY is not set. I can't see a run that demonstrates the action is working properly when the private key IS set.

I'm assuming that the main jamulus repo has a private key, but the ann0see repo does not. But to test this before merging, I guess either ann0see's repo needs a (temporary?) private key, or perhaps we need to merge these changes into a branch on the jamulus repo, so that it can use the jamulus repo secrets?

I don't profess to be an expert (or even moderately knowledgeable) in Debian packaging or repos. Most of my packaging experience is with RPMs on RH-based systems.

See https://github.com/ann0see/jamulus/releases/tag/r3_9_1devrepo
That's a working (?) repo

ann0see · 2023-02-15T17:27:37Z

.github/workflows/autobuild.yml

+          mkdir -p gpghome
+          chmod 700 gpghome
+          echo "${GPG_PRIVATE_KEY}" | gpg --homedir gpghome --import -
+      # Unfortunately download-artifact action doesn't support wild card downloads. Thus downloading all artifacts


While this approach is not perfect, I think it's easier to download all artifacts instead of one step per deb file. In future wildcards for downloading only .deb files might be supported.

Alternative: use gh command in CLI

ann0see · 2023-02-15T17:28:49Z

.github/workflows/autobuild.yml

+          path:                     releasedl/
+      - name:                       Create debian repository
+        run:                        |
+          set -eu


Could be in a separate .sh file. Hoffie once said he's in favor of inlining.

It's easier to review in context although it makes the autobuild.yml grow as it becomes more feature-rich. Maybe we need to have a think about it.

ann0see · 2023-02-15T17:30:02Z

.github/workflows/autobuild.yml

+
+      - name:                       Upload Packages file to release
+        id:                         deb-upload-packagesfile
+        uses:                       actions/upload-release-asset@v1


release-asset is outdated. Changing to the other action might need some work.

ann0see · 2023-02-15T17:31:58Z

linux/setup_repo.sh

+GITHUB_REPOSITORY="jamulussoftware/jamulus"
+
+echo "Setting up Jamulus repo at ${REPO_FILE}..."
+echo "deb https://github.com/${GITHUB_REPOSITORY}/releases/latest/download/ ./" > ${REPO_FILE}


This uses a trick: latest points to the last release tagged as "stable" by GitHub. Thus, the link remains unchanged, but a redirect guarantees the correct source.

ann0see · 2023-02-15T17:32:35Z

linux/setup_repo.sh

@@ -0,0 +1,32 @@
+#!/bin/bash


Where should we host this file? Is this repo good or should it be on the website?

ann0see · 2023-02-18T09:34:51Z

I believe this is now ready for testing and review

ann0see · 2023-04-13T21:17:42Z

@pljones Do you know anyone who knows a bit more about Debian packaging or someone who can review this approach?
Hoffie is quite experienced with bash scripting, and the code looks ok. Nevertheless we should have a few eyes looking over this.

pljones · 2023-04-14T16:42:35Z

Sorry, no - for Linux, I've pretty much been a consumer for packaging technologies. I've no development experience there, nor know anyone who has. (At work we use RPMs and don't even use standard deployment for those, so totally unhelpful here!)

ann0see · 2023-04-15T17:00:16Z

Ok. mirabilos was the Debian maintainer, but he stepped back so I'm not sure if he'd want to review this. ignotus has some knowledge of bash scripts, but I don't think he knows deb packaging. @trebmuh might know a bit as he develops a distribution.

softins · 2023-04-30T16:11:58Z

I've seen this review request, and will review it in the next day or two.

ann0see · 2023-04-30T18:50:15Z

Thank you!

softins · 2023-05-01T12:18:32Z

linux/setup_repo.sh

+echo "Setting up Jamulus repo at ${REPO_FILE}..."
+echo "deb https://github.com/${GITHUB_REPOSITORY}/releases/latest/download/ ./" > ${REPO_FILE}
+echo "Installing Jamulus GPG key at ${KEY_FILE}..."
+curl --fail --show-error -sLo "${KEY_FILE}" https://github.com/${GITHUB_REPOSITORY}/releases/latest/download/key.asc


Just trying this script locally, and this step fails with a 404 for key.asc.

Is this script intended for use by a Github workflow, or by an end user? I can't see it referred to anywhere else in the tree.

Is this script intended for use by a Github workflow, or by an end user? I can't see it referred to anywhere else in the tree.

Purely by our git build to create a repo for apt to auto update from.

You'd need to replace the variables. The release will generate a runnable script.

I've currently checked out ann0see/debrepo, but I can't see anywhere that calls setup_repo.sh. I would have expected it to be referred to somewhere within autobuild.yml? Or within another script somewhere in the tree?

You can tell I'm a bit rusty!

setup_repo.sh is a script which the user should run on his machine to install the repository. The logic to create the repository is in autobuild.yml. I believe the content in autobuild.yaml correctly replaces the content of the setup_repo.sh script and uploads the modified version to the release.
Edit: I think I've removed the replacing logic. You'll need to change the GITHUB_REPOSITORY variable to ann0see/jamulus

ann0see · 2023-05-01T13:36:36Z

Also my repo has a GPG key set.

See e.g https://github.com/ann0see/jamulus/actions/runs/4493966363

pljones · 2023-06-11T11:32:35Z

Just checking: is there anything outstanding on this? Rebase?

Does it push beta and pre releases to the repo? (I forgot to check.)

ann0see · 2023-06-11T15:33:36Z

The standard URL will point to the latest stable release. So even though the files for the Debian repo will get uploaded to beta releases users would need to set the respective direct link to beta/nightly releases.

ann0see · 2023-06-11T19:03:31Z

@softins Are you OK with a merge? Documentation will be added.

softins

@softins Are you OK with a merge? Documentation will be added.

Well it looks sensible, but I'm not able to do any testing myself at the moment. Since @pljones was happy to approve, I'll add my approval too.

ann0see · 2023-06-12T12:57:45Z

Ok. Will do a rebase and then merge it. We can always state that this is still experimental

Co-authored-by: Christian Hoffmann <mail@hoffmann-christian.info>

ann0see · 2023-07-24T20:44:01Z

Dropping needs documentation as I've opened jamulussoftware/jamuluswebsite#937 on the website repository.

ann0see requested a review from hoffie February 14, 2023 21:24

ann0see added this to Triage in Tracking (old) via automation Feb 14, 2023

ann0see requested a review from pljones February 14, 2023 21:25

ann0see added this to the Release 3.10.0 milestone Feb 14, 2023

ann0see force-pushed the debrepo branch from 2f647fd to a4186f9 Compare February 14, 2023 21:31

ann0see moved this from Triage to Waiting on Team in Tracking (old) Feb 14, 2023

ann0see force-pushed the debrepo branch from a4186f9 to 362d219 Compare February 14, 2023 21:34

ann0see commented Feb 15, 2023

View reviewed changes

ann0see added the needs documentation PRs requiring documentation changes or additions label Feb 15, 2023

ann0see force-pushed the debrepo branch 2 times, most recently from 244b1e0 to 691001e Compare March 22, 2023 19:57

ann0see requested a review from softins April 29, 2023 21:29

softins reviewed May 1, 2023

View reviewed changes

ann0see force-pushed the debrepo branch from 691001e to 136a29f Compare May 22, 2023 20:49

pljones approved these changes Jun 11, 2023

View reviewed changes

softins approved these changes Jun 12, 2023

View reviewed changes

Tracking (old) automation moved this from Waiting on Team to In Progress Jun 12, 2023

ann0see and others added 2 commits June 12, 2023 22:05

Autobuild: Create Debian repository on release

c11714b

Co-authored-by: Christian Hoffmann <mail@hoffmann-christian.info>

Linux: Add script to install Debian repository automatically

09a2e17

ann0see force-pushed the debrepo branch from 136a29f to 09a2e17 Compare June 12, 2023 20:08

ann0see merged commit be5ece2 into jamulussoftware:main Jun 12, 2023
12 of 13 checks passed

Tracking (old) automation moved this from In Progress to Done Jun 12, 2023

ann0see deleted the debrepo branch June 12, 2023 20:10

ann0see mentioned this pull request Jul 24, 2023

Add documentation for Debian repository jamulussoftware/jamuluswebsite#937

Closed

ann0see removed the needs documentation PRs requiring documentation changes or additions label Jul 24, 2023

pljones removed this from Done in Tracking (old) Jul 28, 2023

pljones assigned ann0see Jul 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Autobuild: Create Debian repository on release #3013

Autobuild: Create Debian repository on release #3013

ann0see commented Feb 14, 2023 •

edited

ann0see Feb 15, 2023

pljones Feb 18, 2023

ann0see Feb 18, 2023 •

edited

ann0see Mar 22, 2023

ann0see Mar 22, 2023

softins May 1, 2023

ann0see May 1, 2023

ann0see Feb 15, 2023 •

edited

ann0see Feb 15, 2023

pljones Feb 18, 2023

ann0see Feb 15, 2023

ann0see Feb 15, 2023

ann0see Feb 15, 2023

ann0see commented Feb 18, 2023

ann0see commented Apr 13, 2023

pljones commented Apr 14, 2023

ann0see commented Apr 15, 2023

softins commented Apr 30, 2023

ann0see commented Apr 30, 2023

softins May 1, 2023

pljones May 1, 2023

ann0see May 1, 2023

softins May 1, 2023

ann0see May 1, 2023 •

edited

ann0see commented May 1, 2023 •

edited

pljones commented Jun 11, 2023

ann0see commented Jun 11, 2023

ann0see commented Jun 11, 2023

softins left a comment

ann0see commented Jun 12, 2023

ann0see commented Jul 24, 2023

Autobuild: Create Debian repository on release #3013

Autobuild: Create Debian repository on release #3013

Conversation

ann0see commented Feb 14, 2023 • edited

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann0see Feb 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann0see Feb 15, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann0see commented Feb 18, 2023

ann0see commented Apr 13, 2023

pljones commented Apr 14, 2023

ann0see commented Apr 15, 2023

softins commented Apr 30, 2023

ann0see commented Apr 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ann0see May 1, 2023 • edited

Choose a reason for hiding this comment

ann0see commented May 1, 2023 • edited

pljones commented Jun 11, 2023

ann0see commented Jun 11, 2023

ann0see commented Jun 11, 2023

softins left a comment

Choose a reason for hiding this comment

ann0see commented Jun 12, 2023

ann0see commented Jul 24, 2023

ann0see commented Feb 14, 2023 •

edited

ann0see Feb 18, 2023 •

edited

ann0see Feb 15, 2023 •

edited

ann0see May 1, 2023 •

edited

ann0see commented May 1, 2023 •

edited