automate builds with gh actions #78

Migelo · 2024-06-09T01:26:20Z

This draft pull request aims to automate the builds using gh actions vmactions/freebsd-vm@v1 and renovate bot to trigger the builds when https://github.com/jellyfin/jellyfin publish a new release.

I need help with the building part using dotnet.

As per your comment Building the .NET Core SDK on FreeBSD dotnet/source-build#1139 (comment) I tried the simple pkg install dotnet to get dotnet in the FreeBSD VM,
download the latest jellyfin_10.9.6-amd64.tar.xz
TODO build

I don't have any experiences building with dotnet, so if you could give me some pointers, I am happy to spend time to get this automated build process going.

Configure Renovate

Thefrank · 2024-06-09T05:05:57Z

Oh nice!

I added my WIP/experimental ports build to repo. You can check it out here.

I forgot to upload it last week :(

I can add this (and something for the UI part) if I am unable to get a ports version working

Migelo · 2024-06-09T13:06:35Z

Thank you! I will have a look and then try to make it work, but ofc feel free to push commits yourself. :)

Migelo · 2024-06-09T17:45:25Z

I managed to get the environment more or less set up, at least how I understand it should be set up.
The build fails tho, can you please have a look? The error is here step 3, line 4472 and beyond.

Here are the first few lines for convenience.

  cd /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/ &&  dotnet restore Jellyfin.Server --runtime freebsd-x64 --packages /usr/ports/multimedia/jellyfin/work/packages &&  dotnet publish Jellyfin.Server --configuration Release --self-contained --runtime freebsd-x64 --packages /usr/ports/multimedia/jellyfin/work/packages -o bin/publish "-p:DebugSymbols=false;DebugType=none;UseAppHost=true;PublishReadyToRun=false"
  
  Welcome to .NET 8.0!
  ---------------------
  SDK Version: 8.0.100
  
  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate, view the instructions: https://aka.ms/dotnet-https-linux
  
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
    Determining projects to restore...
    Restored /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/src/Jellyfin.MediaEncoding.Keyframes/Jellyfin.MediaEncoding.Keyframes.csproj (in 506 ms).
  /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/src/Jellyfin.Networking/Jellyfin.Networking.csproj : error NU1603: Warning As Error: Microsoft.AspNetCore.HttpOverrides 2.2.0 depends on Microsoft.Extensions.Options (>= 2.2.0) but Microsoft.Extensions.Options 2.2.0 was not found. An approximate best match of Microsoft.Extensions.Options 8.0.0 was resolved. [/usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Jellyfin.Server/Jellyfin.Server.csproj]
  /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/src/Jellyfin.MediaEncoding.Hls/Jellyfin.MediaEncoding.Hls.csproj : error NU1603: Warning As Error: Microsoft.AspNetCore.HttpOverrides 2.2.0 depends on Microsoft.Extensions.Options (>= 2.2.0) but Microsoft.Extensions.Options 2.2.0 was not found. An approximate best match of Microsoft.Extensions.Options 8.0.0 was resolved. [/usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Jellyfin.Server/Jellyfin.Server.csproj]

It goes on and on with the final lines being

  /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Emby.Naming/Emby.Naming.csproj : error NU1603: Warning As Error: Microsoft.Extensions.FileProviders.Abstractions 2.2.0 depends on Microsoft.Extensions.Primitives (>= 2.2.0) but Microsoft.Extensions.Primitives 2.2.0 was not found. An approximate best match of Microsoft.Extensions.Primitives 8.0.0 was resolved. [/usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Jellyfin.Server/Jellyfin.Server.csproj]
  /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Emby.Naming/Emby.Naming.csproj : error NU1603: Warning As Error: Microsoft.Net.Http.Headers 2.2.0 depends on Microsoft.Extensions.Primitives (>= 2.2.0) but Microsoft.Extensions.Primitives 2.2.0 was not found. An approximate best match of Microsoft.Extensions.Primitives 8.0.0 was resolved. [/usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Jellyfin.Server/Jellyfin.Server.csproj]
  /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Emby.Naming/Emby.Naming.csproj : error NU1603: Warning As Error: Microsoft.AspNetCore.Http.Abstractions 2.2.0 depends on System.Text.Encodings.Web (>= 4.5.0) but System.Text.Encodings.Web 4.5.0 was not found. An approximate best match of System.Text.Encodings.Web 8.0.0 was resolved. [/usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Jellyfin.Server/Jellyfin.Server.csproj]
    Failed to restore /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Emby.Naming/Emby.Naming.csproj (in 31 ms).
    Failed to restore /usr/ports/multimedia/jellyfin/work/jellyfin-10.9.2/Emby.Server.Implementations/Emby.Server.Implementations.csproj (in 129 ms).
  *** Error code 1
  
  Stop.
  make: stopped in /usr/ports/multimedia/jellyfin

Thank you for your help!

Thefrank · 2024-06-09T20:25:36Z

So NuGet has this long standing history of looking for or restoring NuGets that are not needed during a specific code path. This can (rarely) happen if there is mismatch between the the requested SDK and the resolved SDK. This can make the determining the FULL list of depends rather challenging.

Some of the ones listed were also stored and restored from local cache when I built jellyfin outside of ports this which is why they got missed here.

I will work on getting that updated with the actual list

On the plus side:

the recently released FreeBSD 14.1 includes ZFS based VM images which will make troubleshooting inside a proper build environment easier

the UI looks like it built without issue

Migelo · 2024-06-09T20:37:31Z

Thank you for the help. From what you wrote I understand that you'll work on an updated list of NuGets that should fix this build.

In the meantime, how would I go about building jellyfin outside of ports? Might be the simpler way in the end.

Excellent news, I love myself some zfs goodness :) I am currently running a FreeBSD VM through quickemu on an Ubuntu 24.04 host.

Thefrank · 2024-06-09T21:46:42Z

The method that I currently use to build Jellyfin is to use Azure Pipelines Agent (I publish a build here) and use a locally hosted instance.

Checkout from tag: jellyfin
dotnet publish Jellyfin.Server --configuration Release --self-contained --runtime freebsd-x64 --output dist/jellyfin-server/ "-p:DebugSymbols=false;DebugType=none;UseAppHost=true"
Publish output as tar
(Run tests: build fails if tests fail)
Checkout from tag: jellyfin-web
npm ci --no-audit && npm run build:production
Publish output as tar
(Run tests: build fails if tests fail)
Download both outputs
Download prebuilt Skia lib
Combine

Thefrank · 2024-06-09T23:06:32Z

I have updated the NuGet list. That should be all of them. This is also more than what is actually consumed by the build but instead everything that is explicitly referenced, transitively referenced, and things that NuGet wants restored anyways.

Thefrank · 2024-06-10T08:30:01Z

fixed an extra file in pkg-plist
poudriere testport -b latest -o multimedia/jellyfin completes without errors. I am not brave enough to try and have it build: node, npm, llvm, and dotnet but it does work with current packages

Migelo · 2024-06-10T10:28:20Z

The first run ran just short of 2h and then the runner crashed...so I first assumed we ran out of ram or disk space, but reading the gh runner docs they mention 16GB of ram and 14GB of disk, so I do not believe we could be crashing due to RAM running out.

Did you run the poudriere command on the port from this repository? I assume so, but I am new to all of this FreeBSD packaging.

Migelo · 2024-06-10T11:08:33Z

The run finished successfully with this!

I think the last thing we need to do now is to extract the artifacts and we're good to go!

Thefrank · 2024-06-10T15:33:19Z

The first run ran just short of 2h and then the runner crashed...so I first assumed we ran out of ram or disk space, but reading the gh runner docs they mention 16GB of ram and 14GB of disk, so I do not believe we could be crashing due to RAM running out.

I think most runners are capped at 2h. If poudriere has to build a large dependence tree it can take hours/days on first run.

Did you run the poudriere command on the port from this repository? I assume so, but I am new to all of this FreeBSD packaging.

The command issued was against ports tree with the jellyfin port files replaced with ones from this repo. This is why poudriere reports an unclean checkout in the log.

log: jellyfin-10.9.2.log

Migelo · 2024-06-11T00:43:21Z

The command issued was against ports tree with the jellyfin port files replaced with ones from this repo. This is why poudriere reports an unclean checkout in the log.

Great, thank you for the explanation.

I think most runners are capped at 2h. If poudriere has to build a large dependence tree it can take hours/days on first run.

It seems the runner runs out of space during the build, see this run for example.

I switched over to ubicloud ubicloud-standard-4 runner. It ended up working and I built 10.9.2 here. I tried running it and it did not work unfortunately, but maybe that's because I was running 10.9.4 on my server and downgrading can cause issues.

I also implemented renovate so that it automatically compares the current release of jellyfin to the version string in the Makefile and then updates the value with the new one and submits a pull request, like this one for 10.9.6.

I noticed the version string is also present in

distinfo
package-lock.json
package.json

I guess we need to fix those as well.

Thefrank · 2024-06-11T02:09:13Z

I used 10.9.2 as a framework and to wait for @mvanbaak feedback on it. I was able to get it to build in poudriere so it should work as a drop in replacement for the current ports version which downloads a prebuilt binary from here.

The upside of it being a source built from ports is people won't complain about it being "binary only" :)

I noticed the version string is also present in

This is working as designed. Any time Jellyfin gets a bump the NuGet files (tedious), Node files (trivial), and pkg-plist (easy) files need to be redone. This also means regenerating the distinfo (easy). Patches are another item that will likely need to be redone often as they are applied without fuzzing.

Migelo · 2024-06-11T13:56:59Z

This is working as designed.

Oki, great! Then what do you recommend? That I switch to poudriere or keep my current setup as defined in the workflow?

Thefrank · 2024-06-12T23:40:21Z

poudriere is the best way to reproduce the FreeBSD packaging process as that is what FreeBSD uses. Anything submitted to port must pass poudriere.

A single dotnet line is the best way to just build the server and a single line npm is the best way to build the web-ui. Until now, this has been my method as I can just run a single YML and have it all done.

AFAIK poudriere lacks a method of automatically bumping/editing files. This is typically the job of the port maintainer (or the person building something locally).

Migelo · 2024-06-13T09:42:56Z

Great, I have been switching to poudriere and compiling on my machine to test. Hopefully I can get a quick build automated here soon.

renovate bot and others added 7 commits June 9, 2024 00:01

Add renovate.json

83d4a8b

Merge pull request #1 from Migelo/renovate/configure

eaaa9e8

Configure Renovate

Update renovate.json

8a7a3c0

Create main.yml

9b2f254

Update main.yml

3aec0e8

add dotnet publish to workflow

bbdd38c

fix fname in actions

1c5f468

Migelo added 11 commits June 9, 2024 15:32

Merge branch 'Thefrank:main' into main

ac7ad7e

add missing packages to the FreeBSD runner

4260320

Update main.yml

6bbb314

fix actions

91f26b9

copy the missing files to their place

9d3a86f

debug gh action

29bf438

fix variable definition in sh

c0ba913

fix filename to be extracted

e355d49

fix openssl version through setenv DEFAULT_VERSIONS

13676b0

fix variables because we are using sh actually

4c05587

remove obsolete lines

37f9fc6

Merge branch 'Thefrank:main' into main

e9c2b37

Migelo added 3 commits June 10, 2024 09:10

add a more beefy runner

e64d047

try ubicloud runner

084243e

Merge branch 'Thefrank:main' into main

cc45564

Migelo added 5 commits June 10, 2024 11:17

test artefacts upload

c3e905c

upload-artifact debug

efc52b0

disable renovate for npm

84b6905

re-enable building

ba5bf8f

add ports back

c6920a1

check if packing artifacts works

047df7a

Migelo force-pushed the main branch from 428dd8b to 047df7a Compare June 10, 2024 16:44

Migelo added 7 commits June 10, 2024 16:50

fix path for artefact

addfc48

do full build with upload-artifact on gh runner

a7522f8

do full build on ubicloud

ec23023

update renovate.json

d78b414

update renovate

cf0308e

renovate fix path

7515957

gh action narrow down artifacts

0d87ef0

Migelo added 7 commits June 11, 2024 16:13

switch build to poudriere

404aed3

create missing distfiles directory

89c0a5d

fix path

2cc0213

speed up parallel build

5c67185

Merge branch 'Thefrank:main' into main

f17f81c

check vm hw

5f18632

use 16 cores in freebsd vm

ebd66a3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

automate builds with gh actions #78

automate builds with gh actions #78

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Migelo commented Jun 9, 2024

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Thefrank commented Jun 10, 2024

Migelo commented Jun 10, 2024

Migelo commented Jun 10, 2024

Thefrank commented Jun 10, 2024

Migelo commented Jun 11, 2024

Thefrank commented Jun 11, 2024

Migelo commented Jun 11, 2024

Thefrank commented Jun 12, 2024

Migelo commented Jun 13, 2024

automate builds with gh actions #78

Are you sure you want to change the base?

automate builds with gh actions #78

Conversation

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Migelo commented Jun 9, 2024

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Migelo commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Thefrank commented Jun 9, 2024

Thefrank commented Jun 10, 2024

Migelo commented Jun 10, 2024

Migelo commented Jun 10, 2024

Thefrank commented Jun 10, 2024

Migelo commented Jun 11, 2024

Thefrank commented Jun 11, 2024

Migelo commented Jun 11, 2024

Thefrank commented Jun 12, 2024

Migelo commented Jun 13, 2024