Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add macOS builder script for CI #345

Merged
merged 38 commits into from Mar 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e389681
builder: add macos variant for dependencies
gnattu Feb 23, 2024
e79ab07
fix: build script for mac
gnattu Feb 25, 2024
a5e4d49
fix: missing configs
gnattu Feb 28, 2024
32e2444
feat: macOS build scripts
gnattu Mar 1, 2024
a7c356a
fix: env vars
gnattu Mar 1, 2024
47057cf
Merge branch 'jellyfin:jellyfin' into jellyfin-macos-build
gnattu Mar 1, 2024
1f6d1df
fix: use relative path
gnattu Mar 1, 2024
24f0cd6
fix: use relative path
gnattu Mar 1, 2024
1eb5940
fix: detect host OS and toolchain version
gnattu Mar 1, 2024
ee46428
fix: leftover
gnattu Mar 1, 2024
40621b6
fix: export prefix early
gnattu Mar 2, 2024
1328c75
feat: prepare dependencies
gnattu Mar 3, 2024
b417855
fix: var
gnattu Mar 3, 2024
1366cde
fix: builder root should use pwd
gnattu Mar 3, 2024
89a5434
fix: use var
gnattu Mar 3, 2024
1f18eae
fix: don’t cd
gnattu Mar 3, 2024
b5f795f
fix: use absolute path
gnattu Mar 3, 2024
f3640c3
fix: use absolute path
gnattu Mar 3, 2024
7e0758f
fix: use continue
gnattu Mar 3, 2024
ecc6383
fix: patches
gnattu Mar 3, 2024
d9f2ed9
fix: build pcre
gnattu Mar 3, 2024
1fbd85e
fix: dependencies
gnattu Mar 3, 2024
2158c38
feat: add mac actions
gnattu Mar 4, 2024
45e4f72
fix: prepare prefix dir
gnattu Mar 4, 2024
d2fd942
fix: sethup homebrew before use
gnattu Mar 4, 2024
fc54818
fix: run on macOS matrix
gnattu Mar 4, 2024
8bff935
fix: use os.name
gnattu Mar 4, 2024
30083c1
fix: compile pcre2 first
gnattu Mar 4, 2024
02e1d9f
fix: pcre2 prefix path
gnattu Mar 4, 2024
c6c812e
feat: do actual build
gnattu Mar 4, 2024
77be219
fix: move files before zip it
gnattu Mar 4, 2024
4dc8699
fix: target detection
gnattu Mar 4, 2024
406f8bd
fix: disalbe libxcb on macOS
gnattu Mar 4, 2024
2585d1b
fix: remove everthing using libx11 on x86_64 runner
gnattu Mar 4, 2024
70d0bfa
doc: add macos build doc
gnattu Mar 4, 2024
d24936c
chore: use more intuative macOS setup for built-in libs
gnattu Mar 6, 2024
31a8c8b
chore: more intuative OpenCL setup on macOS
gnattu Mar 6, 2024
d381804
fix: remove redundant prepare step
gnattu Mar 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
108 changes: 108 additions & 0 deletions .github/workflows/_meta_mac_portable.yaml
@@ -0,0 +1,108 @@
on:
workflow_call:
inputs:
release:
required: false
default: false
type: boolean
secrets:
deploy-host:
required: false
deploy-user:
required: false
deploy-key:
required: false

jobs:
build:
name: 'Build Portable FFmpeg'
runs-on: ${{ matrix.os.name }}
strategy:
fail-fast: true
matrix:
# Currently, macOS 12 is x86 exclusive and macOS 14 is arm exclusive and we have no other way to specify arch
os:
- name: macos-12
arch: x86_64
- name: macos-14
arch: arm64
steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1

- name: Set up Homebrew
id: set-up-homebrew
uses: Homebrew/actions/setup-homebrew@master

- name: Prepare prefix dir
run: |
sudo mkdir /opt/ffbuild
sudo chmod 777 /opt/ffbuild

- name: Build Portable
run: ./builder/buildmac.sh

- name: Upload Artifacts
uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3 # v4.3.1
with:
name: mac-${{ matrix.os.arch }}-portable
path: ./builder/artifacts

publish:
name: Publish Portable Release
if: ${{ inputs.release }}
runs-on: ubuntu-latest
needs:
- build
strategy:
fail-fast: true
matrix:
arch: [arm64, x86_64]

steps:
- name: Set Versions
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
id: set_version
with:
script: |
const tag = context.ref.substring(10)
const no_v = tag.replace('v', '')
const dash_index = no_v.lastIndexOf('-')
const no_dash = (dash_index > -1) ? no_v.substring(0, dash_index) : no_v
core.setOutput('tag', tag)
core.setOutput('no-v', no_v)
core.setOutput('no-dash', no_dash)

- name: Download Artifacts
uses: actions/download-artifact@c850b930e6ba138125429b7e5c93fc707a7f8427 # v4.1.4
with:
name: mac-${{ matrix.arch }}-portable
path: artifact

- name: Upload GH Release Assets
uses: shogo82148/actions-upload-release-asset@c91b7fd867cf280772374042252f0dc8e1ade751 # v1.7.3
with:
upload_url: ${{ github.event.release.upload_url }}
overwrite: true
asset_path: |
./artifact/**/*.tar.xz
./artifact/**/*.sha256sum

- name: Make Sure FFmpeg Directory Exists
uses: appleboy/ssh-action@029f5b4aeeeb58fdfe1410a5d17f967dacf36262 # v1.0.3
with:
host: ${{ secrets.deploy-host }}
username: ${{ secrets.deploy-user }}
key: ${{ secrets.deploy-key }}
script_stop: true
script: |-
mkdir -p /srv/repository/releases/ffmpeg/${{ steps.set_version.outputs.no-v }}/

- name: Upload Release Assets
uses: burnett01/rsync-deployments@45d84ad5f6c174f3e0ffc50e9060a9666d09c16e # 6.0.0
with:
switches: -vrptz
path: ./artifact/*
remote_path: /srv/repository/releases/ffmpeg/${{ steps.set_version.outputs.no-v }}/
remote_host: ${{ secrets.deploy-host }}
remote_user: ${{ secrets.deploy-user }}
remote_key: ${{ secrets.deploy-key }}
5 changes: 5 additions & 0 deletions .github/workflows/build.yaml
Expand Up @@ -47,3 +47,8 @@ jobs:
os: 'linux'
architectures: '["amd64", "arm64"]'
release: false

build_portable_mac:
uses: ./.github/workflows/_meta_mac_portable.yaml
with:
release: false
9 changes: 9 additions & 0 deletions .github/workflows/publish.yaml
Expand Up @@ -52,6 +52,15 @@ jobs:
deploy-user: ${{ secrets.DEPLOY_USER }}
deploy-key: ${{ secrets.DEPLOY_KEY }}

build_publish_mac_portable:
uses: ./.github/workflows/_meta_mac_portable.yaml
with:
release: true
secrets:
deploy-host: ${{ secrets.DEPLOY_HOST }}
deploy-user: ${{ secrets.DEPLOY_USER }}
deploy-key: ${{ secrets.DEPLOY_KEY }}

maintain_repository:
name: Maintain Repository
runs-on: ubuntu-latest
Expand Down
64 changes: 64 additions & 0 deletions builder/Buildmac.md
@@ -0,0 +1,64 @@
# jellyfin-ffmpeg portable versions builder for mac

Portable versions builder of jellyfin-ffmpeg for macOS.

This script is generally made for GitHub Actions' CI runner, and there will be some caveats when running it locally.

A significant limitation is that this script will mutate files in a way that prevents the script from being executed multiple times on a non-clean environment. Follow the instructions below to work with it.

## Package List

For a list of included dependencies check the `scripts.d` directory.
Every file corresponds to its respective package.

For macOS, there will be additionally packages located in `images/macos` as extra static libs. The `00-dep.sh` will also setup necessary environment on a GitHub Runner. You can modify or remove it if you find it unnecessary.

## How to make a build

### Prerequisites

* **[Homebrew](https://brew.sh)**: Make sure Homebrew is installed and set up on your system.
* **[Xcode](https://developer.apple.com/xcode/)**: Ensure that Xcode is installed and properly configured. It's essential to have the full Xcode installation as simply installing the command line toolchain won't include the Metal SDK required for ffmpeg.
- **Verification**: To verify that you have the Metal SDK ready, run the command `xcrun -sdk macosx metal -v`. This command should display version information about the installed Metal SDK. If you encounter an error message such as `xcrun: error: unable to find utility "metal", not a developer tool or in PATH`, it indicates that the incorrect toolchain is selected. In such cases, manually select the Xcode toolchain by running the following command:
```
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
```

### Prepare Prefix Directory

You will need to prepare a directory to install all the libraries to. The default is `/opt/ffbuild/prefix`, which is defined in `buildmac.sh` as `FFBUILD_PREFIX`. You can either create this folder manually and give permission to the user who runs the builder, or you can modify that value to point it to another folder.

### Run Builder

Once you have your environment set up, you can simply run `buildmac.sh`, and it will download libraries and start building. This may take some time, so please be patient.

Generated artifacts will be stored to `artifacts` folder.

### Prepare for next running.

To run another clean build, the easiest way is to remove the `FFBUILD_PREFIX` folder, and then remove `jellyfin-ffmpeg` and re-clone the repo.

If you don't want to rebuild all the dependencies, you can keep the `FFBUILD_PREFIX` folder and remove/comment out the following lines:

```shell
mkdir build
for macbase in images/macos/*.sh; do
cd "$BUILDER_ROOT"/build
source "$BUILDER_ROOT"/"$macbase"
ffbuild_macbase || exit $?
done

cd "$BUILDER_ROOT"
for lib in scripts.d/*.sh; do
cd "$BUILDER_ROOT"/build
source "$BUILDER_ROOT"/"$lib"
ffbuild_enabled || continue
ffbuild_dockerbuild || exit $?
done
```

At this point, the repository could have our patches applied. You want to restore it with `quilt pop -af` before the next run.

## Known issue

- If you are on an Intel Mac and have `libx11` installed with Homebrew, ffmpeg will link to Homebrew's `libx11`, making your generated binary non-portable. We work around this on GitHub's Runner by removing all installed packages that use `libx11`.
2 changes: 2 additions & 0 deletions builder/README.md
Expand Up @@ -2,6 +2,8 @@

Portable versions builder of jellyfin-ffmpeg for Windows and Linux.

For macOS build instructions, please refer to [this documentation](./Buildmac.md).

## Package List

For a list of included dependencies check the `scripts.d` directory.
Expand Down
118 changes: 118 additions & 0 deletions builder/buildmac.sh
@@ -0,0 +1,118 @@
#!/bin/bash
set -xe
cd "$(dirname "$0")"
export BUILDER_ROOT="$(pwd)"
export FFBUILD_PREFIX="/opt/ffbuild/prefix"

get_output() {
(
SELF="$1"
source $1
if ffbuild_enabled; then
ffbuild_$2 || exit 0
else
ffbuild_un$2 || exit 0
fi
)
}

arch=$(uname -m)
TARGET="macarm64"
VARIANT="gpl"
if [ "$arch" = "arm64" ]; then
TARGET="macarm64"
elif [ "$arch" = "x86_64" ]; then
TARGET="mac64"
else
echo "Unknown architecture"
exit 1
fi

source "variants/${TARGET}-gpl.sh"

for addin in ${ADDINS[*]}; do
source "addins/${addin}.sh"
done

for script in scripts.d/*.sh; do
FF_CONFIGURE+=" $(get_output $script configure)"
FF_CFLAGS+=" $(get_output $script cflags)"
FF_CXXFLAGS+=" $(get_output $script cxxflags)"
FF_LDFLAGS+=" $(get_output $script ldflags)"
FF_LDEXEFLAGS+=" $(get_output $script ldexeflags)"
FF_LIBS+=" $(get_output $script libs)"
done

FF_CONFIGURE="$(xargs <<< "$FF_CONFIGURE")"
FF_CFLAGS="$(xargs <<< "$FF_CFLAGS")"
FF_CXXFLAGS="$(xargs <<< "$FF_CXXFLAGS")"
FF_LDFLAGS="$(xargs <<< "$FF_LDFLAGS")"
FF_LDEXEFLAGS="$(xargs <<< "$FF_LDEXEFLAGS")"
FF_LIBS="$(xargs <<< "$FF_LIBS")"
FF_HOST_CFLAGS="$(xargs <<< "$FF_HOST_CFLAGS")"
FF_HOST_LDFLAGS="$(xargs <<< "$FF_HOST_LDFLAGS")"
FFBUILD_TARGET_FLAGS="$(xargs <<< "$FFBUILD_TARGET_FLAGS")"

mkdir build
for macbase in images/macos/*.sh; do
cd "$BUILDER_ROOT"/build
source "$BUILDER_ROOT"/"$macbase"
ffbuild_macbase || exit $?
done

cd "$BUILDER_ROOT"
for lib in scripts.d/*.sh; do
cd "$BUILDER_ROOT"/build
source "$BUILDER_ROOT"/"$lib"
ffbuild_enabled || continue
ffbuild_dockerbuild || exit $?
done

cd "$BUILDER_ROOT"
cd ..
if [[ -f "debian/patches/series" ]]; then
ln -s debian/patches patches
quilt push -a
fi

./configure --prefix=/ffbuild/prefix \
$FFBUILD_TARGET_FLAGS \
--host-cflags="$FF_HOST_CFLAGS" \
--host-ldflags="$FF_HOST_LDFLAGS" \
--extra-version="Jellyfin" \
--extra-cflags="$FF_CFLAGS" \
--extra-cxxflags="$FF_CXXFLAGS" \
--extra-ldflags="$FF_LDFLAGS" \
--extra-ldexeflags="$FF_LDEXEFLAGS" \
--extra-libs="$FF_LIBS" \
$FF_CONFIGURE
make -j$(nproc) V=1

# We have to manually match lines to get version as there will be no dpkg-parsechangelog on macOS
PKG_VER=0.0.0
while IFS= read -r line; do
if [[ $line == jellyfin-ffmpeg* ]]; then
if [[ $line =~ \(([^\)]+)\) ]]; then
PKG_VER="${BASH_REMATCH[1]}"
break
fi
fi
done < "$BUILDER_ROOT"/../debian/changelog

PKG_NAME="jellyfin-ffmpeg_${PKG_VER}_portable_${TARGET}-${VARIANT}${ADDINS_STR:+-}${ADDINS_STR}"
ARTIFACTS_PATH="$BUILDER_ROOT"/artifacts
OUTPUT_FNAME="${PKG_NAME}.tar.xz"
cd "$BUILDER_ROOT"
mkdir -p artifacts
# bsdtar can add files in parent dir, but macOS's native archive utility won't be able to unzip it by double clicking, we have to move it to current dir as a workaround
mv ../ffmpeg ./
mv ../ffprobe ./
tar -cJf "${ARTIFACTS_PATH}/${OUTPUT_FNAME}" ffmpeg ffprobe
cd "${ARTIFACTS_PATH}"
sha256sum ./${OUTPUT_FNAME} > ./${OUTPUT_FNAME}.sha256sum
cd "$BUILDER_ROOT"/..

if [[ -n "$GITHUB_ACTIONS" ]]; then
echo "build_name=${BUILD_NAME}" >> "$GITHUB_OUTPUT"
echo "${OUTPUT_FNAME}" > "${ARTIFACTS_PATH}/${TARGET}-${VARIANT}${ADDINS_STR:+-}${ADDINS_STR}.txt"
fi
19 changes: 19 additions & 0 deletions builder/images/macos/00-dep.sh
@@ -0,0 +1,19 @@
#!/bin/bash

ffbuild_macbase() {
if [[ $TARGET == mac64 ]]; then
# For unknown reason ffmpeg insist to link to against libx11 and is hard to debug
# It does not link against libx11 on arm64, which is strange
# Remove everything using libx11 on github's x86 macOS host as a workaround
brew uninstall ant cairo gradle harfbuzz kotlin libxext libxft libxrender maven openjdk r sbt selenium-server libx11
fi
brew install wget subversion mercurial autoconf automake cmake meson ninja pkg-config coreutils gcc make python-setuptools pcre2 libtool gnu-sed gnu-tar nasm quilt
mkdir /opt/ffbuild/bin
cp "$BUILDER_ROOT"/images/base/git-mini-clone.sh /opt/ffbuild/bin/git-mini-clone
chmod +x /opt/ffbuild/bin/git-mini-clone
cp "$BUILDER_ROOT"/images/base/retry-tool.sh /opt/ffbuild/bin/retry-tool
chmod +x /opt/ffbuild/bin/retry-tool
cp "$BUILDER_ROOT"/images/base/check-wget.sh /opt/ffbuild/bin/check-wget
chmod +x /opt/ffbuild/bin/check-wget
export PATH="/opt/ffbuild/bin:$PATH"
}
9 changes: 9 additions & 0 deletions builder/images/macos/01-gettext.sh
@@ -0,0 +1,9 @@
#!/bin/bash
ffbuild_macbase() {
wget https://ftpmirror.gnu.org/gettext/gettext-0.22.5.tar.gz -O gettext.tar.gz
tar xvf gettext.tar.gz
cd gettext-0.22.5
./configure --disable-silent-rules --disable-shared --enable-static --with-included-glib --with-included-libcroco --with-included-libunistring --with-included-libxml --with-emacs --with-lispdir="$FFBUILD_PREFIX"/share --disable-java --disable-csharp --without-git --without-cvs --without-xz --with-included-gettext --prefix="$FFBUILD_PREFIX"
make -j$(nproc)
make install
}
18 changes: 18 additions & 0 deletions builder/images/macos/02-pcre2.sh
@@ -0,0 +1,18 @@
#!/bin/bash
# Although newer macOS has libpcre built-in, it is absent on macOS12
ffbuild_macbase() {
wget https://github.com/PCRE2Project/pcre2/releases/download/pcre2-10.42/pcre2-10.42.tar.bz2
tar xvf pcre2-10.42.tar.bz2
cd pcre2-10.42
./configure --prefix="$FFBUILD_PREFIX" \
--disable-shared \
--enable-static \
--disable-dependency-tracking \
--enable-pcre2-16 \
--enable-pcre2-32 \
--enable-pcre2grep-libz \
--enable-pcre2grep-libbz2 \
--enable-jit
make -j$(nproc)
make install
}
20 changes: 20 additions & 0 deletions builder/images/macos/03-glib2.sh
@@ -0,0 +1,20 @@
#!/bin/bash
ffbuild_macbase() {
wget https://download.gnome.org/sources/glib/2.78/glib-2.78.4.tar.xz
tar xvf glib-2.78.4.tar.xz
cd glib-2.78.4
meson setup build --default-library=both \
--localstatedir="$FFBUILD_PREFIX"/var \
-Dgio_module_dir="$FFBUILD_PREFIX"/lib/gio/modules \
-Dbsymbolic_functions=false \
-Ddtrace=false \
-Druntime_dir="$FFBUILD_PREFIX"/var/run \
-Dtests=false \
--prefix="$FFBUILD_PREFIX" \
--buildtype=release \
--default-library=static

sed -i '' 's/MacOSX11\.sdk/MacOSX\.sdk/g' build/build.ninja
meson compile -C build --verbose
meson install -C build
}