Skip to content

Commit

Permalink
[WIP] Auto-install Rust compiler toolchain for CI and JS people
Browse files Browse the repository at this point in the history
Since we brought the WebAssembly MPD parser to fix performance issues we
had with some contents at canal+, we added another language (Rust) to
the project which is not compatible with most of the JS-world tools we and
other JS developers usually deal with.

Now that we added the `MULTI_THREAD` experimental feature and embeds,
the WebAssembly generation from that other language is even a required
step in our main build script (`npm run build`).

Consequently, when another developer or a CI needs to build an RxPlayer
(e.g. to test a development before a release is made), they have to
install:
  1. NodeJS + npm
  2. our JavaScript dependencies
  3. A Rust toolchain able to output WebAssembly. We generally go
     through `rustup` here basically an helper tool to install Rust
     toolchains
  4. `binaryen`, which is a collection of tools to transform that
     WebAssembly file.

(1) and (2) are very common use cases that both JS people and CI workflows
do very easily: they mostly already have nodejs + npm installed locally
(or it's very simple to set-up) and the second is only an `npm install`
call away.

(3) and (4) however, though not hard (you have to install rustup and
binaryen which is pretty simple on all platforms we relied on), are more
unusual dependencies that JS developers might be reticent to install and
that are more work to set-up on a CI.

Consequently I'm doing here a proof-of-concept where those tools are
auto-installed locally if not found when doing an `npm run build` call,
letting both people and CI produce a build easily without having to deal
with the dual-language complexity.

---

For rustup it's pretty straightforward as their install script already
take care of the platform detection, target installation and so on. All
I had to do is to make sure binaries were installed locally (`./tmp`) and
did not pollute the developers environment (e.g. no `PATH`
modification).

For binaryen however, it's less simple as they don't propose such
auto-install system.
however, I noticed that they propose prebuilt binaries for the most usual
platforms on their github's release page:
https://github.com/WebAssembly/binaryen/releases

The idea here (not yet finished), is to automatically detect the
platform of the user and fetch the right binaryen binary locally to then
rely on both it and the local rustup to produce builds in case the
command isn't available globally.

Also, because it will be faster and less error-prone for frequent
developers to install both of those globally once and for all, we print a
notice explaining what we do when that auto-install script is run.
  • Loading branch information
peaBerberian committed Feb 2, 2024
1 parent d134224 commit 5ca37f2
Show file tree
Hide file tree
Showing 5 changed files with 391 additions and 5 deletions.
4 changes: 0 additions & 4 deletions .github/workflows/checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,9 @@ jobs:
with:
node-version: ${{ matrix.node-version }}
cache: 'npm'
- uses: actions-rs/toolchain@v1
with:
toolchain: stable
# needed for integration & memory tests codecs support
- run: sudo add-apt-repository multiverse && sudo apt update && sudo apt install -y ubuntu-restricted-extras
- run: npm install
- run: rustup target add wasm32-unknown-unknown
- run: npm run build
# Firefox seems to have issue with integration tests on GitHub actions only
# TODO to check
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
/tmp
/*.keys
/*.log
/.scannerwork
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@
"build:dev": "./scripts/generate_build.mjs --dev-mode",
"build:all": "npm run build:wasm:release && npm run bundle && npm run bundle:min && npm run build",
"build:wasm:debug": "mkdir -p dist && cd ./src/parsers/manifest/dash/wasm-parser && cargo build --target wasm32-unknown-unknown && cp target/wasm32-unknown-unknown/debug/mpd_node_parser.wasm ../../../../../dist/mpd-parser.wasm",
"build:wasm:release": "mkdir -p dist && cd ./src/parsers/manifest/dash/wasm-parser && cargo build --target wasm32-unknown-unknown --release && node ../../../../../scripts/wasm-optimize.mjs target/wasm32-unknown-unknown/release/mpd_node_parser.wasm ../../../../../dist/mpd-parser.wasm && cd ../../../../../ && npm run wasm-strip",
"build:wasm:release": "./scripts/build_wasm_release.sh",
"bundle": "webpack --progress --config webpack.config.mjs --env production",
"bundle:min": "webpack --progress --config webpack.config.mjs --env minify --env production",
"bundle:min:watch": "webpack --progress --config webpack.config.mjs -w --env production --env minify",
Expand Down
61 changes: 61 additions & 0 deletions scripts/build_wasm_release.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/bin/sh

# TODO documentation

print_toolchain_installation_notice() {
echo " +----------------------------------------------------------------------------------+"
echo " | A rust toolchain will be installed locally in a temporary directory (./tmp). |"
echo " | If you intend to develop on the RxPlayer regularly, it is recommended that you |"
echo " | install globally rustup (with the \"wasm32-unknown-unknown\" target) as well as |"
echo " | binaryen. |"
echo " +----------------------------------------------------------------------------------+"
}

has_local_cargo=false
has_local_wasmopt=false
has_installed=false

mkdir -p dist

if [ -f tmp/cargo/bin/cargo ]; then
has_local_cargo=true
elif ! cargo_loc="$(type -p "cargo")" || [[ -z $cargo_loc ]]; then
echo "WARNING: cargo command not found."
print_toolchain_installation_notice
sleep 1
./scripts/install_rust_toolchain.sh
install_rust_toolchain
has_local_cargo=true
has_installed=true
fi

if [ -f tmp/binaryen/bin/wasm-opt ]; then
has_local_wasmopt=true
elif ! wasmopt_loc="$(type -p "wasm-opt")" || [[ -z $wasmopt_loc ]]; then
if $has_installed; then
>&2 echo "Error: did not succeed to install binaryen dependency. Please install it manually."
exit 1
fi

echo "WARNING: wasm-opt command not found."
print_toolchain_installation_notice
sleep 1
./scripts/install_rust_toolchain.sh
has_local_wasmopt=true
fi

# Move to MPD parser directory
cd ./src/parsers/manifest/dash/wasm-parser
echo "Building mpd-parser WebAssembly file with Cargo..."
if $has_local_cargo; then
../../../../../tmp/cargo/bin/cargo build --target wasm32-unknown-unknown --release
else
cargo build --target wasm32-unknown-unknown --release
fi

echo "Optimizing mpd-parser WebAssembly build..."
if $has_local_wasmopt; then
../../../../../tmp/binaryen/bin/wasm-opt target/wasm32-unknown-unknown/release/mpd_node_parser.wasm --signext-lowering --strip-dwarf -O4 -o ../../../../../dist/mpd-parser.wasm
else
wasm-opt target/wasm32-unknown-unknown/release/mpd_node_parser.wasm --signext-lowering --strip-dwarf -O4 -o ../../../../../dist/mpd-parser.wasm
fi
Loading

0 comments on commit 5ca37f2

Please sign in to comment.