diff --git a/docs/self-hosted-runner/SelfHostedRunner.md b/docs/self-hosted-runner/SelfHostedRunner.md new file mode 100644 index 000000000..8a2db7284 --- /dev/null +++ b/docs/self-hosted-runner/SelfHostedRunner.md @@ -0,0 +1,143 @@ +# arduino-ide 2.0.0 for Raspberry Pi + +These are instructions on compiling [arduino-ide 2](https://github.com/arduino/arduino-ide) for arm64 using self-hosted runners. Compiling can be done on raspberry pi os 64-bit or ubuntu arm64. The compiled binaries run on raspberry pi os 64-bit and ubuntu arm64. + +Compiling is done using github actions, the same build process as on X86. + +Because github does not offer linux arm64 runners, compilation is done using a self-hosted runner on a raspberry pi. + +Because github does not recommend running self-hosted runners on public repositories, the arduino-ide repository is forked first, and public PR are switched off on the fork. + +For repeatability and security, compilation is done inside a docker. + +## fork arduino-ide + +Set up a copy of the arduino-ide github. + +- in github, fork [arduino/arduino-ide](https://github.com/arduino/arduino-ide) +- patch your fork with the file _self_hosted_runner.patch_. The patch adds a new build target "self-hosted" to arduino-ide, next to the existing Windows, Ubuntu and MacOS. +- for cosmetics, set a tag to show the release name + +``` +git clone https://github.com/your_git_account/arduino-ide +cd arduino-ide +patch -p1 < docs/self-hosted-runner/self_hosted_runner.patch +git add .github/workflows/build.yml +git add arduino-ide-extension/scripts/download-ls.js +git commit -m arm64 +git push +git tag -a 2.0.0-rc8-arm64 -m self-hosted +git push --tags +``` + + + +## set up docker + +For repeatability and security, the self-hosted runner runs in a docker image. + +- in github, fork [myoung34/docker-github-actions-runner](https://github.com/myoung34/docker-github-actions-runner). This is a dockerfile for self-hosted runners. +- patch your fork with the file _docker-github-actions-runner.patch_. The patch adds build dependencies - node, yarn, and libraries - to the docker image. + +``` +git clone https://github.com/your_git_account/docker-github-actions-runner +cd docker-github-actions-runner +patch -p1 < ../arduino-ide/docs/self-hosted-runner/docker-github-actions-runner.patch +git commit -m arduino-ide +git push +``` + +## configure self-hosted runner +Configure a new self-hosted runner. +On github.com, go to your fork of the arduino-ide. + +- In _Settings -> Moderation options -> Code review limits_, enable "Limit to users explicitly granted read or higher access" +- In _Settings -> General -> Code and Automation -> Actions -> Runners_, click on _New Self-Hosted Runner_, Linux, ARM64. + +## set up raspberry pi + +On the raspberry pi runner: + +- Install Raspberry Pi OS 64-bit or ubuntu arm64. +- install docker using the [convenience script](https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script). + +Copy the shell script to run the local runner to the raspberry pi: + +``` +wget https://raw.githubusercontent.com/arduino/arduino-ide/main/docs/self-hosted-runner/docker-github-actions-runner.sh +chmod +x docker-github-actions-runner.sh +``` + +Edit `docker-github-actions-runner.sh`: + +- ACCESS_TOKEN is your github personal access token. +- REPO_URL is the url of your fork of the arduino-ide +- RUNNER_URL is the url of your fork of docker-github-actions-runner +- WORK_DIR is a directory where the script may create files + +This script will + +- clone RUNNER_URL in WORK_DIR +- create a docker image _self-hosted-runner_, if the image does not exist already +- run the docker image + +Run the script to start the self-hosted runner: +``` +./docker-github-actions-runner.sh +``` +After installing some packages, a self-hosted runner starts up: + +``` +-------------------------------------------------------------------------------- +| ____ _ _ _ _ _ _ _ _ | +| / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ | +| | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| | +| | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ | +| \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ | +| | +| Self-hosted runner registration | +| | +-------------------------------------------------------------------------------- + +# Authentication +``` + +Output should end with `Listening for Jobs` + +## start the build + +On github.com, go to your fork of the arduino-ide. + +- In "Actions", under "Workflows" choose "Arduino IDE", click "Enable Workflow". +- Click "Run workflow". Use workflow from "Tags: 2.0.0-rc8-arm64" (the tag created above) +- On the runner, output should be ``Running job: build (self-hosted)``. You can follow what happens in the build through the github web interface. + +Build time is less than one hour on a raspberry pi 4b, 8gb ram. +``` +√ Connected to GitHub + +Current runner version: '2.294.0' +2022-06-23 13:07:35Z: Listening for Jobs +2022-06-23 13:09:38Z: Running job: build (self-hosted) +2022-06-23 13:54:14Z: Job build (self-hosted) completed with result: Succeeded +``` + +## download binaries + +- After the run, arm64 binaries for raspberry pi are on github, in "Artifacts." +- On github.com, go to your fork of the arduino-ide. Under "All workflows - Showing runs from all workflows" click on "Arduino IDE". The binaries are under "Artifacts - Produced during runtime +": +``Linux_ARM64_app_image`` and +``Linux_ARM64_zip``. Click to download. +- When the build is completed and the binaries downloaded, stop the runner on the raspberry and delete your arduino-ide fork on github. It is no longer needed. + +## note + +Once created, the docker image _self-hosted-runner_ is used until it is removed. + +If you modify one of the shell variables in `docker-github-actions-runner.sh`, remove the docker image to force a rebuild: +`` +docker rmi self-hosted-runner +`` + +not truncated. diff --git a/docs/self-hosted-runner/docker-github-actions-runner.patch b/docs/self-hosted-runner/docker-github-actions-runner.patch new file mode 100644 index 000000000..c6f6b5944 --- /dev/null +++ b/docs/self-hosted-runner/docker-github-actions-runner.patch @@ -0,0 +1,46 @@ +diff --git a/Dockerfile b/Dockerfile +index 4d87988..870bd7b 100644 +--- a/Dockerfile ++++ b/Dockerfile +@@ -1,12 +1,12 @@ + # hadolint ignore=DL3007 +-FROM myoung34/github-runner-base:latest ++FROM myoung34/github-runner-base:ubuntu-bionic + LABEL maintainer="myoung34@my.apsu.edu" + + ENV AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache + RUN mkdir -p /opt/hostedtoolcache + + ARG GH_RUNNER_VERSION="2.294.0" +-ARG TARGETPLATFORM ++ARG TARGETPLATFORM="linux/arm64" + + SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +@@ -17,6 +17,12 @@ RUN chmod +x /actions-runner/install_actions.sh \ + && /actions-runner/install_actions.sh ${GH_RUNNER_VERSION} ${TARGETPLATFORM} \ + && rm /actions-runner/install_actions.sh + ++# install arduino ide build dependencies ++RUN apt-get install -y libxkbfile-dev libsecret-1-dev ++ ++# install nvm ++RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash ++ + COPY token.sh entrypoint.sh / + RUN chmod +x /token.sh /entrypoint.sh + +diff --git a/entrypoint.sh b/entrypoint.sh +index 3b29d58..5df4be8 100644 +--- a/entrypoint.sh ++++ b/entrypoint.sh +@@ -124,5 +124,9 @@ if [[ ${_DISABLE_AUTOMATIC_DEREGISTRATION} == "false" ]]; then + trap deregister_runner SIGINT SIGQUIT SIGTERM INT TERM QUIT + fi + ++. $HOME/.nvm/nvm.sh ++nvm install 14.0.0 ++npm install --global yarn ++ + # Container's command (CMD) execution + "$@" diff --git a/docs/self-hosted-runner/docker-github-actions-runner.sh b/docs/self-hosted-runner/docker-github-actions-runner.sh new file mode 100755 index 000000000..970c3eb73 --- /dev/null +++ b/docs/self-hosted-runner/docker-github-actions-runner.sh @@ -0,0 +1,65 @@ +#!/bin/bash -x + +# before use, modify: +# ACCESS_TOKEN is your github access token +# REPO_URL is the url of your fork of https://github.com/arduino/arduino-ide +# RUNNER_URL is the url of your fork of https://github.com/myoung34/docker-github-actions-runner +# WORK_DIR + +WORK_DIR=~/src +ACCESS_TOKEN=ghp_YOUR_GITHUB_PERSONAL_ACCESS_TOKEN +REPO_URL=https://github.com/YOUR_GITHUB_ACCOUNT/arduino-ide +RUNNER_URL=https://github.com/arduino/docker-github-actions-runner + +# check docker and git available + +if ! command -v docker +then + echo install docker +fi + +if ! command -v git +then + echo install git +fi + +# check if docker image exists + +if [ ! $(docker image inspect self-hosted-runner '--format=""') ] +then + # build docker image + + if [ ! -d ${WORK_DIR} ] + then + mkdir -p ${WORK_DIR} + fi + + DOCKER_DIR=${WORK_DIR}/docker-github-actions-runner + + if [ -d ${DOCKER_DIR} ] + then + echo ${DOCKER_DIR} already exists + exit 0 + fi + + git clone ${RUNNER_URL} ${DOCKER_DIR} + + if [ ! -f ${DOCKER_DIR}/Dockerfile ] + then + echo no Dockerfile + exit 0 + fi + + docker build ${DOCKER_DIR} --tag self-hosted-runner + + if [ ! $(docker image inspect self-hosted-runner '--format=""') ] + then + echo no docker image + exit 0 + fi + +fi + +docker run --env "ACCESS_TOKEN=${ACCESS_TOKEN}" --env RUNNER_SCOPE=repo --env "REPO_URL=${REPO_URL}" --env RUNNER_NAME=arduino-ide-runner self-hosted-runner + +#not truncated diff --git a/docs/self-hosted-runner/self_hosted_runner.patch b/docs/self-hosted-runner/self_hosted_runner.patch new file mode 100644 index 000000000..e48b854d1 --- /dev/null +++ b/docs/self-hosted-runner/self_hosted_runner.patch @@ -0,0 +1,70 @@ +diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml +index 3dceb0c..076b64e 100644 +--- a/.github/workflows/build.yml ++++ b/.github/workflows/build.yml +@@ -37,17 +37,18 @@ jobs: + strategy: + matrix: + config: +- - os: windows-2019 +- certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX # Name of the secret that contains the certificate. +- certificate-password-secret: WINDOWS_SIGNING_CERTIFICATE_PASSWORD # Name of the secret that contains the certificate password. +- certificate-extension: pfx # File extension for the certificate. +- - os: ubuntu-18.04 # https://github.com/arduino/arduino-ide/issues/259 +- - os: macos-latest +- # APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from: +- # https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate +- certificate-secret: APPLE_SIGNING_CERTIFICATE_P12 +- certificate-password-secret: KEYCHAIN_PASSWORD +- certificate-extension: p12 ++# - os: windows-2019 ++# certificate-secret: WINDOWS_SIGNING_CERTIFICATE_PFX # Name of the secret that contains the certificate. ++# certificate-password-secret: WINDOWS_SIGNING_CERTIFICATE_PASSWORD # Name of the secret that contains the certificate password. ++# certificate-extension: pfx # File extension for the certificate. ++# - os: ubuntu-18.04 # https://github.com/arduino/arduino-ide/issues/259 ++# - os: macos-latest ++# # APPLE_SIGNING_CERTIFICATE_P12 secret was produced by following the procedure from: ++# # https://www.kencochrane.com/2020/08/01/build-and-sign-golang-binaries-for-macos-with-github-actions/#exporting-the-developer-certificate ++# certificate-secret: APPLE_SIGNING_CERTIFICATE_P12 ++# certificate-password-secret: KEYCHAIN_PASSWORD ++# certificate-extension: p12 ++ - os: self-hosted + runs-on: ${{ matrix.config.os }} + timeout-minutes: 90 + +@@ -62,6 +63,7 @@ jobs: + registry-url: 'https://registry.npmjs.org' + + - name: Install Python 3.x ++ if: matrix.config.os != 'self-hosted' + uses: actions/setup-python@v2 + with: + python-version: '3.x' +@@ -114,6 +116,10 @@ jobs: + name: Linux_X86-64_zip + - path: '*Linux_64bit.AppImage' + name: Linux_X86-64_app_image ++ - path: '*Linux_ARM64.zip' ++ name: Linux_ARM64_zip ++ - path: '*Linux_ARM64.AppImage' ++ name: Linux_ARM64_app_image + - path: '*macOS_64bit.dmg' + name: macOS_dmg + - path: '*macOS_64bit.zip' +diff --git a/arduino-ide-extension/scripts/download-ls.js b/arduino-ide-extension/scripts/download-ls.js +index 81b9329..7f63204 100755 +--- a/arduino-ide-extension/scripts/download-ls.js ++++ b/arduino-ide-extension/scripts/download-ls.js +@@ -81,6 +81,12 @@ + lsSuffix = 'Linux_64bit.tar.gz'; + clangdSuffix = 'Linux_64bit'; + break; ++ case 'linux-arm64': ++ clangdExecutablePath = path.join(build, 'clangd'); ++ clangFormatExecutablePath = path.join(build, 'clang-format'); ++ lsSuffix = 'Linux_ARM64.tar.gz'; ++ clangdSuffix = 'Linux_ARM64'; ++ break; + case 'win32-x64': + clangdExecutablePath = path.join(build, 'clangd.exe'); + clangFormatExecutablePath = path.join(build, 'clang-format.exe');