Skip to content

Commit

Permalink
Add a ci workflow
Browse files Browse the repository at this point in the history 
This `ci` replaces the following workflows:

- `pages`
- `publish-mempool-benchmarks`
- `haskell-ci`

We build and deploy the GitHub pages on the same workflow because we
need to publish the haddocks, which depend on the Cabal build.
Similarly, once we publish the benchmarks in our site we will need to
share the benchmarks results with the job that publishes the GitHub
pages. Sharing artifacts or caches between GitHub workflows is not
supported, hence we need to put all these jobs in the same workflow.

This commit also removes support for publishing the benchmark plots. The
action we used
https://github.com/benchmark-action/github-action-benchmark is not
compatible with the new method we use to publish GitHub pages.
  • Loading branch information
@dnadales
dnadales committed Jun 2, 2023
1 parent 9f2b8bb commit 96c131b
Showing 1 changed file with 293 additions and 0 deletions.
293 changes: 293 additions & 0 deletions .github/workflows/ci.yml
View file
@@ -0,0 +1,293 @@
name: CI

# Limit concurrent runs of this workflow within a single PR
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

on:
workflow_dispatch:
pull_request:
merge_group:
push:
branches:
- main

jobs:
check-cabal-files:
runs-on: ubuntu-latest
steps:
- name: Install Haskell
uses: input-output-hk/setup-haskell@v1
id: setup-haskell
with:
cabal-version: "3.10.1.0"

- uses: actions/checkout@v3

- name: Cabal check
run: ./scripts/ci/check-cabal-files.sh

build-test-bench-haddocks:
name: Build and test
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
ghc: ["8.10.7"] # ["9.2.7"]
cabal: ["3.10.1.0"]
os: [ubuntu-latest]
env:
# Modify this value to "invalidate" the Cabal cache.
CABAL_CACHE_VERSION: "2023-04-30"
# Modify this value to "invalidate" the secp cache.
SECP_CACHE_VERSION: "2023-04-27"
# current ref from: 27.02.2022
SECP256K1_REF: ac83be33d0956faf6b7f61a60ab524ef7d6a473a

steps:
- uses: actions/checkout@v3

- name: Install Haskell
uses: input-output-hk/setup-haskell@v1
id: setup-haskell
with:
ghc-version: ${{ matrix.ghc }}
cabal-version: ${{ matrix.cabal }}

- uses: actions/checkout@v3

- name: Install build environment
run: |
sudo apt-get update
sudo apt-get -y install libsodium-dev
- name: Configure Cabal to use libsodium
run: |
cat > cabal.project.local <<EOF
package cardano-crypto-praos
flags: -external-libsodium-vrf
EOF
cat cabal.project.local
- name: "LINUX: Install build environment (for secp256k1)"
run: sudo apt-get -y install autoconf automake libtool

- name: Install secp256k1
uses: input-output-hk/setup-secp256k1@v1
with:
git-ref: ${{ env.SECP256K1_REF }}
cache-version: ${{ env.SECP_CACHE_VERSION }}

- name: Update Cabal (Hackage and CHaP)
run: cabal update

- name: Generate dist-newstyle/cache/plan.json
run: cabal build all --dry-run --minimize-conflict-set

# We create a `dependencies.txt` file that can be used to index the cabal-store cache.
#
# We do not use `plan.json` directly because adding a dependency to our
# Cabal files which was already present somewhere else would result in a
# diferent plan, even though the set of dependences is the same.
#
# In the future we should consider using `cabal-cache` like in the
# `cardano-node`'s GitHub workflow.
- name: Record dependencies to be used as cache keys
id: record-deps
run: |
cabal build all --dry-run
cat dist-newstyle/cache/plan.json \
| jq -L .github/workflows/jq-install-plan \
| sort \
| uniq > dependencies.txt
- uses: actions/cache@v3
name: "Cache `cabal store`"
with:
path: ${{ steps.setup-haskell.outputs.cabal-store }}
key: cache-store-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('dependencies.txt') }}
restore-keys: cache-dependencies-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}

# This is used for caching the `dist-newstyle` directory.
- name: Store week number as environment variable
run: echo "WEEKNUM=$(/usr/bin/date -u '+%W')" >> $GITHUB_ENV

# When we restore a previous cache and store a new key, we also store files
# that were part of the last restoration but were not actually used. To
# prevent the cache from growing too quickly we only store a new cache every
# week.
- uses: actions/cache@v3
name: "Cache `dist-newstyle`"
with:
path: |
dist-newstyle
!dist-newstyle/**/.git
key: cache-dist-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}-${{ env.WEEKNUM }}
restore-keys: cache-dist-${{ env.CABAL_CACHE_VERSION }}-${{ runner.os }}-${{ matrix.ghc }}

- name: Build dependencies
run: cabal build --only-dependencies all -j

- name: Build projects [build]
run: cabal build all -j

- name: Test
run: cabal test all -j

- name: Create baseline-benchmark
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
run: |
cabal new-run ouroboros-consensus:mempool-bench -- \
--timeout=60 --csv mempool-benchmarks.csv \
+RTS -T
# TODO: we only care about saving the baseline results when we run on the
# main branch. However we need to restore the cache when we run the
# benchmarks during PRs. The caching mechanism of GitHub actions does not
# allow not to save a cache entry.
#
# The `run_id` is only relevant to store a new benchmarking result when we
# run on the `main` branch. If we run this workflow in the context of a PR,
# then we will save the same results we just restored.
- name: Cache benchmark baseline results
uses: actions/cache@v3
with:
path: baseline-mempool-benchmarks.csv
key: baseline-mempool-benchmarks-${{ runner.os }}-${{ matrix.ghc }}-${{ github.run_id }}
restore-keys: baseline-mempool-benchmarks-${{ runner.os }}-${{ matrix.ghc }}

# We only update the cache if we just ran a benchmark on main.
- name: Copy baseline-benchmark to cache
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
run: cp mempool-benchmarks.csv baseline-mempool-benchmarks.csv

# TODO: this will be necessary when we publish the benchmarks results.
# - name: Upload mempool benchmark baseline results
# if: github.event_name == 'push' && github.ref == 'refs/heads/main'
# uses: actions/upload-artifact@v3
# with:
# name: baseline-mempool-benchmarks
# path: baseline-mempool-benchmarks.csv

# The `fail-if-slower` value is determined ad-hoc based on the variability
# we observed in our benchmarks.
#
# TODO: Uncomment this after this is merged to main. We need to generate the baseline benchmarks first.
# - name: Run mempool benchmarks on pull requests
# if: ${{ github.event_name == 'pull_request' }}
# run: |
# cabal new-run ouroboros-consensus:mempool-bench -- \
# --timeout=60 --baseline baseline-mempool-benchmarks.csv \
# --fail-if-slower 20 \
# +RTS -T

# NB: build the haddocks at the end to avoid unecessary recompilations.
# We build the haddocks only for one GHC version.
- name: Build Haddock documentation
if: |
github.event_name == 'push'
&& github.ref == 'refs/heads/main'
&& ${{ matrix.os=='ubuntu-latest' }}
&& ${{ matrix.ghc=='8.10.7' }}
run: |
cabal build --dry-run --enable-tests all
./scripts/docs/haddocks.sh
tar vzcf haddocks.tgz ./docs/website/static/haddocks
- name: Upload haddocks as an artifact
if: |
github.event_name == 'push'
&& github.ref == 'refs/heads/main'
&& ${{ matrix.os=='ubuntu-latest' }}
&& ${{ matrix.ghc=='8.10.7' }}
uses: actions/upload-artifact@v3
with:
name: haddocks
path: haddocks.tgz
retention-days: 1

# NB: build the haddocks at the end to avoid unecessary recompilations.
# We build the haddocks only for one GHC version.
- name: Build Haddock documentation
if: |
github.event_name == 'push'
&& github.ref == 'refs/heads/main'
&& ${{ matrix.os=='ubuntu-latest' }}
&& ${{ matrix.ghc=='8.10.7' }}
run: |
cabal build --dry-run --enable-tests all
./scripts/docs/haddocks.sh
tar vzcf haddocks.tgz ./docs/website/static/haddocks
- name: Upload haddocks as an artifact
if: |
github.event_name == 'push'
&& github.ref == 'refs/heads/main'
&& ${{ matrix.os=='ubuntu-latest' }}
&& ${{ matrix.ghc=='8.10.7' }}
uses: actions/upload-artifact@v3
with:
name: haddocks
path: haddocks.tgz
retention-days: 1


deploy-documentation:
name: Deploy documentation to GitHub Pages
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: build-test-bench-haddocks
runs-on: ubuntu-latest
defaults:
run:
working-directory: ./docs/website
strategy:
fail-fast: false
steps:
- uses: actions/checkout@v3

- uses: actions/setup-node@v3
with:
node-version: 18
cache: yarn
cache-dependency-path: './docs/website/yarn.lock'

- uses: cachix/install-nix-action@v20

- name: Build PDFs (Consensus report)
run: |
nix build -L .#consensus-pdfs
cp -r --no-preserve=mode,ownership result/ static/pdfs
- name: Download haddocks
uses: actions/download-artifact@v3
with:
name: haddocks

- name: Copy haddocks
run: |
cd ../../
tar vzxf haddocks.tgz
- name: Install dependencies
run: yarn install --frozen-lockfile

- name: Build website
run: yarn build

- uses: actions/upload-pages-artifact@v1
with:
path: ./docs/website/build

- name: Deploy
id: deployment
uses: actions/deploy-pages@v2

# https://github.com/actions/deploy-pages
permissions:
pages: write
id-token: write

environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}

0 comments on commit 96c131b

Please sign in to comment.