Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI for building the fuzzers #1183

Draft
wants to merge 34 commits into
base: master
Choose a base branch
from
Draft
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
73 changes: 73 additions & 0 deletions .github/workflows/build-fuzzer.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Copyright (C) 2020 Matthew Glazar
# See end of file for extended copyright information.

name: build the fuzzers
on:
push:
pull_request:
types: [opened, synchronize]

jobs:
build:
name: ${{ matrix.toolchain.name }}
strategy:
fail-fast: false
matrix:
toolchain:
- { runs_on: ubuntu-latest, name: "Clang 13 libc++", container: "ghcr.io/quick-lint/quick-lint-js-github-clang:v1", CC: clang-13, CXX: clang++-13, CFLAGS: "-stdlib=libc++", CMAKE_BUILD_TYPE: "Debug", }
- { runs_on: ubuntu-latest, name: "Clang 13 libstdc++", container: "ghcr.io/quick-lint/quick-lint-js-github-clang:v1", CC: clang-13, CXX: clang++-13, CFLAGS: "-stdlib=libstdc++", CMAKE_BUILD_TYPE: "Debug", }
- { runs_on: ubuntu-latest, name: "Clang 13 Release libc++", container: "ghcr.io/quick-lint/quick-lint-js-github-clang:v1", CC: clang-13, CXX: clang++-13, CFLAGS: "-stdlib=libc++", CMAKE_BUILD_TYPE: "Release", }
- { runs_on: ubuntu-latest, name: "Clang 13 Release libstdc++", container: "ghcr.io/quick-lint/quick-lint-js-github-clang:v1", CC: clang-13, CXX: clang++-13, CFLAGS: "-stdlib=libstdc++", CMAKE_BUILD_TYPE: "Release", }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our CI is slow, and I don't want to add too many more slow jobs.

Let's cut this down to one build. Just "Clang 13 Release libstdc++" should be good.

runs-on: ${{ matrix.toolchain.runs_on }}
container: ${{ matrix.toolchain.container }}
env:
CMAKE_BUILD_TYPE: ${{ matrix.toolchain.CMAKE_BUILD_TYPE }}
CMAKE_C_COMPILER: ${{ matrix.toolchain.CC }}
CMAKE_C_FLAGS: ${{ matrix.toolchain.CFLAGS }}
CMAKE_CXX_COMPILER: ${{ matrix.toolchain.CXX }}
CMAKE_CXX_FLAGS: ${{ matrix.toolchain.CFLAGS }}

steps:
- name: checkout
uses: actions/checkout@8e5e7e5ab8b370d6c329ec480221332ada57f0ab # v3.5.2

- name: install dependencies (Homebrew)
if: ${{ matrix.toolchain.homebrew_packages }}
run: brew install ${{ matrix.toolchain.homebrew_packages }}

- name: configure
run: |
env | grep '^ASAN_OPTIONS\|^CMAKE\|^QUICK_LINT_JS' | sort
mkdir build
cd build
CC=$CMAKE_C_COMPILER CXX=$CMAKE_CXX_COMPILER CFLAGS='-fsanitize=address,undefined,fuzzer-no-link $CMAKE_C_FLAGS' CXXFLAGS='-fsanitize=address,undefined,fuzzer-no-link $CMAKE_CXX_FLAGS' cmake -G Ninja -DCMAKE_BUILD_TYPE=$CMAKE_BUILD_TYPE -DQUICK_LINT_JS_ENABLE_LLVM_LIBFUZZER_TESTS=ON ..
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Also add -DBUILD_TESTING=NO to speed up compilation.

cd ..
rol1510 marked this conversation as resolved.
Show resolved Hide resolved
shell: bash

- name: build
run: ninja -C build

- name: sample run
run: |
mkdir fuzz-tmp
ls build/fuzz/
# try running every fuzzer for a very short time
for FILE in build/fuzz/quick-lint-js-fuzz-*; do echo running: $FILE; $FILE fuzz-tmp -runs=100 || exit 1; done
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must fix: We should not run the fuzzers on PRs or branches. Random fuzzer failures should not block a PR or a release.

We can run fuzzers nightly, for example. We can also run fuzzers on PRs and branches if there is a fixed seed or corpus.

In other words:

Job trigger Compile fuzzers Run fuzzers Acceptable?
PR yes no OK
PR yes yes, fixed seed or corpus OK
PR yes yes, random seed BAD (introduces flakiness)
branch push yes no OK
branch push yes yes, fixed seed or corpus OK
branch push yes yes, random seed BAD (introduces flakiness)
nightly yes no BAD (PRs/branches do this already; waste of resources)
nightly yes yes, fixed seed or corpus BAD (PRs/branches do this already; waste of resources)
nightly yes yes, random seed OK


# quick-lint-js finds bugs in JavaScript programs.
# Copyright (C) 2020 Matthew Glazar
#
# This file is part of quick-lint-js.
#
# quick-lint-js is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# quick-lint-js is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with quick-lint-js. If not, see <https://www.gnu.org/licenses/>.