Skip to content
Tree: 142c38ee4d
Commits on Mar 20, 2019
  1. dashboard/app: request test/bisect jobs separately

    dvyukov committed Mar 20, 2019
    Allow separate sets of managers for patch testing and for bisection.
    This makes things more flexible on syz-ci deployment side.
    Remove previous hacks for bisection deployment.
    Update #501
  2. pkg/bisect: use CheckoutCommit instead of SwitchCommit

    dvyukov committed Mar 20, 2019
    Use CheckoutCommit instead of SwitchCommitto get the crashing commit.
    The problem is with trees like linux-next. They require at least fetching tags
    and maybe even different tree (-history).
    Use CheckoutCommit which at least fetches tags which is enough
    for recent commits.
    Update #501
  3. pkg/vcs: wrap git invocations in a helper method

    dvyukov committed Mar 20, 2019
    There is a bunch of repetition to invoke git.
    Wrap it into a helper method.
  4. pkg/vcs: remove unused functions

    dvyukov committed Mar 20, 2019
    Bisect is not needed after introduction of Bisecter interface.
  5. executor: update fdio import path

    mvanotti authored and dvyukov committed Mar 20, 2019
    The Fuchsia team is going to remove the `lib/fdio/util.h` library. They
    have already moved all the functions to new header files.
    I have seen that fuchsia uses `fdio_service_connect`, which has been
    moved to the `lib/fdio/directory.h` header file.
    This commit just changes the import path in the fuchsia executor, and in
    the corresponding generated go file (I made that change by running `make
  6. docs, pkg/vcs, sys/fuchsia: update fuchsia urls

    mvanotti authored and dvyukov committed Mar 20, 2019
    Recently the fuchsia team decided to merge all their subrepos into one
    big git repo. This meant that the "zircon" repo doesn't exist anymore.
    Instead almost everything is under the fuchsia repo.
    This change updates all the reference I could find in the code that
    point to the zircon and docs repo to make them point to the new fuchsia
  7. syz-manager: fix typo in dashboard stats upload error

    ajdlinux authored and dvyukov committed Mar 20, 2019
    Signed-off-by: Andrew Donnellan <>
Commits on Mar 19, 2019
  1. pkg/report: fix matching for traps in kernel mode

    markjdb authored and dvyukov committed Mar 19, 2019
    We already have a pattern to match "Fatal trap N: ..." panics, but the
    format of our backtraces has changed since it was added.  Add another
    pattern which correctly matches a few instances generated by syzbot.
  2. dashboard/app: report bisection results to external reporting

    dvyukov committed Mar 19, 2019
    Update #501
  3. pkg/build: copy kernel.full to the objdir after a build

    markjdb authored and dvyukov committed Mar 18, 2019
    This helps ensure that we can create coverage reports on FreeBSD.
  4. sys/targets: use kernel.full instead of kernel.debug

    markjdb authored and dvyukov committed Mar 18, 2019
    The latter contains only debug symbols and is meant to be used with the
    kernel executable.  That is, the kernel executable contains a
    .gnu_debuglink pointer to kernel.debug.  kernel.full contains
    everything, including a copy of the kernel's text section, which we want
    when enumerating __sanitizer_cov_trace_pc() calls for a coverage report.
Commits on Mar 18, 2019
  1. dashboard/app: fix a typo in comment

    dvyukov committed Mar 18, 2019
  2. sys/linux: restrict SYSLOG_ACTION_CONSOLE_LEVEL

    dvyukov committed Mar 18, 2019
    Fuzzer must not mess with console, turn it on/off, change log level, etc.
    Otherwise it turns off kernel output on console.
  3. dashboard/config: add freebsd service script

    dvyukov committed Mar 18, 2019
  4. pkg/osutil: kill subprocesses more reliably

    dvyukov committed Mar 18, 2019
    In some cases we start scp, which starts ssh,
    then kill scp but the ssh subprocess is not killed.
    As the result cmd.Wait hangs waiting for EOF on the stdout/stderr,
    which are still kept alive by ssh subprocess. But ssh just hangs forever.
    Create a process group for each command and kill whole process group.
    Hopefully this will help.
  5. dashboard/app: make email tests more interesting

    dvyukov committed Mar 18, 2019
    1. Mail bugs for second and third reportings to different emails
       so that it's possible to distinguish where they are actually mailed.
    2. Add bisection test where we skip bug in the second reporting.
       Bisection results should go straigth to third as well.
  6. sys/openbsd: add chflags descriptions

    mptre authored and dvyukov committed Mar 18, 2019
  7. sys/linux: Add rfkill description for Linux

    a13xp0p0v authored and dvyukov committed Mar 17, 2019
  8. Update

    skrtbhtngr authored and dvyukov committed Mar 6, 2019
    change chown to chmod
Commits on Mar 17, 2019
  1. pkg/vcs: fix too long line

    dvyukov committed Mar 17, 2019
    Also restructure because we point to this file from syzbot docs.
  2. dashboard/app, syz-ci: bisection support

    dvyukov committed Mar 2, 2019
    This adds bulk of support for bisection to dashboard/app and syz-ci:
    - APIs to send bisection jobs and accept results
    - syz-ci logic to execute bisection jobs
    - formatting of emails with results
    - showing of results on dashboard
    Some difficulties we have to overcome:
    - since linux is frequently build/boot broken, lots of bisections are inconclusive,
      need to present such results too
    - git bisect is poorly suitable for automation, have to resort to output parsing (is output stable?)
    - git bisect turns out to fail (exit with non-0 status) when bisection is inconclusive
      (multiple potential cause commits)
    - older syzkaller revisions can't be built with newer (broken) kernel header, e.g.:
      ebtables.h:197:19: error: invalid conversion from ‘void*’ to ‘ebt_entry_target*’
    - newer compilers produce more warnings and break old syzkaller builds, e.g.:
      kvm.S.h:6:12: error: ‘kvm_asm64_vm86’ defined but not used [-Werror=unused-const-variable=]
    - figuring relevant emails to CC from a commit is non-trivial:
      besides commit author, there can be some emails in commit tags, or not,
      which tags to use is an interesting question (some may include irrelevant emails)
      we can also run on the commit, but this can produce too wide
      list if commit touches lots of files, it can also produce too small list,
      and then we need to resort to blame
    - for inconclusive bisection we probably don't need to include emails referenced
      in the commits (there can be too many of these commits)
    - need to be careful to exclude own syzbot email from commit CC list,
      now syzbot emails are referenced in some commits (Reported-by/Tested-by/etc)
      (can cause some kind of infinite recursion)
    - lots of commits reference stable mailing list,
      we should not include it in CC because it's referenced for backports rather then bug reports
    - since we add new Bug entity fields which we use in queries,
      whole datastore need to be upgrades to add the new field to index
    - we must not discard the crash that was used for bisection
      (treat it as a reported crash)
    - bisection results need 2 forms of reports:
      one when we add bisection results to already reported bug
      another when we report a bug first time with bisection results
    - when reporting a bug with bisection results we need to use the crash
      that was used for bisection
    - some fraction of bisections will probably fail with various errors
      and we will need some mechanism to retry bisection after the root cause is resolved
      this is not implemented yet
    - linux-next is problematic for 2 reasons:
      fix bisection can't possibly run on linux-next as commits are not reachable from HEAD
      lots of commits are missing in linux-next (even in linux-next-history)
      e.g. we have some c63e9e91a254a52 which is now missing in linux-next/linux-next-history
    - older kernels can't be build with fresh gcc/binutils/perl/make/glibc
      for now we have to stop at v3.9 (this only requires switching gcc several times along the way)
    - kernels past v4.11 do not build with gcc 7 and 8 (undefined reference to `____ilog2_NaN')
    - v4.1 and back have only compiler-gcc5.h
    - v3.17 and back have only compiler-gcc4.h
    - v3.6 and back do not have make olddefconfig
    - compat socket calls can't be bisected past "x86/entry/syscalls: Wire up 32-bit
      direct socket calls" (v4.10) because of
    - v2.6.28 and below does not work with modern make:
      *** mixed implicit and normal rules: deprecated syntax
    - v3.8 build fails:
      Can't use 'defined(@array)' (Maybe you should just omit the defined()?) at kernel/ line 373.
      kernel/Makefile:134: recipe for target 'kernel/timeconst.h' failed
    - make 3.81 works for v2.6.28.
      3.81 almost works with current HEAD, you need to run make twice because first run spuriously fails with:
    - v2.6.28 with gcc-4.9.4 broken with:
      include/linux/kvm.h:240:9: error: duplicate member ‘padding’
    - but even defconfig fails:
      VDSO    arch/x86/vdso/
      gcc: error: elf_x86_64: No such file or directory
      gcc: error: unrecognized command line option ‘-m’
      It seems that we also need old binutils.
    - for v3.8 and below we need perl-5.14.4.
      Unfortunately this or any manually built perl doesn't work for later kernels:
      Can't locate in @inc
    - kernels starting from 4.14 and older are boot broken:
    - kernels older than 4.12 are broken during netdev setup
      (fixed by commit 675c8da049fd6556eb2d6cdd745fe812752f07a8)
    Update #501
  3. vm/qemu: detect boot errors faster

    dvyukov committed Mar 14, 2019
    Currently we try to ssh into the machine for 10 minutes
    even if it crashed right away. Make qemu exit on kernel panic
    and stop ssh'ing when qemu exits.
    Handling bad kernels fast is actually important for bisection.
    Update #501
  4. pkg/instance: fix boot error detection

    dvyukov committed Mar 16, 2019
    Currently we truncate output up to rep.EndPos after unexpected reboot.
    But report sets EndPos to the _last_ report in output,
    so if there are any other errors they are all skipped after
    truncation to EndPos. Truncate just one line instead.
  5. syz-fuzzer: communicate image testing errors to caller

    dvyukov committed Mar 16, 2019
    Prefix image testing errors with BUG: so that they are detected
    by whoever is analyzing the output. Otherwise currently they are
    dignosed as just "lost connection to test machine".
  6. dashboard/app: add handler for config migration

    dvyukov committed Mar 15, 2019
    updateBugReporting adds missing reporting stages to bugs in a single namespace.
    Use with care. There is no undo.
    This can be used to migrate datastore to a new config with more reporting stages.
    This functionality is intentionally not connected to any handler.
    Before invoking it is recommented to stop all connected instances just in case.
  7. syz-ci: fix formatting of error message

    dvyukov committed Mar 13, 2019
    PrependContext prepends the string rather than format.
  8. tools/syz-testbuild: add utility for kernel build testing

    dvyukov committed Mar 13, 2019
    syz-testbuild tests kernel build/boot on releases as it will be done by pkg/bisect.
    This allows to ensure that, for example, a change to kernel config won't break
    build/boot on older releases and consequently won't break bisection process.
    The binary needs to run under root because it creates images.
    The kernel checkout given to the tool will be cleaned and used for in-tree builds.
    Example invocation:
    sudo syz-testbuild -kernel_src $LINUX_CHECKOUT \
    	-config dashboard/config/upstream-kasan.config \
    	-sysctl dashboard/config/upstream.sysctl \
    	-cmdline dashboard/config/upstream-apparmor.cmdline \
    	-userspace $WHEEZY_USERSPACE \
    	-bisect_bin $BISECT_BIN
    A suitable wheezy userspace can be downloaded from:
    A set of binaries required for bisection (older compilers) can be downloaded from:
  9. pkg/vcs: reset repo more

    dvyukov committed Mar 13, 2019
    If we cherry-pick some fixes during bisection
    we need to "git reset" repo before "git bisect reset".
    Otherwise it will fail. Reset repo in more points.
    Update #501
  10. pkg/vcs: try to work around massive linux kernel build/boot breakages

    dvyukov committed Mar 12, 2019
    Linux kernel is frequently build/boot broken. Even on release tags.
    Everything past 4.15 is broken with our config for multiple reasons.
    This makes bisection process almost infeasible.
    Try to work around some breakages by disabling configs are we go back in time.
    Update #501
  11. vm/qemu: s/Boot/boot/

    dvyukov committed Mar 12, 2019
    Make Boot lower-case to not confuse readers that it's somehow exported.
    It is not. Pure implementation detail.
  12. pkg/bisect: various improvements

    dvyukov committed Mar 12, 2019
    A bunch of improvements after more wide bisection testing.
    Improve logging.
    Support returning several commits for inconclusive bisection.
    Return Report with the final crash.
    Remove code that was moved to pkg/vcs.
    Update #501
  13. syz-ci: add flag that allows to not start managers

    dvyukov committed Mar 12, 2019
    For bisection testing.
    Update #501
You can’t perform that action at this time.