Skip to content
Permalink
Browse files

all: switch to Go 1.12

Differences in code formatting between Go versions cause constant
problems for us (golang/go#25161).
Currently we support 1.9 and 1.10. Switch to newer 1.11 and 1.12.

Fixes #1013
  • Loading branch information...
dvyukov committed Mar 14, 2019
1 parent 1e82084 commit d23e90a7b44b62b0d3b116a6d30849ce66051529
@@ -2,8 +2,8 @@ language: go
dist: xenial

go:
- "1.9"
- "1.10"
- "1.11"
- "1.12"

before_install:
- echo $PATH
@@ -308,9 +308,8 @@ presubmit:
presubmit_parallel: test test_race arch check_links

test:
ifeq ("$(TRAVIS)$(shell go version | grep 1.9)", "true")
# Collect coverage report for codecov.io when running on travis (uploaded in .travis.yml).
# Note: Go 1.9 does not support -coverprofile when testing multiple packages.
ifeq ("$(TRAVIS)$(shell go version | grep 1.11)", "true")
# Collect coverage report for codecov.io when testing Go 1.12 on travis (uploaded in .travis.yml).
env CGO_ENABLED=1 $(GO) test -short -coverprofile=coverage.txt ./...
else
# Executor tests use cgo.
@@ -46,6 +46,33 @@ Unassigned issues from the [bug tracker](https://github.com/google/syzkaller/iss
If you want to work on something non-trivial, please briefly describe it on the [syzkaller@googlegroups.com](https://groups.google.com/forum/#!forum/syzkaller) mailing list first,
so that there is agreement on high level approach and no duplication of work between contributors.

## Go

`syzkaller` is written in [Go](https://golang.org), and a `Go 1.11` or `Go 1.12`
toolchain is required for build. The toolchain can be installed with:

```
go get golang.org/dl/go1.12
go1.12 download
# Default download path is here.
~/sdk/go1.12/bin/go version
export GOROOT=$HOME/sdk/go1.12
export PATH=$HOME/sdk/go1.12/bin:$PATH
```

Then get and build `syzkaller`:

``` bash
go get -u -d github.com/google/syzkaller/...
cd $HOME?/go/src/github.com/google/syzkaller/
make
```

Note: older versions of Go toolchain formatted code in a slightly
[different way](https://github.com/golang/go/issues/25161).
So if you are seeing unrelated code formatting diffs after running `make generate`
or `make format`, you may be using Go 1.10 or older. In such case update to Go 1.11+.

## How to create a pull request

- First, you need an own git fork of syzkaller repository. Nagivate to [github.com/google/syzkaller](https://github.com/google/syzkaller) and press `Fork` button in the top-right corner of the page. This will create `https://github.com/YOUR_GITHUB_USERNAME/syzkaller` repository.
@@ -64,20 +91,6 @@ This adds git origin `my-origin` with your repository and checks out new branch
- Change/add files as necessary.
- Commit changes locally. For this you need to run `git add` for all changed files, e.g. `git add sys/linux/sys.txt`. You can run `git status` to see what files were changed/created. When all files are added (`git status` shows no files in `Changes not staged for commit` section and no relevant files in `Untracked files` section), run `git commit` and enter commit description in your editor.
- Run tests locally (`make install_prerequisites` followed by `make presubmit`).
- *Important* If you've run `go fmt` and you're seeing the presubmit fail on
`check_diff`, then you may need to use an older version of go to format your
code. (Version 1.11 in particular introduced a breaking change, see
[here](https://github.com/golang/go/issues/25161) and
[here](https://github.com/golang/go/issues/26228) for details). A
simple way to do this is:

```
go get golang.org/dl/go1.10
go1.10 download
# Default download path is here.
~/sdk/go1.10/bin/go fmt [target files]
```

- Push the commit to your fork on github with `git push my-origin my-branch`.
- Nagivate to [github.com/google/syzkaller](https://github.com/google/syzkaller) and you should see green `Compare & pull request` button, press it. Then press `Create pull request`. Now your pull request should show up on [pull requests page](https://github.com/google/syzkaller/pulls).
- If you don't see `Create pull request` button for any reason, you can create pull request manually. For that nagivate to [pull requests page](https://github.com/google/syzkaller/pulls), press `New pull request`, then `compare across forks` and choose `google/syzkaller`/`master` as base and `YOUR_GITHUB_USERNAME/syzkaller`/`my-branch` as compare and press `Create pull request`.
@@ -4,10 +4,10 @@ This page describes how to execute existing syzkaller programs for the purpose
of bug reproduction. This way you can replay a single program or a whole
execution log with several programs.

1. Setup Go toolchain (if you don't yet have it, you need version 1.9 or higher):
Download latest Go distribution from (https://golang.org/dl/). Unpack it to `$HOME/go1.9`.
1. Setup Go toolchain (if you don't yet have it, you need version 1.11 or higher):
Download latest Go distribution from (https://golang.org/dl/). Unpack it to `$HOME/go1.11`.
``` bash
$ export GOROOT=$HOME/go1.9
$ export GOROOT=$HOME/go1.11
$ export GOPATH=$HOME/gopath
```

@@ -77,7 +77,7 @@ Then run FIDL compiler backend `fidlgen` with syzkaller generator, which compile

To run `syz-ci` locally for Fuchsia, you need:

- Go 1.10 toolchain (in `/go1.10` dir in the example below)
- Go 1.12 toolchain (in `/go1.12` dir in the example below)
- bootstrapped Fuchsia checkout (in `/bootstrap/fuchsia` dir in the example below)
- bootstrap `syz-ci` binary (in the current dir, build with `make ci`)
- `syz-ci` config similar to the one below (in `ci.cfg` file in the current dir)
@@ -87,7 +87,7 @@ To run `syz-ci` locally for Fuchsia, you need:
"name": "testci",
"http": ":50000",
"manager_port_start": 50001,
"goroot": "/go1.10",
"goroot": "/go1.12",
"syzkaller_repo": "https://github.com/google/syzkaller.git",
"managers": [
{
@@ -66,7 +66,7 @@ For some details on fuzzing the kernel on an Android device check out [this page

### Syzkaller

`syzkaller` is written in [Go](https://golang.org), a `Go 1.9+` toolchain is required for build.
`syzkaller` is written in [Go](https://golang.org), a `Go 1.11+` toolchain is required for build.

Go distribution can be downloaded from https://golang.org/dl/.
Unpack Go into a directory, say, `$HOME/goroot`.
@@ -1,11 +1,8 @@
# Setup: Linux or Mac OS host, Android device, arm64 kernel

Prerequisites:
- go1.8+ toolchain (can be downloaded from [here](https://golang.org/dl/))
- Android Serial Cable or [Suzy-Q](https://chromium.googlesource.com/chromiumos/platform/ec/+/master/docs/case_closed_debugging.md) device to capture console output is preferable but optional. syzkaller can work with normal USB cable as well, but that can be somewhat unreliable and turn lots of crashes into "lost connection to test machine" crashes with no additional info.

- Build syzkaller

In case you have old Android `/dev/ion` driver:

```sh
@@ -53,28 +53,8 @@ Host *
Before fuzzing, connect to the machine and keep the connection open so all scp
and ssh usage will reuse it.

## Go

Install Go 1.8.1:
``` bash
wget https://storage.googleapis.com/golang/go1.8.1.linux-amd64.tar.gz
tar -xf go1.8.1.linux-amd64.tar.gz
mv go goroot
export GOROOT=`pwd`/goroot
export PATH=$PATH:$GOROOT/bin
mkdir gopath
export GOPATH=`pwd`/gopath
```

## Syzkaller

Get and build syzkaller:
``` bash
go get -u -d github.com/google/syzkaller/...
cd gopath/src/github.com/google/syzkaller/
make
```

Use the following config:
```
{
@@ -15,37 +15,14 @@ These were tested on an NXP Pico-Pi-IMX7D following the instructions [here](http

If feasible, recompile and reinstall the Linux kernel with any debugging options available on your board.

## Install Go

Install Go as follows:
``` bash
wget https://storage.googleapis.com/golang/go1.9.2.linux-amd64.tar.gz
tar -xf go1.9.2.linux-amd64.tar.gz
export PATH=`pwd`/go/bin:$PATH
mkdir gopath
export GOPATH=`pwd`/gopath
```

## Build syzkaller code

### Initialize a working directory and set up environment variables

Create a working directory. Also make sure GOROOT, GOPATH are defined and exported as instructed earlier.

``` bash
go get -u -d github.com/google/syzkaller/...
cd gopath/src/github.com/google/syzkaller/
mkdir workdir
```

### Build syzkaller executables
## Build syzkaller executables

Run make.
```
make TARGETOS=linux TARGETARCH=arm
```

### Create a manager configuration file
## Create a manager configuration file

Create a manager config myboard.cfg, replacing the environment
variables `$GOPATH`, `$KERNEL` (path to kernel build dir for the ARM32 board), and `$DEVICES` (the device ID for your board as reported by adb devices) with their actual values. Change any other flags as needed for your ARM board.
@@ -196,29 +196,8 @@ To kill the running QEMU instance:
kill $(cat vm.pid)
```
## Go
Install Go 1.8.1:
``` bash
wget https://storage.googleapis.com/golang/go1.8.1.linux-amd64.tar.gz
tar -xf go1.8.1.linux-amd64.tar.gz
mv go goroot
export GOROOT=`pwd`/goroot
export PATH=$GOROOT/bin:$PATH
mkdir gopath
export GOPATH=`pwd`/gopath
```
## syzkaller
Get and build syzkaller:
``` bash
go get -u -d github.com/google/syzkaller/...
cd gopath/src/github.com/google/syzkaller/
mkdir workdir
make
```
Create a manager config like the following, replacing the environment
variables `$GOPATH`, `$KERNEL` and `$IMAGE` with their actual values.
```
@@ -243,6 +222,7 @@ variables `$GOPATH`, `$KERNEL` and `$IMAGE` with their actual values.
Run syzkaller manager:
``` bash
mkdir workdir
./bin/syz-manager -config=my.cfg
```
@@ -86,8 +86,8 @@ bash
return fmt.Errorf("failed to write image file: %v", err)
}
for src, dst := range map[string]string{
".config": "kernel.config",
"key": "key",
".config": "kernel.config",
"key": "key",
"obj/kern/akaros-kernel": "kernel",
"obj/kern/akaros-kernel-64b": "obj/akaros-kernel-64b",
} {
@@ -31,7 +31,7 @@ func (fu fuchsia) build(targetArch, vmType, kernelDir, outputDir, compiler, user
}
for src, dst := range map[string]string{
"out/" + arch + "/obj/build/images/fvm.blk": "image",
".ssh/pkey": "key",
".ssh/pkey": "key",
"out/build-zircon/kernel-" + arch + "-gcc/obj/kernel/zircon.elf": "obj/zircon.elf",
"out/build-zircon/multiboot.bin": "kernel",
"out/" + arch + "/fuchsia.zbi": "initrd",
@@ -33,12 +33,12 @@ func TestCheckRepoAddress(t *testing.T) {
"http://host.xz:123/path/to/repo.git/": true,
"https://chromium.googlesource.com/chromiumos/third_party/kernel": true,
"https://fuchsia.googlesource.com": true,
"": false,
"foobar": false,
"linux-next": false,
"foo://kernel.ubuntu.com/ubuntu/ubuntu-zesty.git": false,
"git://kernel/ubuntu.git": false,
"gitgit://kernel.ubuntu.com/ubuntu/ubuntu-zesty.git": false,
"": false,
"foobar": false,
"linux-next": false,
"foo://kernel.ubuntu.com/ubuntu/ubuntu-zesty.git": false,
"git://kernel/ubuntu.git": false,
"gitgit://kernel.ubuntu.com/ubuntu/ubuntu-zesty.git": false,
})
}

@@ -219,11 +219,11 @@ func TestEscapingPaths(t *testing.T) {
"file/../../file": true,
"../file": true,
"./file/../../file/file": true,
"": false,
".": false,
"file": false,
"./file": false,
"./file/..": false,
"": false,
".": false,
"file": false,
"./file": false,
"./file/..": false,
}
for path, want := range paths {
got := escapingFilename(path)

5 comments on commit d23e90a

@wdebruij

This comment has been minimized.

Copy link
Contributor

replied Jul 1, 2019

Before this commit and [1], docs/linux/setup_ubuntu-host_qemu-vm_x86-64-kernel.md and others detailed every step needed to create a hermetically built syzkaller environment.

This commit removes the information on how to build syzkaller, skipping straight to "create a manager config .. Run syzkaller manager". Which is odd if no previous step explains how to build syzkaller.

Unintentional? Please consider restoring that part, thanks.

[1] also removed the gcc build instructions in favor of pointing to pre-built gcc binaries. for those of us hesitant to run someone else's unsigned and untrusted binaries, please consider restoring the original info.

[1] commit d28f4ce ("Update setup_ubuntu-host_qemu-vm_x86-64-kernel.md")

@dvyukov

This comment has been minimized.

Copy link
Collaborator Author

replied Jul 1, 2019

The Go setup instructions were moved to docs/contributing.md
We can't have them duplicated across 2 dozens of specific instructions and keep up to date.

We now don't need a custom gcc. Building gcc is hard. Out instructions did not work everywhere and will break over time. We don't want to maintain general build instructions for all possible dependencies for all possible distributions and versions. Please refer elsewhere for building gcc if you want a custom one, but otherwise just use the distro-provided one.

@wdebruij

This comment has been minimized.

Copy link
Contributor

replied Jul 1, 2019

I see. Might be useful to add a short statement with a pointer to that doc just before the first use of syzkaller.

And also to clarify that no custom built GCC or CLANG (e.g., wrt flags) is needed, so can just use the distro provided if it matches the version. In other words, make more clear that the provided list is a courtesy, but there is no technical need to use specifically those binaries.

@dvyukov

This comment has been minimized.

Copy link
Collaborator Author

replied Jul 2, 2019

Please file an issue or better send a PR so that this is not lost.

@wdebruij

This comment has been minimized.

Copy link
Contributor

replied Jul 3, 2019

Done: #1270

Please sign in to comment.
You can’t perform that action at this time.