Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: Fuzz corpus header should support CRLF line endings #52268

Closed
moorereason opened this issue Apr 10, 2022 · 6 comments
Closed

testing: Fuzz corpus header should support CRLF line endings #52268

moorereason opened this issue Apr 10, 2022 · 6 comments
Labels
fuzz NeedsFix
Milestone

Comments

@moorereason
Copy link

@moorereason moorereason commented Apr 10, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\XXX\AppData\Local\go-build
set GOENV=C:\Users\XXX\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\XXX\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\XXX\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\XXX\src\github.com\pelletier\go-toml\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\XXX\AppData\Local\Temp\go-build3190186843=/tmp/go-build -gno-record-gcc-switches

What did you do?

Checkout a Go project with a fuzzing corpus and begin fuzzing:

git clone https://github.com/pelletier/go-toml.git
cd ./go-toml
checkout fixes
go test -fuzz FuzzUnmarshal

What did you expect to see?

I expected the fuzzer to startup successfully and begin fuzzing.

What did you see instead?

--- FAIL: FuzzUnmarshal (0.04s)
    fuzz_test.go:22: "testdata\\fuzz\\FuzzUnmarshal\\0c08b924aa26af23d16b77029b4f5567a96af93402e0472fef7b837b37580d7c": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\249a017d8ffa0d88d41c594ada5399f764833f64050180cb39f106d12666853f": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\391ce67b866d58464d067c5981e99951fd499896453a2fe2779a6bfe7df11bf5": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\49ae83283c5cfbed874bcacbfb342de107b69c2a3d3c35c2d99f67345fe89946": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\69ea82e85bb4ad966ae86bb05eea3af69ed1a826c846f1bf2282f329b4d8fb36": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\746910fac6deb42e7e62dce60b333c81c39d844a122c9075894c023400e8fdbf": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\844d1171a3adc95e51627f0e9378bbaa193fe2d0284f4ba8ed19a5226a8cbc74": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\900c54b5071113ae2a5365b3156377ddd775b9d38add61629544903cb9c94b00": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\c87e8eb4e8fa40fd3c89d8ed16a049ce81592ded710965ee29a4e506eb5427b3": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d0e5d6fc3b865cfae5480c01f301b87c932722ae0c1d692deea413349ea028be": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d2cecea5b8ee5f148419671cef0644178cdba1ee91d3f295be38da576e89ef40": unmarshal: unknown encoding version: go test fuzz v1
FAIL
exit status 1
FAIL    github.com/pelletier/go-toml/v2 0.455s

Root Cause

The issue is that the corpus file format must use LF line endings. Git was "helpfully" converting everything to CRLF by default on Windows. In order to use the fuzzer, the Git config must contain core.autocrlf=false and core.eol=lf.

@seankhliao
Copy link
Contributor

@seankhliao seankhliao commented Apr 10, 2022

I think this is the wrong place to fix things, testdata/ can contain arbitrary data and letting git mess with them can cause all sorts of issues (eg CRLFs in golden files). There are per repo settings available via .gitattributes, which can and should be set.

@moorereason
Copy link
Author

@moorereason moorereason commented Apr 11, 2022

Also, if CRLF will not be supported, the corpus file format documentation should make that explicit.

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Apr 11, 2022

@dmitshur dmitshur added the NeedsInvestigation label Apr 11, 2022
@dmitshur dmitshur added this to the Backlog milestone Apr 11, 2022
@dominikh
Copy link
Member

@dominikh dominikh commented Apr 23, 2022

My two cents: Yes, letting git mess with testdata can be bad, but for the large majority of users, it isn't, and fuzzing is the first instance of it being bad. Seeing that the format could trivially support CRLF, and seeing how this issue will affect every multi-OS user of fuzzing, I think this should be fixed in Go. It's an easy fix that helps a lot of users and makes fuzz testdata work automatically, without having to learn about git mechanisms.

It's also in line with the rest of the Go toolchain, which supports CRLF line endings well.

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 27, 2022

Change https://go.dev/cl/402074 mentions this issue: internal/fuzz: trim carriage return from version line

@katcipis
Copy link

@katcipis katcipis commented May 20, 2022

+1

I missed this issue because I searched for issues with the "fuzz" tag, but I also ran into problems when testing things on multiple OS's on GHA. More details here.

@dmitshur dmitshur added the fuzz label May 20, 2022
@dmitshur dmitshur added NeedsFix and removed NeedsInvestigation labels May 21, 2022
@dmitshur dmitshur removed this from the Backlog milestone May 21, 2022
@dmitshur dmitshur added this to the Go1.19 milestone May 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fuzz NeedsFix
Projects
None yet
Development

No branches or pull requests

6 participants