Skip to content

testing: Fuzz corpus header should support CRLF line endings #52268

Description

@moorereason

What version of Go are you using (go version)?

$ go version
go version go1.18 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\XXX\AppData\Local\go-build
set GOENV=C:\Users\XXX\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\XXX\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\XXX\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\XXX\src\github.com\pelletier\go-toml\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\XXX\AppData\Local\Temp\go-build3190186843=/tmp/go-build -gno-record-gcc-switches

What did you do?

Checkout a Go project with a fuzzing corpus and begin fuzzing:

git clone https://github.com/pelletier/go-toml.git
cd ./go-toml
checkout fixes
go test -fuzz FuzzUnmarshal

What did you expect to see?

I expected the fuzzer to startup successfully and begin fuzzing.

What did you see instead?

--- FAIL: FuzzUnmarshal (0.04s)
    fuzz_test.go:22: "testdata\\fuzz\\FuzzUnmarshal\\0c08b924aa26af23d16b77029b4f5567a96af93402e0472fef7b837b37580d7c": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\249a017d8ffa0d88d41c594ada5399f764833f64050180cb39f106d12666853f": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\391ce67b866d58464d067c5981e99951fd499896453a2fe2779a6bfe7df11bf5": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\49ae83283c5cfbed874bcacbfb342de107b69c2a3d3c35c2d99f67345fe89946": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\69ea82e85bb4ad966ae86bb05eea3af69ed1a826c846f1bf2282f329b4d8fb36": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\746910fac6deb42e7e62dce60b333c81c39d844a122c9075894c023400e8fdbf": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\844d1171a3adc95e51627f0e9378bbaa193fe2d0284f4ba8ed19a5226a8cbc74": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\900c54b5071113ae2a5365b3156377ddd775b9d38add61629544903cb9c94b00": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\c87e8eb4e8fa40fd3c89d8ed16a049ce81592ded710965ee29a4e506eb5427b3": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d0e5d6fc3b865cfae5480c01f301b87c932722ae0c1d692deea413349ea028be": unmarshal: unknown encoding version: go test fuzz v1
        "testdata\\fuzz\\FuzzUnmarshal\\d2cecea5b8ee5f148419671cef0644178cdba1ee91d3f295be38da576e89ef40": unmarshal: unknown encoding version: go test fuzz v1
FAIL
exit status 1
FAIL    github.com/pelletier/go-toml/v2 0.455s

Root Cause

The issue is that the corpus file format must use LF line endings. Git was "helpfully" converting everything to CRLF by default on Windows. In order to use the fuzzer, the Git config must contain core.autocrlf=false and core.eol=lf.

Activity

  1. seankhliao commented on Apr 10, 2022

    @seankhliao
    Member

    I think this is the wrong place to fix things, testdata/ can contain arbitrary data and letting git mess with them can cause all sorts of issues (eg CRLFs in golden files). There are per repo settings available via .gitattributes, which can and should be set.

  2. moorereason commented on Apr 11, 2022

    @moorereason
    Author

    Also, if CRLF will not be supported, the corpus file format documentation should make that explicit.

  3. dmitshur commented on Apr 11, 2022

    @dmitshur
    Member

    CC @golang/fuzzing.

  4. added
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on Apr 11, 2022
  5. added this to the Backlog milestone on Apr 11, 2022
  6. dominikh commented on Apr 23, 2022

    @dominikh
    Member

    My two cents: Yes, letting git mess with testdata can be bad, but for the large majority of users, it isn't, and fuzzing is the first instance of it being bad. Seeing that the format could trivially support CRLF, and seeing how this issue will affect every multi-OS user of fuzzing, I think this should be fixed in Go. It's an easy fix that helps a lot of users and makes fuzz testdata work automatically, without having to learn about git mechanisms.

    It's also in line with the rest of the Go toolchain, which supports CRLF line endings well.

  7. gopherbot commented on Apr 27, 2022

    @gopherbot
    Contributor

    Change https://go.dev/cl/402074 mentions this issue: internal/fuzz: trim carriage return from version line

  8. katcipis commented on May 20, 2022

    @katcipis

    +1

    I missed this issue because I searched for issues with the "fuzz" tag, but I also ran into problems when testing things on multiple OS's on GHA. More details here.

  9. added
    fuzzIssues related to native fuzzing support
    on May 20, 2022
  10. added
    NeedsFixThe path to resolution is known, but the work has not been done.
    and removed
    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
    on May 21, 2022
  11. modified the milestones: Backlog, Go1.19 on May 21, 2022
  12. locked and limited conversation to collaborators on May 21, 2023
  13. added a commit that references this issue on Sep 19, 2024
    c3470ca
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    FrozenDueToAgeNeedsFixThe path to resolution is known, but the work has not been done.fuzzIssues related to native fuzzing support

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions