Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: -run= not working with fuzz subtest names #48149

Open
rsc opened this issue Sep 2, 2021 · 1 comment
Open

cmd/go: -run= not working with fuzz subtest names #48149

rsc opened this issue Sep 2, 2021 · 1 comment

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Sep 2, 2021

% rm -r testdata/corpus && gotip test -run=XXX -fuzz=Fuzz'$'
found a crash, minimizing...
gathering baseline coverage, elapsed: 1.0s, workers: 16, left: 1948
--- FAIL: Fuzz (1.48s)
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xabfYo -->\n\n ^so --6|Zu   <!-- foo -->\n"
                out: "<p>\x02 <!-- \xab\xabfYo --></p>\n<p>^so --6|Zu   <!-- foo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xabfYo --&gt;</p>\n<p>^so --6|Zu   <!-- foo --></p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n\n ^so --6|Zu"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n<p>^so --6|Zu</p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n<p>^so --6|Zu</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n\n ^s"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n<p>^s</p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n<p>^s</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!-- \xab\xb1fYo -->"
                out: "<p>\x02<!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xab\xb1fYo -->"
                out: "<p>\x02<!--\xab\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!--\xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1fYo -->"
                out: "<p>\x02<!--\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!--\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1Yo -->"
                out: "<p>\x02<!--\xb1Yo --></p>\n"
                gout: "<p>\x02&lt;!--\xb1Yo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1o -->"
                out: "<p>\x02<!--\xb1o --></p>\n"
                gout: "<p>\x02&lt;!--\xb1o --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1 -->"
                out: "<p>\x02<!--\xb1 --></p>\n"
                gout: "<p>\x02&lt;!--\xb1 --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1-->"
                out: "<p>\x02<!--\xb1--></p>\n"
                gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
    
    Crash written to testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83
    To re-run:
    go test rsc.io/markdown -run=Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83
FAIL
exit status 1
FAIL	rsc.io/markdown	1.638s

Bug 1t: The re-run command should not include the package name if I didn't include the package name. That is, if I ran go test and not go test <args>, then it should say go test -run... not go test <pkgname> -run.... It's jarring to see the package name appear when I wasn't using it explicitly.

Bug 2: The -run= argument does not match the test failure:

r% gotip test -run=Fuzz
--- FAIL: Fuzz (0.06s)
    --- FAIL: Fuzz/testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 (0.00s)
        fuzz_test.go:60: in: "\x02<!--\xb1-->"
            out: "<p>\x02<!--\xb1--></p>\n"
            gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
FAIL
exit status 1
FAIL	rsc.io/markdown	0.189s

Note that the failure has an extra 'testdata/corpus' in its name. I assume that was meant to be left out of the name printed by the test.

Bug 3: The -run argument does not work to select subtests of fuzz targets:

r% gotip test -run=Fuzz/somethingelse
--- FAIL: Fuzz (0.06s)
    --- FAIL: Fuzz/testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 (0.00s)
        fuzz_test.go:60: in: "\x02<!--\xb1-->"
            out: "<p>\x02<!--\xb1--></p>\n"
            gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
FAIL
exit status 1
FAIL	rsc.io/markdown	0.180s

Saying Fuzz/somethingelse should have matched somethingelse against testdata and not matched. Or the implication of the suggested command is that it should have matched somethingelse against c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 and still not matched. Either way, the test should not have run, but it did.

/cc @golang/fuzzing

@rsc rsc added this to the Go1.18 milestone Sep 2, 2021
@jayconrod jayconrod added the fuzz label Sep 9, 2021
@katiehockman
Copy link
Member

@katiehockman katiehockman commented Sep 14, 2021

Bug 2: The -run= argument does not match the test failure:
...
Note that the failure has an extra 'testdata/corpus' in its name. I assume that was meant to be left out of the name printed by the test.

@rsc Can you clarify what you mean here? I don't see the bug. Are you just noting that it says 'testdata/corpus' (now 'testdata/fuzz') in the fail line? That's because it corresponds to the corpus file written in 'testdata/corpus'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants