-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cras fuzzer #880
Add cras fuzzer #880
Conversation
cras is the audio server that runs on ChromeOS. The first cras fuzzer tests the client's message interface. Signed-off-by: Dylan Reid <dgreid@chromium.org>
Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). 📝 Please visit https://cla.developers.google.com/ to sign. Once you've signed, please reply here (e.g.
|
@dgreid you don't provide any seed corpus, do you? |
On Mon, Oct 9, 2017 at 6:29 PM, Kostya Serebryany ***@***.***> wrote:
@dgreid <https://github.com/dgreid> you don't provide any seed corpus, do
you?
I didn't. However, I do have one checked in with the source code.
I'll modify build.sh to pull it.
Thanks for the reminder,
Dylan
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64jLl3j9bhR7xyxwFsQOefK1s-NDKks5sqshjgaJpZM4PxCDq>
.
|
@dgreid does this fuzz target create a thread per input? This is not a huge problem and most of the things will work, but
(asan has a hard limit on the total number of threads ever created. You may consider it a hard-to-fix asan bug) b) this may cost you some efficiency and effectiveness. Does the logic of fuzzing this code require more than one thread? |
On Tue, Oct 10, 2017 at 12:17 PM, Kostya Serebryany ***@***.***> wrote:
@dgreid does this fuzz target create a thread per input?
Yes, see below.
This is not a huge problem and most of the things will work, but
a) you may occasionally see crashes like this:
==16==AddressSanitizer: Thread limit (4194304 threads) exceeded. Dying.
(asan has a hard limit on the total number of threads ever created. You may consider it a hard-to-fix asan bug)
b) this may cost you some efficiency and effectiveness.
Does the logic of fuzzing this code require more than one thread?
If not, and if it's a simple change, I recommend to make the fuzz target single-threaded.
The fuzzing code doesn't need multiple threads, however, the code that
is being fuzzed parses the input then pushes an internal IPC message
to its main control thread.
Is there any way to re-use that thread across inputs? If the server
instance could be created once, re-using it would both limit the
number of threads created and provide slightly better coverage.
Thanks,
Dylan
…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Does this help? http://llvm.org/docs/LibFuzzer.html#startup-initialization
Why? |
Because it will be way faster
That's a clear win.
and leaving the objects initialized might catch other bugs.
That's both good and bad.
Good: this way we may find bugs where one input corrupts the global state and the other input crashes the process because the state was already broken.
Bad: such bugs won't be reproducible by running a single input.
My favorite example: https://sourceware.org/bugzilla/show_bug.cgi?id=18043#c19
|
The target is not working on oss-fuzz :( This is what I see in every log file:
It does work locally inside the docker. |
Looks like shm_open is failing. Is that something restricted? If so, I can
refactor that bit to inject a buffer instead. Any ram scratch space will do
for the purposes of this environment.
…On Tue, Oct 10, 2017 at 6:57 PM, Kostya Serebryany ***@***.*** > wrote:
The target is not working on oss-fuzz :(
This is what I see in every log file:
Bot: oss-fuzz-linux-zone1-worker-cras-5554
Time ran: 0.047016
INFO: Seed: 388265914
INFO: Loaded 1 modules (6735 guards): 6735 [0xa07980, 0xa0e2bc),
INFO: 0 files found in /new
INFO: 0 files found in /cras_rclient_message
INFO: A corpus is not provided, starting from an empty corpus
==1== ERROR: libFuzzer: fuzz target exited
#0 0x4ec213 in __sanitizer_print_stack_trace _asan_rtl_
#1 0x5dd884 in fuzzer::Fuzzer::ExitCallback() /src/libfuzzer/FuzzerLoop.cpp:212:5
#2 0x7fbc3dc82ff7 in __run_exit_handlers /build/glibc-9tT8Do/glibc-2.23/stdlib/exit.c:82
#3 0x7fbc3dc83044 in exit /build/glibc-9tT8Do/glibc-2.23/stdlib/exit.c:104
#4 0x531646 in cras_system_state_init /src/adhd/cras/src/server/cras_system_state.c:82:3
#5 0x51b1b3 in LLVMFuzzerTestOneInput /src/adhd/cras/src/fuzz/rclient_message.cc:19:3
It *does* work locally inside the docker.
Does cras_system_state_init do something interesting e.g. with the
network that might be banned by our sandbox?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64rXul1lmi7PnxrBkuf0Z-Z0Lap21ks5srCCSgaJpZM4PxCDq>
.
|
Yea, I'd expect shm is banned by the sandbox (@oliverchang should know better). |
Yep, /dev/shm is not available right now for the time being. |
Thanks guys. I uploaded a change to factor the shm init out. It will
hopefull land later today.
…On Tue, Oct 10, 2017 at 10:18 PM, Oliver Chang ***@***.***> wrote:
Yep, /dev/shm is not available right now for the time being.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64uXhl-5oXmqX_AeSyaCbxp3-CBzAks5srE-NgaJpZM4PxCDq>
.
|
Did it? |
On Thu, Oct 12, 2017 at 2:24 PM, Kostya Serebryany ***@***.*** > wrote:
and later today.
Did it?
It's currently in our commit queue, which isn't working that quickly today.
It should land in the next hour or two unless the CQ flakes again.
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64j8WGcvoOu1VfX4kwsJYvRp2C6ALks5sroOWgaJpZM4PxCDq>
.
|
On Thu, Oct 12, 2017 at 2:26 PM, Dylan Reid ***@***.***> wrote:
On Thu, Oct 12, 2017 at 2:24 PM, Kostya Serebryany <
***@***.***> wrote:
> and later today.
>
> Did it?
>
It's currently in our commit queue, which isn't working that quickly
today. It should land in the next hour or two unless the CQ flakes again.
After a long day of git-on-borg issues, the shm refactoring is now landed.
Is there an easy way for me to test with a sandbox setup similar to the one
used for OSSFuzz?
Thanks,
Dylan
|
The logs still show immediate failure.
I think you need this: |
Setting up the exact sandbox environment is a little difficult, but I've added a script to simulate most of it. Do test this, try doing: $ python infra/helper.py pull_images
$ python infra/helper.py build_image cras
$ python infra/helper.py build_fuzzers cras
$ python infra/helper.py shell base-runner
(within base-runner ocker)
# run_minijail /out/cras/rclient_message Right now this is instantly exiting. I did some debugging and found that the exit is happening here:
which is this line: https://chromium.googlesource.com/chromiumos/third_party/adhd/+/e83ecfea08763ad5f6c9f1758432a7d56aaf5e47/cras/src/fuzz/rclient_message.cc#39 (Note that this particular sandbox is mostly redundant after recent refactoring, and will be removed in the near future after some blockers are addressed.) |
On Mon, Oct 16, 2017 at 4:17 PM, Oliver Chang ***@***.***> wrote:
Setting up the exact sandbox environment is a little difficult, but I've
added a script to simulate most of it.
Do test this, try doing:
$ python infra/helper.py pull_images
$ python infra/helper.py build_image cras
$ python infra/helper.py build_fuzzers cras
$ python infra/helper.py shell base-runner
(within base-runner ocker)# run_minijail /out/cras/rclient_message
Right now this is instantly exiting.
I did some debugging and found that the exit is happening here:
(gdb) bt
#0 __GI_exit (status=-1) at exit.c:104
#1 0x000000000051b31b in LLVMFuzzerInitialize () at /src/adhd/cras/src/fuzz/rclient_message.cc:39
#2 0x00000000005c867f in FuzzerDriver () at /src/libfuzzer/FuzzerDriver.cpp:531
#3 0x00000000005bf8f9 in main () at /src/libfuzzer/FuzzerMain.cpp:20
which is this line: https://chromium.googlesource.
com/chromiumos/third_party/adhd/+/e83ecfea08763ad5f6c9f1758432a7
d56aaf5e47/cras/src/fuzz/rclient_message.cc#39
Oh, that'll be an easy fix. I'll give the run_minijail command a try
tomorrow and make sure I've got that running clean. Then I'll circle back
here and update you.
Thanks for all the help!
… (Note that this particular sandbox is *mostly* redundant after recent
refactoring, and will be removed in the near future after some blockers are
addressed.)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64i-Xu3Dk7cYgxUaJ4_Asi767F27Pks5ss-QEgaJpZM4PxCDq>
.
|
On Mon, Oct 16, 2017 at 9:07 PM, Dylan Reid ***@***.***> wrote:
On Mon, Oct 16, 2017 at 4:17 PM, Oliver Chang ***@***.***>
wrote:
> Setting up the exact sandbox environment is a little difficult, but I've
> added a script to simulate most of it.
>
> Do test this, try doing:
>
> $ python infra/helper.py pull_images
> $ python infra/helper.py build_image cras
> $ python infra/helper.py build_fuzzers cras
> $ python infra/helper.py shell base-runner
>
> (within base-runner ocker)# run_minijail /out/cras/rclient_message
>
> Right now this is instantly exiting.
>
> I did some debugging and found that the exit is happening here:
>
> (gdb) bt
> #0 __GI_exit (status=-1) at exit.c:104
> #1 0x000000000051b31b in LLVMFuzzerInitialize () at /src/adhd/cras/src/fuzz/rclient_message.cc:39
> #2 0x00000000005c867f in FuzzerDriver () at /src/libfuzzer/FuzzerDriver.cpp:531
> #3 0x00000000005bf8f9 in main () at /src/libfuzzer/FuzzerMain.cpp:20
>
> which is this line: https://chromium.googlesource.
> com/chromiumos/third_party/adhd/+/e83ecfea08763ad5f6c9f17584
> 32a7d56aaf5e47/cras/src/fuzz/rclient_message.cc#39
>
I've got a theoretical fix for this landed.
However I'm having trouble testing the above because the base-runner image
doesn't have run_minijail in step 4.
…
Oh, that'll be an easy fix. I'll give the run_minijail command a try
tomorrow and make sure I've got that running clean. Then I'll circle back
here and update you.
Thanks for all the help!
> (Note that this particular sandbox is *mostly* redundant after recent
> refactoring, and will be removed in the near future after some blockers are
> addressed.)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#880 (comment)>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AAG64i-Xu3Dk7cYgxUaJ4_Asi767F27Pks5ss-QEgaJpZM4PxCDq>
> .
>
|
Could you please do a git pull and run |
On Tue, Oct 17, 2017 at 1:58 PM, Oliver Chang ***@***.***> wrote:
Could you please do a git pull and run python infra/helper.py pull_images
?
Got it, yes that fixed it. With this and the latest change to the test
which landed this morning, the fuzzer has been running locally for about 15
minutes.
Thanks again for all the help.
… —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64npDPrPaI5T5ykNhwVv3PwN3h0S_ks5stRTggaJpZM4PxCDq>
.
|
@dgreid we had two full days of fuzzing, please check the coverage dashboard now (via oss-fuzz.com) |
Thanks, that looks like what I expected. I don't have permission to check
the logs on the crashes it found with either my google.com or gmail.com
account. Is there a way to allow that?
Thanks,
Dylan
…On Fri, Oct 20, 2017 at 11:07 PM, Kostya Serebryany < ***@***.***> wrote:
@dgreid <https://github.com/dgreid> we had two full days of fuzzing,
please check the coverage dashboard now (via oss-fuzz.com)
to see if the coverage is sane.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64tzVSa7oKIhh3KQYqujGo1vpWXjSks5suYoFgaJpZM4PxCDq>
.
|
your chromium account has the access. |
Thanks Kostya,
Everything is working great.
Dylan
…On Sun, Oct 22, 2017 at 4:33 AM, Kostya Serebryany ***@***.*** > wrote:
your chromium account has the access.
Add any other e-mail you want here: https://github.com/google/oss-
fuzz/blob/master/projects/cras/project.yaml
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#880 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAG64kjvZ5Izl5WEG2dnnA3qTrnpRbGjks5suql9gaJpZM4PxCDq>
.
|
cras is the audio server that runs on ChromeOS. The first cras fuzzer tests the client's message interface. Signed-off-by: Dylan Reid <dgreid@chromium.org>
cras is the audio server that runs on ChromeOS.
The first cras fuzzer tests the client's message interface.
Signed-off-by: Dylan Reid dgreid@chromium.org