-
Notifications
You must be signed in to change notification settings - Fork 554
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python process pileup, needs reproduction steps. #150
Comments
It already found the heartbleed, see logs 2019-02-08 14:40:40,543 - run_bot - INFO - Process the crash group (file=fuzz-1, fuzzed_key=25002300-d91f-4b14-b894-6e2a5f4ed858, return code=1, crash time=4, crash type=Heap-buffer-overflow and there should be testcase on localhost:9000. can you paste the job definitions you have, maybe some processes are piling up. Did you modify any templates or anything ? Are you running AFL or libFuzzer ? Need more reproducer instructions. |
can you also start from a clean slate and see if you can reproduce. like try pkill -9 -f clusterfuzz and pkill -9 -f gcloud. |
I followed the steps exactly for heartbleed, should be the same as on the wiki. It says it found heartbleed right away - but I was checking the test cases in the UI and it was empty from 2pm - 12am. Did not change any templates. |
this could be related to your broken config as in #136. your ports seem to be not free, can you try a restart and see if you can still reproduce. |
also which platform and os version you are running. |
I'm doubtful I have those ports blocked, as I showed checking the ports right after clusterfuzz tried using them revealed them to be unused. I restarted anyway! I'm on Ubuntu 18.04 LTS. |
I've restarted and heartbleed indeed now shows up in the test cases. Should I do anything else? |
Metadata says the following:
Which could explain why I didn't see the test case for many hours. |
Something was up with clearing ports. Basically, run_bot could not send testcase to run_server since run_server wasn't running and failing on not being able to find the port. run_bot was finding the crash fine, but run_server didnt get it. |
I'm also running into this, but with less impact. I did a fresh install on an Ubuntu 18.04 LTS VM. Followed the pre-requisites: https://google.github.io/clusterfuzz/getting-started/prerequisites/ I then ran the local server, went to the web UI, and created a bot instance a few minutes later in a separate tab. I see the port error in my logs, though interestingly enough things work (I see the test case in the testcases UI). My full server logs are here: https://pastebin.com/JKya8WNB - bot logs are here: https://pastebin.com/YN6vCRBZ Hope this helps! |
@mhlakhani - this should get fixed as part of #197. these never happen in production since we use appengine cron. locally, we create some threads to run every 60 secs. we can make them more reliable. @oliverchang as fyi. |
Terrible issue title but we'll fix it to something better (or close it) with more investigation.
I left clusterfuzz running overnight trying to find heartbleed and the fuzzer seems to have hung according to the bot logs:
(full log)
This has resulted in quite a few Python processes from clusterfuzz maxing out the CPU from the system and the clusterfuzz server from what it looks like trying to stop somewhat frantically: https://paste.ubuntu.com/p/b8JW2sX44w/
The end result of this is that the web interface is unresponsive and all calls time out with a 503.
The text was updated successfully, but these errors were encountered: