Local development: Uploading large number of input cases times out web ui for some time

[kcwu]

How to reproduce

1. run a server and some bots locally

2. add a job, activate the fuzzing job. So bots started to fuzz.

3. Upload a zip file as input testcases. The zip file contains thousands cases.

Then one bot picked up the zip and working on those cases.
According to its bot.log, the bot happily updates testcases continuously (about 5 per second).

2019-02-12 23:57:08,090 - run_bot - INFO - Updated testcase 5059196596781056 (bug -).

So far it looks good. But something went wrong:

1. About one minute later, this message appeared in server's console output continuously (several per second).

| INFO     2019-02-12 15:45:21,771 module.py:861] cron-service: "GET /_ah/stop HTTP/1.1" 503 -

I don't know this is harmless or not. The said bot still update testcases continously according its bot.log.

2. The web ui is broken.
   Any access to http://localhost:9000/ shows 503 - This request has timed out.
   After I accessed the above link, the server console started to output below message continuously (several per second) --- but I only access above link once.

| INFO     2019-02-12 15:50:55,732 module.py:861] default: "GET /_ah/stop HTTP/1.1" 503 -

So, the current situation is:

1. bot still updates test cases
2. lots of error message in server's conole
3. I cannot access web ui.

If I restart the server, it will keep silence for a while and then repeat above issue again (output error message....).

[kcwu]

After the bot finished update cases, the error messages stop and the web ui becomes functional again.

[inferno-chromium]

How big was testcase archive ? what is average size for testcase. Note that local development scenario is not recommended for large scale tests. Recommended to do production setup for these.

[kcwu]

all test cases are less than 30 bytes.

[kcwu]

The zip file is about 4mb.

[inferno-chromium]

How long was the web ui inaccessible for ? > 1min ? this shouldn't happen again as this step was more about unpacking testcases quickly and storing them in db. we can add a throttle, but unsure if it is worth it in unpack_task. can you try a time.sleep there and maybe try a PR.

[kcwu]

the web ui is inaccessible for more than 5 minutes.
BTW, I just realized the primary usage of "upload testcase" is for bad cases like crashes, not for seed.
So I think I used this feature incorrectly.

In other words, I feel uploading lots of cases is unusual and thus this issue is not important (given it's local only). So you are free to close this issue.
OTOH, revising doc or ui to indicate "bad testcase" might be a good idea. Anyway, up to you.

[inferno-chromium]

anyone can use it to upload thousands of crashes for analysis. so, we can see and make a simple throttle in unpack_task.

[inferno-chromium]

yes throttling is not a good idea, considering this works perfectly in prod environment and has been tried multiple times by other Googlers. For local environment, i dont see a usecase to upload thousands, maybe someone need 1 or <10-50.