Stress Test

[jason-huh]

As running a contest, the critical errors related to "scalability" have arisen. It would be much better to help the users to find potential problems.

For example,

a. Receiving the hundreds of submissions, the judgehosts cannot handle the huge size of judging list due to the default value of the limitation in mysql.
b. Memory is not sufficient so that connections pool would suffer from starvation as the contest is getting hot.
c. Judgehosts keep the judgings in the local disk storage and the amount would easily go up to gigabytes. I had to resize the disk and reboot the system during the contest.

Unless we prepare the stress test, these kinds of critical errors which would have very bad and irreversible effects on the ongoing contest are really difficult to be found.

[eldering]

First, can I ask you what size of contest you were running: how many teams, how many submissions, how many judgehosts did you have?

a. Which specific mysql settings are you referring to? Note that already a couple of relevant ones are checked on the "config check" page in the admin webinterface.
b. Are you talking about physical memory? Of the domserver or judgehosts?
c. We should probably mention this more clearly in the manual. Also, in the current development branch we've already reduced disk usage by reusing the compiled binaries among all testcase runs.

What kind of stress test would you propose?

[jason-huh]

a. They were "max_error_counts" and "max_allowed_packet" in mysql. I had to modify the default value as of approximately 10-15 teams, 100-150 submissions, 2 judgehosts.
b. Physical Memory of domserver. I located the DBMS "WITH" the domserver. It was 1GB and I extended it to 2GB. As of hundred teams, I extended it to 4GB.

In my experience, the communication between server and hosts often became the bottleneck. It would be better to do stress-test by inserting mock submissions and judgings and checking whether DBMS can tolerate the size of those requests. The stress-test is literally stressful. Why don't we do that before the releases and give users just the figures in the documentation?

[eldering]

See https://github.com/ubergeek42/domjudge-gatling for a way to stress test a DOMjudge instance.
Closing this issue here, since we don't have diret plans to integrate this more closely into DOMjudge.