Measuring time and memory usage with --cg option

[kylekim2123]

I’m building an algorithm scoring system using Isolate and have some questions.

I am running users’ code in Isolate as a separate process and measuring the time and memory usage. For Java code, which runs in multiple threads, I use the –cg option for scoring.

I have two questions:

1. I use the cg-mem value as the memory measurement result, but it is often significantly less than max-rss. Since max-rss is the maximum memory usage of a single process and cg-mem is the sum of the memory usage of multiple processes, why is cg-mem less? Is it okay to grade based on cg-mem?

   time:0.016
   time-wall:0.021
   max-rss:9856
   csw-voluntary:5
   csw-forced:2
   cg-mem:3136
   exitcode:0
   

2. I understand that the time value when using the –cg option is the sum of the execution times of multiple processes. Is it better to use wall-time rather than time for grading?

Thanks. 🙇

[gollux]

Ad 1: The difference is likely due to different accounting of shared memory pages (e.g., libc or the JVM). They are counted towards RSS, but with control groups, they are accounted on the first cgroup that brings them to the memory (e.g., by loading the executable from the disk). You can therefore expect a little noise, maybe a little more with Java as uses more shared code. Probably the only cure is to create a copy of the shared code for each sandbox.

Ad 2: It is definitely better to use the sum of execution times. With wall time, you could easily get around the time limit by creating more threads. Well, unless the topic of the task you are scoring is efficient parallelization.

[kylekim2123]

@gollux Thank you very much. Your help has been very valuable. I will close the issue.