SpaceTally logic should handle large images

[VincentBlondeau]

In our image we have 200 000 classes, and a GC of 5 seconds. So, we need 11 days to run the following code: SpaceTally printSpaceAnalysis

The main question is that why do we need the GC in the middle?

[MarcusDenker]

I think the idea is that the data would be less correct without it (even with doing a GC right before the instances that are just created for the report do change the result, but not that much).

I think that a correct version would need to analyse the image "from the outside" to do a report... e.g. just on a dead image or via some VM level code that stops everything while counting.

[bencoman]

some VM level code that stops everything while counting

IIUC, on Linux the image is stored as mmap'd segments, so maybe clone [1] with:

• CLONE_VM unset - so the parent image can continue while the child has a copy-on-write static image to process
• CLONE_FILES set - so that the child can communicate results back to the parent

Half an example here [2]

[1] https://linux.die.net/man/2/clone
[2] https://eli.thegreenplace.net/2018/launching-linux-threads-and-processes-with-clone/

clone(2): create child process - Linux man page

clone() creates a new process, in a manner similar to fork(2). It is actually a library function layered on top of the underlying clone() system call, ...

Launching Linux threads and processes with clone - Eli Bendersky's website

[VincentBlondeau]

Ps: I want to run it under Windows ;)

[eliotmiranda]

The Spur VM provides an allObjects primitive which means one can collect all the data in one primitive and then do the analysis on that one collection with no GC necessary. I’m sure a full space tally of a 1Gb image implemented above the allObjects primitive would take much less than 10 minutes.

[Ducasse]

Thanks for the pointer.

[VincentBlondeau]

Fixed in #6177 implemented following @eliotmiranda advice and works like a charm!