xt/examples-compilation.t is too memory-hungry and there are issues if we change EVAL to anything else

[taboege]

The problem

Running the full xt/examples-compilation.t test brings my laptop to its knees: the memory used just keeps growing, so that I have to run it twice with half the tests each. They are over 5000 tests currently, but loading all those into memory is not the problem. The memory usage keeps rising to critical levels while those tests are executed.

Suggestions

Look for memory leaks. It might also be worthwhile to do one subtest per file and avoid loading all the tests into memory at the beginning.

[JJ]

That might make it even slower... Although there are known issues with this like this one rakudo/rakudo#2324 (which was solved, for no reason), this is not really intended to be run over all examples, but only over the ones that have been modified. Other than that, it's slow enough as it is, making it run one by one would be terribly slow...

[JJ]

The problem is that tests might be written when there were only a few examples. Now there are some files that have hundreds of them. And examples are extracted, added to an array, and then tested...
I'll try and see the way of not keeping them in an array but testing them on the fly...

[JJ]

Opening it back again because of #2782. It might have something to do with this, since it happens only if the whole set is run.

[taboege]

Cool, thanks!

[coke]

I'm sad to see this go. We spent a lot of effort to speed up the runtime of tests, and I would have liked to see a ticket get opened in rakudo to track the memory issue.

There are, btw, many ways to run the the tests against just a subset of files, which avoids the cost of running the file against the whole suite of docs. (util/update-and-test, the TEST_FILES env var)

[JJ]

@coke It's not gone for any meaning of the word. It's just working in a different way. Why do you feel it's gone?

[JJ]

Ah, OK, the speed up is gone. We're trading memory for speed. It's not a big deal as long as you don't run the whole length of tests.
The memory issue is relatively simple, I guess. You're creating lots of stuff in a block with lexical scope and it's simply not been cleaned up. I don't think it's a bug per se, But if you feel it should be solved in some other way, please reopen and we can try and find ways to fix that.

[JJ]

@coke I see this 349ecc6 I don't think speed is such an issue nowadays but, again, if you feel it is, please reopen and let's chat.

[JJ]

(I just clicked on the wrong button, so let's say do close it again if it's not an issue)

[JJ]

As @taboege pointed out, it's a flapper in the sense that it does not seem to error if we run that on the single file, it fails consistently if it's done on a group of files.