This is a potential fix for the intermittent failures here:
#2441 (comment)

I still can't reproduce them locally. However, I could see that the
GPU cache size in that test was getting dangerously close to the
maximum size of the texture supported by the GL implementation.

That would fit with the issues we're seeing on CI, where there
seems to be items not being drawn, and they are affected even
though the test in question doesn't draw any gradients.

This is not a very elegant fix - however it will hopefully fix
the issues we are seeing on CI for now. We can revisit this
later to consider a better fix.