RFC: Limit build parallelism by available memory too, add tunables (#804)

[pmatilai]

This PR is to address two different but related issues:

1. On 32bit systems, available virtual memory is in practise very limited and easy to exhaust on systems with many CPUs. (RhBug:1729382)
2. There are systems with lot of CPUs (virtual or otherwise) but relatively limited memory, where just looking at available CPUs causes severe trashing. (RhBug:1118734)

This effectively caps the number of threads on 32bit to just four, but tunable by changing the thread size estimate. Number of processes is capped to gigabytes of memory or number of CPU's, whichever is smaller, and tunable by changing the process size estimate.

The series adds a new macro primitive for getting system memory information and builds bunch of heuristics on top of those. There's a lot of subtleties involved and no doubt some of them I've gotten wrong here, so this PR is not intended for immediate merging, but more as a basis for discussion and other feedback.

What bothers me personally here is that it adds quite a bit of brittle heuristics that we never needed before, heuristics that will inevitably go wrong. The 32bit thread issue could be handled by just slapping a hard limit, with just a couple of lines of code. OTOH, the many cpus but little memory -case is legit and solving does require heuristics no matter what. And since heuristics will go wrong sooner or later, there needs to be tunables, which is why so much of this is done in macro level despite being somewhat painful.

[sharkcz]

for the record, there are already packages like ceph or firefox that do their own heuristics, and I'm not sure it's always correct, so having a system way is definitely step in the right direction

[pavlinamv]

I tested this PR on my laptop.

• The expansion of macro %{getmem:virt}" finishes with an error. It is because getmem returns 0. This is caused by the fact that function getmem_virt returns SIZE_MAX. After dividing by 1024^2 it is still to high to fit into return_type of getmem (unsigned int).

• If there is a wrong parameter after '%{getmem:' , then no warning or error is emitted

• Why you use a new type of built-in macro synatx %{getmem:} instead of something close to the existing built-in macros like %getmem_avail?

[pmatilai]

* The expansion of macro %{getmem:virt}" finishes with an error. It is because getmem returns 0. This is caused by the fact that function getmem_virt returns SIZE_MAX. After dividing by 1024^2 it is still to high to fit into return_type of getmem (unsigned int).

Right. I probably only tested this in 32bit context where its more relevant. There might be more similar issues lurking (it's only an RFC for a reason)

* If there is a wrong parameter after '%{getmem:' , then no warning or error is emitted

Yeah, known. Missing/wrong arguments to built-in macros are wildly inconsistent in how they behave, some emit errors, some just fail silently etc.

* Why you use a new type of built-in macro synatx %{getmem:} instead of something close to the existing built-in macros like %getmem_avail?

It's not a new syntax, it's a built-in with an argument like several others. I don't see a point of adding multiple macros that all return different aspects of the same thing, that's what a parameter is for. Purely a matter of style of course.

[pavlinamv]

Yeah, known. Missing/wrong arguments to built-in macros are wildly inconsistent in how they behave, some emit errors, some just fail silently etc.

Evidently better way is to emit an error.

[pmatilai]

Evidently better way is to emit an error.

No kidding? 😂

[pmatilai]

This is broken and not going to go anywhere soon. I'll resubmit once fixed.