Find a better way to build libraries

[partouf]

Currently, building trunk libraries for compilers can take a day or longer.

It might be a better idea to apply a priority system based on compiler and library usage to spread out builds and not spend time on building things that nobody uses anyway.

Would require a registry of some sort where we store statistics on compilers and libraries.

Would perhaps require storing/loading data about when a library has been last been built.

[partouf]

A large part of the ec2 instances monthly costs are because of the library builder at the moment (because it's finally working correctly)

[partouf]

tonight first time just popular compiler builds, we'll see how it goes

[partouf]

builder only spent 2ish hours tonight instead of 5, so it's looking ok so far

[partouf]

(of course it's still a lot of time for technically not doing anything 0 packages built OK, 342 skipped, and 0 failed build)

[partouf]

No real significant change.

Not a lot of compiles can be checked per second. Possible because of calling python everytime we need a conan hash, partly because of all the individual network calls to the conan server.

There's an even more significant problem going on which should be addressed first:
Builds that seem to succeed, but do not produce any binaries.

Example:

Apr 29 05:46:22 [librarybuilder](https://my.papertrailapp.com/systems/librarybuilder/events?focus=1588813930464821250&selected=1588813930464821250) [librarycpp.build](https://my.papertrailapp.com/events?q=program%3Alibrarycpp.build&focus=1588813930464821250&selected=1588813930464821250) 2023-04-29 03:46:22,260 lib.installable.installable INFO     Building unifex trunk for [arm64g730,,/opt/compiler-explorer/arm64/gcc-7.3.0/aarch64-unknown-linux-gnu,Linux,Debug,,,,]
Apr 29 05:46:26 [librarybuilder](https://my.papertrailapp.com/systems/librarybuilder/events?focus=1588813949280468992&selected=1588813949280468992) [librarycpp.build](https://my.papertrailapp.com/events?q=program%3Alibrarycpp.build&focus=1588813949280468992&selected=1588813949280468992) 2023-04-29 03:46:26,745 lib.installable.installable INFO     Build succeeded in /tmp/staging/cffe5c7e-5bce-4581-93d7-1a51581bc1be/arm64g730_5db678e5bbff24bd42f5c5c824e46b1772768962f4654e942a2d595f072eba0e
Apr 29 05:46:26 [librarybuilder](https://my.papertrailapp.com/systems/librarybuilder/events?focus=1588813949280468995&selected=1588813949280468995) [librarycpp.build](https://my.papertrailapp.com/events?q=program%3Alibrarycpp.build&focus=1588813949280468995&selected=1588813949280468995) 2023-04-29 03:46:26,746 lib.installable.installable INFO     No binaries found to export

4 seconds wasted for nothing. Should this not be marked as failed, so that it doesn't retry the next day?

[partouf]

Popular compiler building works Ok.
But we still spend a lot of time on building (trunk) libraries that are barely used, that should also be limited. We have the usage stats for it, so it should be doable.

[partouf]

I want to remind myself of the pressing need to do something with the available library statistics.

Example: Currently we spend at least 1 minute per compiler on building mfem trunk which is updated fairly often, but there are only 4 usages per month for this library. This is just a waste of resources.

Of course it "Would Be Nice" to inform users about then 1. which commit of headers and 2. which commit is actually built, but the first one is not saved anywhere except for the .git info, I don't know how easy it is to add this information to the UI.

[partouf]

Example: Currently we spend at least 1 minute per compiler on building mfem trunk which is updated fairly often, but there are only 4 usages per month for this library. This is just a waste of resources.

As a short term alternative we can also consider deprecating the trunk version and introducing a numbered version instead.