Performance updates and misc.

[sgoggins]

Description

• I think we should check the number of cores and use that information for setting parallel processes in the next release. My heuristic was, on a 64 CPU server, I set it to 50. This is working well. I think ~14 processes running in parallel less that those on the hardware is optimal. We have to check for "CPU", NOT "Cores" because the core checking does not work on OSX. So, we can go with ~14 less than the CPUS available for safety, or a default of 14. Whichever is greater.

We could also simply check the OS, and if its a linux flavor, evaluate cores.

[sgoggins]

There is also a patch for the commits table here, setting the affiliation data to NULL instead of a string "NULL". The updates on the string "NULL" are very expensive.

[IsaacMilarky]

This mostly looks good although I would change the repo scheduling to be more balanced now that we have a lot more processes to use and don't have to have as many messages in the queue.

[ABrain7710]

I agree with @IsaacMilarky comment. Maybe like 40 processes for core and 15 for secondary? This would about triple the number of processes each worker

[IsaacMilarky]

I would set max_repo to 30 for start_primary_collection and I would set max_repo to 30 for start_facade_collection. Otherwise LGTM :)

Made changes.

[IsaacMilarky]

I still don't think we should schedule more repos than processes we have available.

[sgoggins]

I went with 30 for primary and 20 for secondary @ABrain7710 because secondary takes so much longer and core is always waiting for it to finish.