Not all available cores are used by taskmanager

[glamperd]

I'm running snarkjs on an EC2 machine, c7a.48xlarge. The machine has 192 cores, but the computation uses only 64 cores.

I invoke a snarkjs powersoftau prepare phase2 command in the CLI. It's a large (2^28) computation, so it takes many hours. Once it is well under way, looking at top, the process shows %CPU around 6400. I understand this to mean that 64 threads are running.

I see the concurrency factor is determined from os.cpus(). It's odd, but this command:
node -e "console.log(require('os').cpus().length)" returns 192, not 64, so this problem is deeper than it first appears.

Node.js doco recommends using os.availableParallelism over os.cpus(), but the result is the same, at least when run as a single node command:
node -e "console.log(require('os').availableParallelism()) returns 192.

I believe the node recommendation is to address cases such as running in docker with a restricted CPU allowance. That wouldn't apply to this issue, but might be worth considering switching to the node.js recommendation.

[glamperd]

I see ffjavascript forces the concurrency to a max of 64, "for memory reasons". I'm wondering if we can push this a bit further. I'll experiment.

[glamperd]

I ran a 'prepare' command on an instance with 192 cores, and the 64-core limit disabled. Had no issues. I think, ideally, the limit would be removable with a command line flag. However, I've opted to run with a 64-core machine. The process is partly CPU -intensive and partly read/write-intensive, in about equal parts. It takes a little longer with a 64 core machine compared to a 192-core instance, but if there's no deadline it's more cost effective.