Fix CPU count detection in test runner on OS X

[JukkaL]

It seems to default to 2 CPUs on OS X which can waste resources.

[onyb]

This is probably because mypy/waiter.py is using os.sched_getaffinity, which is not supported on all non-glibc platforms, including OS X.

Solution 1: Use multiprocessing.cpu_count(), but I am not sure if this will give us the real number of available processors.

Solution 2: Get kernel parameters by calling sysctl using subprocess. Relevant parameters on a Macbook Pro with multiprocessing.cpu_count() as 4 are as follows. Can we use this?

hw.ncpu = 4
hw.activecpu = 4
hw.physicalcpu = 2
hw.physicalcpu_max = 2
hw.logicalcpu = 4
hw.logicalcpu_max = 4

Looking forward to make my first contribution to mypy. :)

[ddfisher]

From a quick test on my Macbook Pro, multiprocessing.cpu_count() seems to return the number of virtual cores. Using the number of virtual cores seems reasonable to me.

I'd prefer to avoid shelling out for portability reasons.

[ddfisher]

Thanks for looking into this! :)

[onyb]

@ddfisher: For the test cases to pass, I presume the annotation for cpu_count() must be added to the typeshed/stdlib/3/multiprocessing/__init__.pyi file. Should I submit this change as a separate PR to typeshed?

[ddfisher]

Yes please!

[onyb]

multiprocessing.cpu_count() on Travis CI returns 32. However, the actual number of cores, as reported by len(os.sched_getaffinity(0)) is 2. I am afraid there might be some turbulence on build environments with Python versions not supporting os.sched_getaffinity (like Python 3.2).

Please advise.

[gvanrossum]

Honestly the whole approach to set the default base on CPU count feels a bit over-engineered. Why not just default to 1 or 2? You can use the -j flag to override it.

[onyb]

@gvanrossum: In that case, you can close this issue. Clearly, multiprocessing.cpu_count() can return unpredictable values, especially in CI environments.

[gvanrossum]

Sorry for the wild goose chase!