BLD: rewrite np.distutils.exec_command.exec_command()

[pitrou]

exec_command() is currently a mess of several implementations using outdated Python APIs and various hacks. This rewrites it to use the standard subprocess module. See PR #7614 for previous discussion.

I've tested this on Python 2.7 and 3.5, Windows and Linux.

[pitrou]

There's wacky failure on one of the Travis builders that I can't really understand...

[rgommers]

There's wacky failure on one of the Travis builders that I can't really understand...

Looks odd indeed, I restarted that build to make sure it's a real issue.

[rgommers]

Hmm, it fails reproducibly. Many times /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8).

[rgommers]

Will be a nice cleanup, thanks @pitrou.

[rgommers]

Scary piece of untested code to touch though:)

[pitrou]

It does have tests in numpy/distutils/test_exec_command. As for the failure, it seems it's counting the number of warnings and for a reason fails if they are > 32? Perhaps the previous implementation didn't let the warnings through (it's difficult to make out, as the old code is so convoluted...).

[rgommers]

Would be good to check also a Scipy build on Windows, to make sure we don't make the "too many open files" errors worse there. I recently had subprocess issues with that in another context, because it has a pretty bad bug on Python 2.7 with close_fds not working, so hard to work around.

[rgommers]

It does have tests in numpy/distutils/test_exec_command.

Indeed it does, forgot about that. The redirect_stdout context manager there brings back some unpleasant memories:)

[pitrou]

Hmm, it fails reproducibly. Many times /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8).

I've solved the issue by setting the locale to something existing in the build script. This was really a bug in the build script.

[pitrou]

Would be good to check also a Scipy build on Windows

I've tried too, but I end up with "numpy.distutils.system_info.NotFoundError: no lapack/blas resources found". I guess someone more expert than me will have to try it out.

[juliantaylor]

I guess there are no known problems on windows?
Its probably best to just merge it and see what happens then. Can you maybe rebase it again to get an up to date ci suite?

[juliantaylor]

thanks this was long overdue refactoring

[minrk]

Bonus comment from #8767 for posterity: this patch allows parallel builds to actually run in parallel on macOS.