DLL path checks/fixes

[tylerjereddy]

Fixes #57 (eventually)

Currently contains:

• 1 commit to reduce the sizes of the test matrices to improve iteration times (to be reverted when PR is ready)
• 1 commit to augment one of our current wheels test scripts to verify that all DLLs packaged with SciPy wheels are at the same base path

Still likely needs:

• perhaps rename check_license.py now that it is doing more than that (suggested by Pauli)
• confirmation that the msvcp140.dll improper location is actually detected by the new "test"
• placing msvcp140.dll in the correct/updated DLL basepath location

[rgommers]

CI is not happy yet:

Successfully installed scipy-1.5.0.dev0+cf4a407
  File "../check_license.py", line 28
    for filename in Path(install_basedir).rglob('*.dll')

SyntaxError: invalid syntax

[tylerjereddy]

Error message on Windows/Appveyor looks nice now:

python ..\check_license.py
mismatch between current DLL file path:  C:\Python38-x64\lib\site-packages\scipy\extra-dll\msvcp140.dll and the 
reference file path for packaged DLLs:  C:\Python38-x64\lib\site-packages\scipy\.libs

More work needed to only do this on Windows platform & other things noted above.

[tylerjereddy]

The detection of the problem by a "test" in CI looks good now.

I'm iterating on the actual fix now.. since we actually patch over distutils in some cases, using the NumPy version doesn't quite seem safe for making a decision on the DLL storage path.

I've tentatively tried importing the NumPy that is available at wheel build time & using the standard library to parse the source code proper for an indication that .libs is being used, otherwise falling back on extra-dll. I don't know if that is particularly fragile..

[pv]

I'd maybe just hardcode the DLL path, as it's unlikely Numpy will ever change it from .libs (and the test here will then catch it). More likely changes in the future are from Python.org bumping the msvcrt version, so it's probably better to keep it simple here.

[tylerjereddy]

Ok, I backed out the last fix & substituted with a fix that simply replaces extra-dll with .libs in your original DLL copy code.

I guess we'll see if CI likes that. Did NumPy ever explicitly use extra-dll, or was that just coming from us, or was it a Python-wide default of some sort?

[rgommers]

I guess we'll see if CI likes that. Did NumPy ever explicitly use extra-dll, or was that just coming from us, or was it a Python-wide default of some sort?

never a Python thing, IIRC we had the same naming and build setup in NumPy and SciPy. Or it was a multibuild thing (which is still us, more or less)

[pv]

IIRC, extra-dll was used by us because we pinned numpy-distutils to a prerelease version to get the windows build working ASAP. It was changed a bit later to .libs in numpy before the relevant Numpy release, so Numpy releases always used .libs.

[tylerjereddy]

CI was happy with the reduced test matrix scenario--now restoring the full test matrix with fingers crossed.

[tylerjereddy]

32-bit Linux failures look unrelated -- I've opened an issue

The single Appveyor failure so far does look real & related to my changes, though most entries are passing at least:

python ..\check_installed_package.py
Traceback (most recent call last):
  File "..\check_installed_package.py", line 79, in <module>
    main()
  File "..\check_installed_package.py", line 73, in main
    check_dll_paths(mod)
  File "..\check_installed_package.py", line 35, in check_dll_paths
    reference_basepath = os.path.dirname(list_filepaths.pop(0))
  File "C:\Python35\lib\ntpath.py", line 239, in dirname
    return split(p)[0]
  File "C:\Python35\lib\ntpath.py", line 204, in split
    d, p = splitdrive(p)
  File "C:\Python35\lib\ntpath.py", line 139, in splitdrive
    if len(p) >= 2:
TypeError: object of type 'WindowsPath' has no len()
Command exited with code 1

Most likely the DLL path checking code is not quite portable to Python 3.5 on Windows.

[tylerjereddy]

I reproduced the Windows Python 3.5 issue locally & managed to fix the pathlib issues with some string coercion---let's hope CI agrees.

[tylerjereddy]

Incidentally, check_installed_package.py also works to detect the DLL issue in local pip-installed wheels of SciPy 1.3.2 on Windows.

[tylerjereddy]

Windows/Appveyor now all green with DLL adjustments. The 32-bit Linux issues are not related & were briefly discussed separately with Ralf in main repo issue, so in this goes.