mach-nix fails to find scipy dependency

[RaitoBezarius]

Hello, thank you for your project, it looks super promising.

Just got this:

The Package 'scipy' cannot be found in the dependency DB used by mach-nix.
Please check the following:
  1. Does the package actually exist on pypi? Please check https://pypi.org/project/scipy/
  2. Does the package's initial release date predate the mach-nix release date?
     If so, either upgrade mach-nix itself or manually specify 'pypi_deps_db_commit' and
     'pypi_deps_db_sha256 for a newer commit of https://github.com/DavHau/pypi-deps-db/commits/master
If none of that works, there was probably a problem while extracting dependency information by the crawler maintaining the database.
Please open an issue here: https://github.com/DavHau/pypi-crawlers/issues/new

By trying mach-nix on github.com/mangaki/zero dependencies.
I'm pretty sure that scipy exists on PyPI and was released before mach-nix :P ; but I don't know what's going on.

[DavHau]

Hey, thanks for reporting the issue.

It appears that the pypi crawler which is maintaining the dependency graph fails on extracting the dependencies for scipy with the following traceback:

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install scipy`   (last SciPy release on PyPI)


Traceback (most recent call last):
  File "<string>", line 5, in <module>
  File "setup.py", line 540, in <module>
    setup_package()
  File "setup.py", line 516, in setup_package
    from numpy.distutils.core import setup
ModuleNotFoundError: No module named 'numpy'

The database containing detailed extraction errors (like this one) is currently non public. I will try to build a little website where people can access status information about all crawled libraries ASAP.

Scipy tries to import numpy before giving the installer the chance to learn about the setup requirements. This behaviour is unusual.

I already noticed that scipy and some other related libraries do some very hacky stuff in their setup.py which is non standard. They try to detect the installer environment and do different things depending on that. I guess they make wrong assumptions about the environment the crawler uses to extract the requirements. I tried to call setuptools the same way pip does it, but obviously you can see that scipy detects that it is not being installed by pip.

The fix for this has to be done in the pypi-crawlers project.
If i get some time i will try to look into it.

[RaitoBezarius]

Maybe, that could be fixed in SciPy code, too. That's super great work so far, and I am so thankful for what you have done, I would love to give you an hand on adding support for wheels or whatever. I can also pester upstream about their way of doing stuff. That's better than nothing.

[DavHau]

It makes me super happy to hear that my work is useful to you. Getting some help would be amazing!
Yeah, i think supporting wheels would be the next bigger step towards increasing the number of supported libraries. Having a general fallback to wheels could also solve the current issue, since the wheel release of scipy might just work.
Pestering upstream sounds fun ;) , but maybe we should have another look at their setup.py file before doing that. The assumptions i made so far are based on a very rough observation. Nothing really solid.
So yeah, your help with anything of that would be great!
This is my first open source project this size, so I'm not sure what are the best steps on my side to make it easy for other people to get involved.
Should I open some tickets including my thoughts on how to move forward on the important topics? Feel free to add me on matrix (@hsngrmpf:matrix.org) or email me.

[RaitoBezarius]

@DavHau I'll try to jump on Matrix later this day, I think that'd be super helpful to know how you envision moving forward (your future milestones, what you think must be done to achieve wheels support, known limitations, nice-to-have, etc.), so that people can jump in and try to contribute.

Classical "contributing" headlines in the README or Contributing.md would be also helpful for first-time contributors! (a Code of Conduct is also a nice-to-have generally.)

I don't know how hard would it be to debug the PyPI crawler on only one package like SciPy, I don't want to run it on the whole universe (as it looks like it takes a certain amount of time :D), so that'd be great if there were a bit of documentation on "debugging", I can take care of it if you could just show me some pointers on how to do it!

Pestering upstream sounds fun ;) , but maybe we should have another look at their setup.py file before doing that. The assumptions i made so far are based on a very rough observation. Nothing really solid.

Well, it sounds like they're doing strange stuff, I just read their setup.py, but I can understand that they need to hack their way to support a lot of platforms.

[DavHau]

Thanks for your suggestions. That's helpful. I'm going to open some new issues with my thoughts soon and let you know.

Debugging the crawler's extraction on a single package shouldn't be difficult. I will add some documentation on that asap.

[DavHau]

I added some instructions on how to debug dependency extraction to pypi-crawlers . To test my instructions i debugged scipy's setup.py already.
Actually this setup.py cannot work with distutils / setuptools installers. It definitely imports numpy to early in any case. Our environment doesn't make a difference there.
Executing the setup.py inside the python docker container which is debian based also fails with the same error.
I wouldn't call it broken because of that. Today i learned, the setup.py-style setup is considered legacy and there are other ways to declare dependencies like pyproject.toml (PEP 518) which is also supported by pip. Scipy's build seems to depend on it.
The way i see it, adding support for PEP 518 (pyproject.toml) to pypi-crawlers would be the right way to include scipy and some other libraries. It should be fairly simple since it's just about parsing the project.toml file. Nothing must be fake-installed.

[RaitoBezarius]

Makes sense, I think that Poetry2Nix already support PEP518 so we can reuse their work I think

[DavHau]

It should be fairly simple since it's just about parsing the project.toml file. Nothing must be fake-installed.

I need to correct this. PEP-518 is just for specifying the minimum build system requirements which doesn't necessarily include all requirements we are interested in. Therefore we would actually need to build an environment with these minimum requirements, and then execute the usual extraction inside this environment. I described it in more detail in the corresponding issue for pypi-crawlers

[RaitoBezarius]

We could also use stuff like this: https://pypi.org/simple/tensorflow/ in order to generate index of wheels URLs. That's something that Poetry2Nix cannot do for example at the moment, I'm trying to see how URL to PyPI are generated, but I don't find the proper PEP for this.

…

On Apr 29 2020, at 5:18 pm, DavHau ***@***.***> wrote: > It should be fairly simple since it's just about parsing the > project.toml file. Nothing must be fake-installed. I need to correct this. PEP-518 is just for specifying the minimum build system requirements which doesn't necessarily include all requirements we are interested in. Therefore we would actually need to build an environment with these minimum requirements, and then execute the usual extraction inside this environment. I described it in more detail in the corresponding issue for pypi-crawlers — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. [ { ***@***.***": "http://schema.org", ***@***.***": "EmailMessage", "potentialAction": { ***@***.***": "ViewAction", "target": "#5 (comment)", "url": "#5 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { ***@***.***": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[DavHau]

Crawling all packages URL names is not a problem at all and done in under 30min. We need to do that anyways, since we need to learn which kind of releases are available and so on. But storing URls actually takes a lot of space currently which could be reduced. I opened an issue concerning this here: DavHau/pypi-crawlers#2 Let's collect our ideas there. I also didn't find a proper spec so far.

I also opened #7 for adding wheel support.

[DavHau]

Mach-nix version 2.0.0 has two new providers nixpkgs and wheel. The problem should be resolved by this