[FEATURE] Use grayskull with any pypi mirror/server

[lucharo]

Is your feature request related to a problem? Please describe.
As per my reading and testing of this tool, I do not think it is is possible to make a conda recipe out of any pypi mirror just out of the official one (ref to code: https://github.com/conda-incubator/grayskull/blob/d72b59804ab6800d18f1cef40c03ff22565ff2b2/grayskull/pypi/pypi.py#L40).
Describe the solution you'd like
I would like to be able to use grayskull to make conda recipes out of any python package listed in any arbitrarily provided pypi mirror or even locally. Perhaps by having a CLI command option:

grayskull --mirror https://pypi.myenterprise.com <python-package-name>

Describe alternatives you've considered
Building a conda recipe manually and having undesired behaviours.

[marcelotrevisani]

Yes, that is something which is good to have indeed!

For another repositories that would be relatively easy to change, for local/specific paths it will need a bit more of modifications. But it is something doable.

I will add it to my plans, thanks for this request!

[lucharo]

Nice, thanks Marcelo! I think this feature will be especially useful in enterprise environments (which is precisely my use case) 😄

[lucharo]

Is this line:
https://github.com/conda-incubator/grayskull/blob/d72b59804ab6800d18f1cef40c03ff22565ff2b2/grayskull/pypi/pypi.py#L40

The one actually responsible for grayskull looking in pypi.org only? If I changed that one to my entreprise pypi server would it work?

[marcelotrevisani]

Is this line:
https://github.com/conda-incubator/grayskull/blob/d72b59804ab6800d18f1cef40c03ff22565ff2b2/grayskull/pypi/pypi.py#L40

The one actually responsible for grayskull looking in pypi.org only? If I changed that one to my entreprise pypi server would it work?

It will depend if your enterprise pypi server will return the metadata correctly. If they return it like pypi it will work, otherwise I am afraid that might not work
But it is on my plans to also generate the recipe from sdist packages located anywhere, that will solve your problem as well

[lucharo]

Actually, I think it would be really useful to have grayskull look in all the channels specified by the .pypirc and .condarc the user has. What do you think @marcelotrevisani ?

Edit: not .pypirc but .pip/pip.conf instead

[marcelotrevisani]

Yeap, that is a nice improvement to have, a bit of work but it is doable :)

[lucharo]

I've just tested grayskull with https://test.pypi.org as the host and it works well:

https://github.com/lc5415/grayskull/blob/340e9ea463f293221cd7ca0a444c0df06b531cb6/grayskull/pypi/pypi.py#L40

It was surprisingly difficult to find up-to-date public pypi servers but I guess test.pypi.org will do for now. While it's good that the functionality is there with a different host than pypi.org I think the CLI may need to change a bit (e.g. grayskull --host test.pypi.org.

I'd be happy to submit a PR but there's quite a bit of code to dig through. I'd be happy to dig deeper if you give me any pointers @marcelotrevisani
This is what I get for pytest:

#### Initializing recipe for pytest (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package pytest-0.0.0.dev1.tar.gz
pytest-0.0.0.dev1.tar.gz 100% Time:  0:00:00   1.1 MiB/s|####################################################################|
Recovering information from setup.py
Executing injected distutils...
Recovering metadata from setup.cfg
Checking >> numpy 100% |##############################################################################|[Elapsed Time: 0:00:00]
Recovering license info from spdx.org ...
Matching license file with database from Grayskull...
License type: MIT
License file: LICENSE.txt
Host requirements:
  - pip
  - python

Run requirements:
  - numpy
  - python

RED: Missing packages
GREEN: Packages available on conda-forge
Maintainers:
   - LuisF3

#### Recipe generated on /Users/luischavesrodriguez/Documents/ExtratimeWork/grayskull for pytest ####

[marcelotrevisani]

I've just tested grayskull with https://test.pypi.org as the host and it works well:

https://github.com/lc5415/grayskull/blob/340e9ea463f293221cd7ca0a444c0df06b531cb6/grayskull/pypi/pypi.py#L40

It was surprisingly difficult to find up-to-date public pypi servers but I guess test.pypi.org will do for now. While it's good that the functionality is there with a different host than pypi.org I think the CLI may need to change a bit (e.g. grayskull --host test.pypi.org.

I'd be happy to submit a PR but there's quite a bit of code to dig through. I'd be happy to dig deeper if you give me any pointers @marcelotrevisani
This is what I get for pytest:

#### Initializing recipe for pytest (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package pytest-0.0.0.dev1.tar.gz
pytest-0.0.0.dev1.tar.gz 100% Time:  0:00:00   1.1 MiB/s|####################################################################|
Recovering information from setup.py
Executing injected distutils...
Recovering metadata from setup.cfg
Checking >> numpy 100% |##############################################################################|[Elapsed Time: 0:00:00]
Recovering license info from spdx.org ...
Matching license file with database from Grayskull...
License type: MIT
License file: LICENSE.txt
Host requirements:
  - pip
  - python

Run requirements:
  - numpy
  - python

RED: Missing packages
GREEN: Packages available on conda-forge
Maintainers:
   - LuisF3

#### Recipe generated on /Users/luischavesrodriguez/Documents/ExtratimeWork/grayskull for pytest ####

That would be great!
I believe you will need to add the option --host to the parser in __main__.py with the default to be the pypi server, and you will need to pass the host info to the create_recipe
https://github.com/conda-incubator/grayskull/blob/eebe5d423a1a7b5e6369fac962771a40da157a9c/grayskull/__main__.py#L121

This part here
https://github.com/conda-incubator/grayskull/blob/eebe5d423a1a7b5e6369fac962771a40da157a9c/grayskull/pypi/pypi.py#L40
You will need to add the host as a parameter to the __init__ and the URL_PYPI_METADATA will need to be a attribute of the object PyPi

[marcelotrevisani]

Just one thing, I believe would be better to name it as mirror instead of host. Because host already has a meaning in the conda recipe

[FernandezMathieu]

Same need here, would be great !

[marcelotrevisani]

That was developed already.
you can use the option --pypi-url and pass the pypi URL of your local server or mirror