Allow passing a package mapping for pypi->conda-forge packages

[mariusvniekerk]

We should have some way of having a mapping where pypi package names (such as docker vs docker-py) can be mapped from pypi names to conda names.

[marcelotrevisani]

We do have it on grayskull :)
Please see
https://github.com/marcelotrevisani/grayskull/blob/master/grayskull/pypi/config.yaml

[marcelotrevisani]

So we will need to just add a new entry to that file, something like

docker:
  import_name: docker
  conda_forge: docker-py

[marcelotrevisani]

Do you know other differences of project names between pypi and conda-forge?
I just have a few in that file

[mariusvniekerk]

We can probably back it out of the feedstock graph + libcfgraph

[marcelotrevisani]

I don't know what you mean, sorry
Can you explain a bit more please?

[mariusvniekerk]

going to take a shot at generating this automatically from our metadata in https://github.com/regro/cf-graph-countyfair/tree/master/node_attrs

[mariusvniekerk]

So looking at the conda-forge metadata i can generate a list of 174 packagest that have a non-case mismatch between conda-forge and pypi name.

[mariusvniekerk]

we can probably generate this automatically as part of the conda-forge bot and publish it to github somewhere?

[mariusvniekerk]

Automatically found mismatches

Boruta:
  conda_forge: boruta_py
  import_name: boruta
Flask-Cors:
  conda_forge: flask_cors
  import_name: flask_cors
Flask-JSON:
  conda_forge: flask_json
  import_name: flask_json
SoundFile:
  conda_forge: pysoundfile
  import_name: soundfile
aiohttp_cors:
  conda_forge: aiohttp-cors
  import_name: aiohttp_cors
backtrace:
  conda_forge: python-backtrace
  import_name: backtrace
bash_completion:
  conda_forge: py-bash-completion
  import_name: bash_completion
captest:
  conda_forge: pvcaptest
  import_name: captest
cdo:
  conda_forge: python-cdo
  import_name: cdo
click_config_file:
  conda_forge: click-config-file
  import_name: click_config_file
cockroachdb:
  conda_forge: cockroachdb-python
  import_name: cockroachdb
conda_lock:
  conda_forge: conda-lock
  import_name: conda_lock
conda_mirror:
  conda_forge: conda-mirror
  import_name: conda_mirror
coreapi:
  conda_forge: python-coreapi
  import_name: coreapi
coreschema:
  conda_forge: python-coreschema
  import_name: coreschema
cryptography_vectors:
  conda_forge: cryptography-vectors
  import_name: cryptography_vectors
cufflinks:
  conda_forge: python-cufflinks
  import_name: cufflinks
dask:
  conda_forge: dask-core
  import_name: dask
datadotworld:
  conda_forge: datadotworld-py
  import_name: datadotworld
datalad_container:
  conda_forge: datalad-container
  import_name: datalad_container
delegator.py:
  conda_forge: delegator
  import_name: delegator
dials_data:
  conda_forge: dials-data
  import_name: dials_data
dye_score:
  conda_forge: dye-score
  import_name: dye_score
empyrical_dist:
  conda_forge: empyrical-dist
  import_name: empyrical_dist
esprima:
  conda_forge: esprima-python
  import_name: esprima
et_xmlfile:
  conda_forge: et-xmlfile
  import_name: et_xmlfile
evalml:
  conda_forge: evaml-core
  import_name: evalml
extract-msg:
  conda_forge: msg-extractor
  import_name: extract_msg
flake8_nb:
  conda_forge: flake8-nb
  import_name: flake8_nb
flatbuffers:
  conda_forge: python-flatbuffers
  import_name: flatbuffers
flex:
  conda_forge: flex-swagger
  import_name: flex
gnupg:
  conda_forge: gnupg-py
  import_name: gnupg
google:
  conda_forge: googlesearch
  import_name: googlesearch
graphviz:
  conda_forge: python-graphviz
  import_name: graphviz
h2o:
  conda_forge: h2o-py
  import_name: h2o
hdfs:
  conda_forge: python-hdfs
  import_name: hdfs
iam_units:
  conda_forge: iam-units
  import_name: iam_units
ib_insync:
  conda_forge: ib-insync
  import_name: ib_insync
intake-astro:
  conda_forge: intake-accumulo
  import_name: intake_astro
intake_geopandas:
  conda_forge: intake-geopandas
  import_name: intake_geopandas
ioos_tools:
  conda_forge: ioos-tools
  import_name: ioos_tools
jupyter_jaeger:
  conda_forge: jupyter-jaeger
  import_name: jupyter_jaeger
jupyterlab_git:
  conda_forge: jupyterlab-git
  import_name: jupyterlab_git
jupyterlab_latex:
  conda_forge: jupyterlab-latex
  import_name: jupyterlab_latex
libaio:
  conda_forge: python-libaio
  import_name: libaio
md_toc:
  conda_forge: md-toc
  import_name: md_toc
message_ix:
  conda_forge: message-ix
  import_name: message_ix
mss:
  conda_forge: python-mss
  import_name: mss
nbconvert_utils:
  conda_forge: nbconvert-utils
  import_name: nbconvert_utils
ndarray_listener:
  conda_forge: ndarray-listener
  import_name: ndarray_listener
neo:
  conda_forge: python-neo
  import_name: neo
nest_asyncio:
  conda_forge: nest-asyncio
  import_name: nest_asyncio
node-semver:
  conda_forge: python-node-semver
  import_name: semver
nvidia-ml-py3:
  conda_forge: nvidia-ml
  import_name: pynvml
pandas_flavor:
  conda_forge: pandas-flavor
  import_name: pandas_flavor
paragraph:
  conda_forge: python-paragraph
  import_name: paragraph
prince:
  conda_forge: prince-factor-analysis
  import_name: prince
pvlib:
  conda_forge: pvlib-python
  import_name: pvlib
pyct:
  conda_forge: pyct-core
  import_name: pyct
pytest_check_links:
  conda_forge: pytest-check-links
  import_name: pytest_check_links
python-datamatrix:
  conda_forge: datamatrix
  import_name: datamatrix
python-fileinspector:
  conda_forge: fileinspector
  import_name: fileinspector
python-pseudorandom:
  conda_forge: pseudorandom
  import_name: pseudorandom
python-pygaze:
  conda_forge: pygaze
  import_name: pygaze
python-qdatamatrix:
  conda_forge: qdatamatrix
  import_name: qdatamatrix
python-qnotifications:
  conda_forge: qnotifications
  import_name: QNotifications
quantum-grove:
  conda_forge: grove
  import_name: grove
radio_beam:
  conda_forge: radio-beam
  import_name: radio_beam
rocketpyalpha:
  conda_forge: rocketpy
  import_name: rocketpy
sage_flatsurf:
  conda_forge: sage-flatsurf
  import_name: flatsurf
sagemaker_inference:
  conda_forge: sagemaker-inference-toolkit
  import_name: sagemaker_inference
sagemaker_mxnet_training:
  conda_forge: sagemaker_mxnet_container
  import_name: sagemaker_mxnet_container
sounddevice:
  conda_forge: python-sounddevice
  import_name: sounddevice
spherical-functions:
  conda_forge: spherical_functions
  import_name: spherical_functions
stjudecloud-oliver:
  conda_forge: oliver
  import_name: oliver
termstyle:
  conda_forge: python-termstyle
  import_name: termstyle
torch_geometric:
  conda_forge: pytorch_geometric
  import_name: torch_geometric
useDAVE:
  conda_forge: dave
  import_name: numpy
vsts:
  conda_forge: vsts-python-api
  import_name: vsts

[CJ-Wright]

it would be good if this listing was independent of grayskull itself, so we can generate it dynamically

[marcelotrevisani]

we can probably generate this automatically as part of the conda-forge bot and publish it to github somewhere?

That would be nice indeed

[marcelotrevisani]

I have a webservice in my personal vps which I use it sometimes to just generate the recipe do a project on pypi using grayskull for me
It is possible to create an api to retrieve that list
Actually there is an api there already, but it just retrieve the recipe, I can extended or create a new service just for this part

[mariusvniekerk]

Okay we have started producing this mapping as part of the bot. It should update roughly hourly

https://github.com/regro/cf-graph-countyfair/blob/master/mappings/pypi/grayskull_pypi_mapping.yaml

[BastianZim]

Hi, just to check; what's currently the correct place to add such quirks? Is it still https://github.com/conda-incubator/grayskull/blob/master/grayskull/pypi/config.yaml?

Reason is that I would like to add fastjsonschema (fastjsonschema vs python-fastjsonschema) and flair (flair vs python-flair).

[marcelotrevisani]

Hi, just to check; what's currently the correct place to add such quirks? Is it still https://github.com/conda-incubator/grayskull/blob/master/grayskull/pypi/config.yaml?

Reason is that I would like to add fastjsonschema (fastjsonschema vs python-fastjsonschema) and flair (flair vs python-flair).

Hi @BastianZim ,
Yes, that is correct. You will need to add it there, just follow the previous entries and it should be fine. :)

[BastianZim]

Perfect, thank you @marcelotrevisani!

[sgbaird]

Hmm, I worked around this somewhat manually using conda-souschef (marcelotrevisani/souschef#32). Not sure I understand the idea behind using config.yaml as described here. Is the idea to do a local installation of grayskull and edit that file? Also https://github.com/conda- incubator/grayskull/blob/master/grayskull/pypi/config.yaml redirects to a dead link: https://github.com/conda-incubator/grayskull/blob/main/grayskull/pypi/config.yaml.

[BastianZim]

Sometimes, the names in conda-forge are different than in PyPI. What this config.yaml does is it saves all of the differences and then when you run Grayskull on a package, it will give out the correct dependency.

For example, let's assume you have package x which has a dependency called y but on conda-forge the dependency is called py-y. When you now run Grayskull on x the dependency list will include py-y and not y.

[sgbaird]

@BastianZim where should config.yaml be placed? Should I make a folder structure in the project: grayskull/pypi/config.yaml?

Didn't see any files on grayskull called config.yaml: https://github.com/conda-incubator/grayskull/search?q=config.yaml

More just wondering how/if I should be using config.yaml.

As I mentioned, my workaround has been to manually replace those Anaconda/PyPI naming differences using conda-souschef, but didn't realize there might already be some functionality for that.

[BastianZim]

The file already exists, it was just moved to here: https://github.com/conda-incubator/grayskull/blob/main/grayskull/strategy/config.yaml

And yeah, GitHub search isn't always the best...

You can use souschef but if you add the respective package to the "official" yaml, others will be able to use it as well.

[sgbaird]

@BastianZim this is great! Thanks for letting me know, and good to know about GitHub search. Didn't realize that - will probably stick with VS Code Ctrl+Shift+F when I want to be sure. Will open a PR for kaleido --> python-kaleido