Default "type_stub_modules" deduction using "types-" prefix is not working (TypeError: '<' not supported between instances of 'ModuleProviderType' and 'ModuleProviderType').

[AlirezaRoshanzamir]

Description
According to the python_requirement's type_stub_modules field documentation, the requirements with the "types-" prefix is deduced as a stub automatically, but running any command (such as check, test, and etc) raises:

TypeError: '<' not supported between instances of 'ModuleProviderType' and 'ModuleProviderType'

The simple use-case:

.
├── BUILD
├── main.py
└── pants.toml

# BUILD

python_requirement(
    name='requirement',
    requirements=[
        'docopt',
        'types-docopt'
    ],
    # type_stub_modules=['docopt']  # If the comment is removed, the error will be gone.
)

python_sources(name='sources', dependencies=[':requirement'])

# pants.toml

[GLOBAL]
pants_version = '2.10.0'
use_deprecated_python_macros = false
backend_packages = [
    'pants.backend.python',
    'pants.backend.python.typecheck.mypy',
]

# main.py

from docopt import docopt

However, by uncommenting the type_stub_modules=['docopt'], the error will gone.

Pants version: 2.10.0
OS: Ubuntu 20.04.3 LTS

[Eric-Arellano]

Thank you so much for this report! I will take a look today.

[Eric-Arellano]

By the way, it is recommended generally that you have a distinct Python requirement target for each of those requirements.

python_requirement(
    name='docopt',
    requirements=['docopt'],
)

python_requirement(
    name='types-docopt',
    requirements=['types-docopt'],
)

If you do it that way, there is no need to set the type stub modules field. The defaults will work.

That can be annoying to type that all out, so many people use the Python requirements target generator: https://www.pantsbuild.org/docs/python-third-party-dependencies#requirementstxt

[AlirezaRoshanzamir]

Thank you for your instant response and fix.

We are deciding to make our company Pythons' projects Pants compatible. The main reason for this migration is to avoid too many files and tools when developing code and managing the interdependence between projects:

1. Tox (tox.ini) for virtual environment manipulation, clean task execution, and multi-python version testing.
2. Many requirements.in (installation, development, type-check, format, ...) in editable/non-editable versions. They install first-party dependencies using the flags for relative Python projects and requirements.txt installation (which has an inconsistent relative path resolution between relative requirements.in and relative Python project).
3. Setuptools (setup.py) for packaging which reads one of the requirements.in files for preventing duplication.
4. Makefile for some virtual environment independent tasks and a single-point-of-entry for task running. Also, automatic installation of non-Python dependencies (such as Docker images) is handled using Makefiles.
5. Flake8, isort, mypy, and pytest tools and corresponding config files. However, the config files can be put in a single file (such as tox.ini).
6. Lock files and Pip-tools for dependency pinning.
7. There is also a Makefile at the root of the repository to run tasks in parallel on projects that support them.
8. Some other files and tools (such as Twine, Jenkins/Jenkinsfile, MANIFEST.in, sphinx-build, etc).

For novice developers, there are many tools and flows to learn for starting development. So, we prefer a single BUILD (which Pants is not happy with that and we put different BUILD files for src/ and tests/ directories) for each project (similar to MSBuild, Ant, and Maven) with all properties needed and targets/tasks supported.

[benjyw]

Hi @AlirezaRoshanzamir , welcome to Pants! Glad we could fix this issue, and just wanted to let you know that the Pants slack workspace is a great place to ask questions: https://www.pantsbuild.org/docs/getting-help

[benjyw]

Also, you should be able to have a single top-level BUILD file if you really want one, with overrides= if you need to override metadata for a specific file. But we've found that it's generally convenient to have a BUILD file in each leaf directory, even if it's a very trivial one, so that's what ./pants tailor will generate. But it shouldn't make any difference to the end-user experience either way, since end users can operate directly on files and directories, and don't need to know about BUILD targets etc.

[AlirezaRoshanzamir]

@benjyw

Thanks a lot. I will definitely join.

But there seems to be a set of assumptions that make this somewhat problematic (such as #15084) or non-boilerplate (for example, the default "sources" field in the "python_sources" and "python_tests" field which assumes that all the sources are in the current directory).

I should share and discuss this issue in the Slack workspace.