PyB plugins use wrong external tool dependencies

[arcivanov]

Throughout the PyB internal plugins use project.build_depends_on:

• Flake8
• Coverage

Firstly, PyB build does not really depend on those. It's the runtime use of the plugin that requires these external tools.

Therefore, I propose the following:

• Create another method project.plugin_depends_on(package_name, version_spec=None, external_tool=None)
  • package_name is package name 😄
  • version_spec, if specified is a required PEP0440 SpecifierSet (see Plugin-specified PyB API version constraints #271)
  • external_tool if specified is then name of the executable to check with assert_can_execute

project.plugin_depends_on will:

• via pip show <package_name> check the version of the package installed
• if nothing is installed, pip install <package_name>[version_spec] will be executed
• if package is installed but version_spec is specified and version is not contained in SpecifierSet pip install --upgrade <package_name>[version_spec] will be run
• once package versions are settled, if external_tool is specified the assert_can_execute will be performed.

@mriehl @esc review please

[arcivanov]

@mriehl @esc thoughts? I'm going to start implementing as stated, please weigh in if you have objections

[mriehl]

get_package_version doesn't seem to always work, on my machine (using a virtualenv) for instance it always reports None even when the packages are installed, leading to long build times since dependencies are installed all the time.

I'm investigating why that is.

[mriehl]

For example

» pip list | grep cram
cram (0.6)

but

>>> list(search_packages_info("cram"))
[]

[mriehl]

Ahhh. In the pip version I have, search_packages_info expects a Traversable:

https://github.com/pypa/pip/blob/13d43e3af8b5c8e8c7c66dde8a9f76b340697c3e/pip/commands/show.py#L54

So this works:

>>> list(search_packages_info(["cram"]))
[{'files': ['../../../bin/cram', 'cram-0.6.dist-info/DESCRIPTION.rst', 'cram-0.6.dist-info/METADATA', 'cram-0.6.dist-info/RECORD', 'cram-0.6.dist-info/WHEEL', 'cram-0.6.dist-info/pydist.json', 'cram-0.6.dist-info/top_level.txt', 'cram.py', 'cram.pyc'], 'name': 'cram', 'license': 'GNU GPL', 'author': 'Brodie Rao', 'metadata-version': '2.0', 'home-page': 'https://bitheap.org/cram/', 'summary': 'A simple testing framework for command line applications', 'version': '0.6', 'location': '/data/home/mriehl/workspace/pybuilder/venv/lib/python2.7/site-packages', 'requires': [], 'author-email': 'brodie@bitheap.org'}]

Isn't that the case on your pip version @arcivanov ?

[arcivanov]

@mriehl search_packages_info always works for me with an iterable query. I'm going to create a separate issue to increase plugin dependency check performance.

PS: I see what you mean! Good catch!

[mriehl]

@arcivanov I think there's still an issue :/ the latest build from master installed unittest-xml-reporting twice, for example (but I'm not sure why that is).

https://travis-ci.org/pybuilder/pybuilder/jobs/95025723#L749

and

https://travis-ci.org/pybuilder/pybuilder/jobs/95025723#L1443

[mriehl]

Ah, looking at https://github.com/arcivanov/pybuilder/blob/da5f51009fd0df2889d9939de39e276049e386ca/src/main/python/pybuilder/pip_utils.py#L137 it seems the pip query is using the package name string as an iterable, e.G. ["u", "n", "i", "t", "t", "e", "s", "t".....] so this doesn't match anything. We should return [mixed.name] when normalizing instead.

[arcivanov]

@mriehl at https://github.com/arcivanov/pybuilder/blob/da5f51009fd0df2889d9939de39e276049e386ca/src/main/python/pybuilder/pip_utils.py#L137 we now always have a collection there (as_list(mixed)), no?

[mriehl]

Yes, that's right.. I'm not sure why travis installs plugin runtime dependencies twice then. Will investigate tomorrow morning

[arcivanov]

I'm investigating already, don't worry about it. I suspect this is related to how pip is implemented and that it doesn't actually refresh internal package cache every time running in-process.

[arcivanov]

@mriehl yep, it's internal pip implementation

/Users/arcivanov/pyhome/current/bin/python "/Applications/PyCharm CE.app/Contents/helpers/pycharm/utrunner.py" /Users/arcivanov/Documents/src/arcivanov/pybuilder/src/unittest/python/pip_utils_tests.py::PipVersionTests::test_test_install_test true
Testing started at 13:09 ...
{}
Collecting unittest-xml-reporting
  Using cached unittest_xml_reporting-1.13.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.0 in /Users/arcivanov/pyhome/master/lib/python2.7/site-packages (from unittest-xml-reporting)
Installing collected packages: unittest-xml-reporting
Successfully installed unittest-xml-reporting-1.13.0
{}

Process finished with exit code 0

for code

        print(pip_utils.get_package_version("unittest-xml-reporting"))
        pip_utils.pip_install("unittest-xml-reporting")
        print(pip_utils.get_package_version("unittest-xml-reporting"))

[mriehl]

Yup works great now! Thanks 👍