Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix plugin resolution to respect interpreter ABI. #8005

Merged

Conversation

@jsirois
Copy link
Member

commented Jul 3, 2019

Previously, changing the python interpreter selected by Pants (e.g: by
editing [python-setup] interpreter_constraints in pants.ini) did not
invalidate an exact requirement plugin resolve leading to runtime errors
loading requirements for the wrong ABI. This is fixed by mixing the
interpreter ABI into the plugin resolve hash and a test is added that
fails without this fix.

@jsirois

This comment has been minimized.

Copy link
Member Author

commented Jul 3, 2019

This issue was discovered when bumping the toolchain repo interpreter constraints via:

 [python-setup]
-interpreter_constraints: ["CPython>=3.6,<4"]
+interpreter_constraints: ["CPython>=3.7,<4"]

I had a local repo that had already setup the same Pants version (1.18.0.dev1) under CPython 3.6.

@jsirois jsirois requested review from Eric-Arellano, benjyw and stuhood Jul 3, 2019

@Eric-Arellano
Copy link
Contributor

left a comment

Code looks good and all valid. Suggestions are to try to make test a bit clearer what's going on.


self.assertEqual(2, len(working_set.entries))

safe_rmtree(repo_dir)

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Jul 3, 2019

Contributor

Brief comment explaining why you do this would help.

# We remove the repo_dir to simulate...

This comment has been minimized.

Copy link
@jsirois

jsirois Jul 3, 2019

Author Member

That's on lines 139-140 below in the self.fail message.

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Jul 3, 2019

Contributor

Agreed. My confusion is why does Py36 still pass? Why do we test Py36 again and delete the repo_dir before doing so?

self.fail('Plugin re-resolution is expected for an incompatible interpreter and it is '
'expected to fail since we removed the dist `repo_dir` above.')

with self.plugin_resolution(interpreter=python36,

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Jul 3, 2019

Contributor

I recommend moving this above the py37 case and attaching a comment like:

# Even with a deleted repo_dir, this should still work because the interpreter version has not changed.

This comment has been minimized.

Copy link
@jsirois

jsirois Jul 3, 2019

Author Member

That's exactly what the test above does: https://github.com/pantsbuild/pants/pull/8005/files/30706e2a8459e7a45b767523eac0eefc6f1fdf79#diff-d996ef17ea0368099997c262eeb24516R104

If you feel strongly that I should repeat this here I can, but this test builds on that test.

This comment has been minimized.

Copy link
@jsirois

jsirois Jul 4, 2019

Author Member

OK - I am not a fan of over-explaining, but that characterization is in the eye of the beholder. I added a comment so that you can read just this one test and get it without reading the test above.

jsirois added some commits Jul 3, 2019

Fix plugin resolution to respect interpreter ABI.
Previously, changing the python interpreter selected by Pants (e.g: by
editing `[python-setup] interpreter_constraints` in `pants.ini`) did not
invalidate an exact requirement plugin resolve leading to runtime errors
loading requirements for the wrong ABI. This is fixed by mixing the
interpreter ABI into the plugin resolve hash and a test is added that
fails without this fix.
Fix missing BUILD dep.
Refactor
tests/python/pants_test/backend/python/interpreter_selection_utils.py to
live in tests/python/pants_test since its now used outside of the python
backend tests.
Explain test_exact_requirements_interpreter_change.
Although redundant to the story told by test_exact_requirements above,
it's sometimes nice to have local information when your focus is narrow.

@jsirois jsirois force-pushed the jsirois:plugins/fix-resolve-exact-invalidation branch from 30706e2 to 97eadfd Jul 4, 2019

@Eric-Arellano
Copy link
Contributor

left a comment

I don't get why isort keeps changing it's expectations on us.. This was the original style, and then I had to do 6ab5f7a#diff-8ac5fea0ec4c55d0e1e1d6e231f9d2ae to fix the failing lint shard. And now it looks like we have to revert it again :/

Beyond that, looks great.

'expected to fail since we removed the dist `repo_dir` above.')

# But for a compatible interpreter the exact resolve results should be re-used and load
# directly from the still in-tact cache.

This comment has been minimized.

Copy link
@Eric-Arellano

Eric-Arellano Jul 4, 2019

Contributor

Thank you. This is very helpful!

@jsirois

This comment has been minimized.

Copy link
Member Author

commented Jul 6, 2019

I don't get why isort keeps changing it's expectations on us.

It looks there are at least two problems with how we isort:

  1. In the Pants project itself: we don't specify import wrap style and so rely on some default:

    pants/.isort.cfg

    Lines 1 to 10 in a87c1a5

    # See the menu of settings available here:
    # https://github.com/timothycrosley/isort/wiki/isort-Settings
    [settings]
    line_length=100
    indent=2
    lines_after_imports=2
    known_first_party=internal_backend,pants,pants_test
    default_section=THIRDPARTY
    not_skip=__init__.py

    The default wrap mode is GRID (0):
    https://github.com/timothycrosley/isort/blob/4.3.20/isort/settings.py#L70-L126
  2. The isort configuration system includes searching for ~/.isort.cfg, ~/.config/isort.cfg and ~/.editorconfig:
    https://github.com/timothycrosley/isort/blob/4.3.20/isort/settings.py#L169-L181

I suspect you have one of the configs in 2.

#8020

@jsirois jsirois merged commit 96c152a into pantsbuild:master Jul 6, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jsirois jsirois deleted the jsirois:plugins/fix-resolve-exact-invalidation branch Jul 6, 2019

illicitonion added a commit that referenced this pull request Jul 10, 2019

Fix plugin resolution to respect interpreter ABI. (#8005)
Previously, changing the python interpreter selected by Pants (e.g: by
editing `[python-setup] interpreter_constraints` in `pants.ini`) did not
invalidate an exact requirement plugin resolve leading to runtime errors
loading requirements for the wrong ABI. This is fixed by mixing the
interpreter ABI into the plugin resolve hash and a test is added that
fails without this fix.

Also refactor
tests/python/pants_test/backend/python/interpreter_selection_utils.py to
live in tests/python/pants_test since its now used outside of the python
backend tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.