Add `[python].resolves_to_constraints_file` #16420

Eric-Arellano · 2022-08-05T00:04:01Z

Constraints files are extremely useful for pinning problematic transitive deps.

We abused constraints files early on as a poor-man's lockfile via [python].requirement_constraints. While we want to deprecate that in favor of [python].resolves, it is useful to allow constraint files for resolves.

Specifically, each resolve should be allowed its own constraints file. For example, if resolve A uses Django 2, it may need to pin different problematic transitive deps of Django than resolve B using Django 3.

--

This PR is the first out of a project to allow specifying config for lockfiles on a per-resolve basis, rather than globally for all resolves: https://docs.google.com/document/d/1HAvpSNvNAHreFfvTAXavZGka-A3WWvPuH0sMjGUCo48/edit#.

--

This requires us now tracking constraints_file_hash in the lockfile metadata, so that when the user changes a constraints file, we know to regenerate the lock.

Note that this means we have a V3 of lockfile metadata. It is okay to keep using V2 -- the next time people run generate-lockfiles, they will be bumped.

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois · 2022-08-05T00:28:21Z

... pip would complain.

Exactly! Let Pip do so and do not nanny tools if you don't need to. But that's neither here nor there, I'm just continually pointing out poor argument noise when I hear it. I think its fine to start with just constraints files as a whole. Synthesizing them from other configuration sources later and letting the proper nanny (Pip) do the nannying, is always an option.

jsirois

AFAICT, including constraints but not invalidating the lock based on constraints changing is a bug.

src/python/pants/backend/python/goals/lockfile_test.py

[ci skip-rust] [ci skip-build-wheels]

Eric-Arellano · 2022-08-08T18:51:02Z

src/python/pants/backend/python/util_rules/pex_requirements.py

+    """Configuration from `[python]` that impacts how the resolve is created."""
+
+    constraints_file: tuple[Digest, FileEntry] | None


This will soon also have --only-binary and --no-binary. Possibly [python-repos] options, pending Slack discussion

Eric-Arellano · 2022-08-08T18:51:41Z

src/python/pants/core/util_rules/lockfile_metadata.py

-            if isinstance(self, cls):
+            # Note that we do exact version matches so that authors can subclass earlier versions.
+            if type(self) is cls:


Thanks @chrisjrn for the fix

Eric-Arellano · 2022-08-08T18:52:56Z

src/python/pants/backend/python/util_rules/pex_requirements.py

+    all_tool_resolve_names = tuple(
+        sentinel.resolve_name for sentinel in union_membership.get(GenerateToolLockfileSentinel)
+    )


This is weird. It means that our error message for unrecognized resolve names is going to include JVM tools like javac. I could not think of a good way to filter down to only Python tools.

Its also a bit weird in the small that you could ask for ResolvePexConfig for the resolve "junit" and get back a non-error answer at all. It definitely seems like a level of typing structure is missing from the infra here.

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois · 2022-08-08T19:25:10Z

src/python/pants/backend/python/goals/lockfile.py

+        pip_args_file = "__pip_args.txt"
+        extra_args.extend(["-r", pip_args_file])
+        pip_args_file_content = "\n".join(
+            [f"--no-binary {pkg}" for pkg in python_setup.no_binary]


This was pre-existing, but changing the order of the python_setup args produces a new file here and the subsequent downstream invalidations that flow when it need not, --no-binary / --only-binary is not an order-sensitive thing.

This was pre-existing

My next PR will likely be reworking this mechanism to be resolve-specific, so I can clean that up there.

jsirois · 2022-08-08T19:36:15Z

src/python/pants/backend/python/util_rules/pex_requirements.py

+    all_tool_resolve_names = tuple(
+        sentinel.resolve_name for sentinel in union_membership.get(GenerateToolLockfileSentinel)
+    )


Its also a bit weird in the small that you could ask for ResolvePexConfig for the resolve "junit" and get back a non-error answer at all. It definitely seems like a level of typing structure is missing from the infra here.

jsirois · 2022-08-08T19:37:44Z

src/python/pants/backend/python/util_rules/pex_requirements.py

@@ -306,6 +381,9 @@ def validate_metadata(
        lockfile=lockfile,
        user_interpreter_constraints=interpreter_constraints,
        user_requirements=user_requirements,
+        maybe_constraints_file_path=constraints_file_path_and_hash[0]


Types certainly mean 0 must be the correct index, but its definitely not reader-friendly.

Agreed. I considered using NamedTuple, but felt confusing to introduce yet another class when we already have so many. So I was hoping the path_and_hash[0] would be clear enough it means path. Thoughts?

I think I expressed my thoughts! I'd personally break out a 2 field dataclass, but I ship-it-ed because this is all fashion and not too important to the end goal.

I second that this should be a dataclass.

benjyw · 2022-08-09T02:12:20Z

src/python/pants/backend/python/util_rules/pex_requirements.py

+    constraints_file: tuple[Digest, FileEntry] | None
+
+    @property
+    def constraints_file_path_and_hash(self) -> tuple[str, str] | None:


It would be better if this returned a dataclass and not a tuple whose unpacking is not self-explanatory. Reading the code that uses this was confusing.

But this can wait for a followup.

John agreed with you. Will do :)

benjyw · 2022-08-09T02:13:32Z

src/python/pants/backend/python/util_rules/pex_requirements.py

@@ -306,6 +381,9 @@ def validate_metadata(
        lockfile=lockfile,
        user_interpreter_constraints=interpreter_constraints,
        user_requirements=user_requirements,
+        maybe_constraints_file_path=constraints_file_path_and_hash[0]


I second that this should be a dataclass.

…bclasses (#16449) Closes #16443, which is necessary for the per-resolve config project from https://docs.google.com/document/d/1HAvpSNvNAHreFfvTAXavZGka-A3WWvPuH0sMjGUCo48/edit# to work properly. See #16420 (comment). As discussed on the ticket, this is not a very elegant solution. But we don't have the rule graph introspection mechanism necessary to automagically figure out what type each tool is, and that also seems brittle. As explained in this PR's updated docs, plugin authors don't have to technically make this change. [ci skip-rust] [ci skip-build-wheels]

@huonw

Improves upon #16420. @huonw wisely suggested we avoid using `hash` for constraints files because it makes Git diffs and merge conflicts much worse. python-poetry/poetry#496 rust-lang/cargo#7070 [ci skip-rust] [ci skip-build-wheels]

Constraints files are extremely useful for pinning problematic transitive deps. We abused constraints files early on as a poor-man's lockfile via `[python].requirement_constraints`. While we want to deprecate that in favor of `[python].resolves`, it is useful to allow constraint files for resolves. Specifically, each resolve should be allowed its own constraints file. For example, if resolve A uses Django 2, it may need to pin different problematic transitive deps of Django than resolve B using Django 3. -- This PR is the first out of a project to allow specifying config for lockfiles on a per-resolve basis, rather than globally for all resolves: https://docs.google.com/document/d/1HAvpSNvNAHreFfvTAXavZGka-A3WWvPuH0sMjGUCo48/edit#. -- This requires us now tracking `constraints_file_hash` in the lockfile metadata, so that when the user changes a constraints file, we know to regenerate the lock. Note that this means we have a V3 of lockfile metadata. It is okay to keep using V2 -- the next time people run `generate-lockfiles`, they will be bumped. [ci skip-rust] [ci skip-build-wheels]

…bclasses (pantsbuild#16449) Closes pantsbuild#16443, which is necessary for the per-resolve config project from https://docs.google.com/document/d/1HAvpSNvNAHreFfvTAXavZGka-A3WWvPuH0sMjGUCo48/edit# to work properly. See pantsbuild#16420 (comment). As discussed on the ticket, this is not a very elegant solution. But we don't have the rule graph introspection mechanism necessary to automagically figure out what type each tool is, and that also seems brittle. As explained in this PR's updated docs, plugin authors don't have to technically make this change. [ci skip-rust] [ci skip-build-wheels]

@huonw

…sbuild#16469) Improves upon pantsbuild#16420. @huonw wisely suggested we avoid using `hash` for constraints files because it makes Git diffs and merge conflicts much worse. python-poetry/poetry#496 rust-lang/cargo#7070 [ci skip-rust] [ci skip-build-wheels]

Add [python].resolves_to_constraints_file

9a2b1f1

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

Eric-Arellano added the category:new feature label Aug 5, 2022

Eric-Arellano requested review from jsirois and benjyw August 5, 2022 00:04

jsirois reviewed Aug 5, 2022

View reviewed changes

src/python/pants/backend/python/goals/lockfile_test.py Show resolved Hide resolved

Eric-Arellano mentioned this pull request Aug 5, 2022

Change --python-no-binary to be per-resolve #16421

Closed

Track constraints file in lockfile metadata

1c24cc1

[ci skip-rust] [ci skip-build-wheels]

Eric-Arellano commented Aug 8, 2022

View reviewed changes

Eric-Arellano requested review from jsirois and chrisjrn August 8, 2022 18:53

Fix test

d3ed245

# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]

jsirois approved these changes Aug 8, 2022

View reviewed changes

Eric-Arellano mentioned this pull request Aug 8, 2022

Distinguish between Python vs JVM tool lockfile requests #16443

Closed

Eric-Arellano enabled auto-merge (squash) August 8, 2022 19:48

Eric-Arellano disabled auto-merge August 8, 2022 19:57

Eric-Arellano mentioned this pull request Aug 9, 2022

GenerateToolLockfileSentinel encouraged to use language-specific subclasses #16449

Merged

benjyw approved these changes Aug 9, 2022

View reviewed changes

Eric-Arellano merged commit fda17c9 into pantsbuild:main Aug 9, 2022

Eric-Arellano deleted the constraints-file-with-locks branch August 9, 2022 02:16

Eric-Arellano mentioned this pull request Aug 10, 2022

Use a list of requirement constraints for lockfile invalidation #16469

Merged

Eric-Arellano mentioned this pull request Oct 24, 2022

Add [python].resolves_to_target_os to tell Pex whether to generate lockfile for macOS, Linux, or both #17276

Open

huonw mentioned this pull request Jul 12, 2023

Prepare 2.18.0a0 huonw/pants#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `[python].resolves_to_constraints_file` #16420

Add `[python].resolves_to_constraints_file` #16420

Eric-Arellano commented Aug 5, 2022 •

edited

jsirois commented Aug 5, 2022

jsirois left a comment

Eric-Arellano Aug 8, 2022

Eric-Arellano Aug 8, 2022

Eric-Arellano Aug 8, 2022

jsirois Aug 8, 2022

Eric-Arellano Aug 8, 2022

jsirois Aug 8, 2022

Eric-Arellano Aug 8, 2022

jsirois Aug 8, 2022

jsirois Aug 8, 2022

Eric-Arellano Aug 8, 2022

jsirois Aug 8, 2022

benjyw Aug 9, 2022

benjyw Aug 9, 2022

benjyw Aug 9, 2022

Eric-Arellano Aug 9, 2022

benjyw Aug 9, 2022

		"""Configuration from `[python]` that impacts how the resolve is created."""

		constraints_file: tuple[Digest, FileEntry] \| None

Add [python].resolves_to_constraints_file #16420

Add [python].resolves_to_constraints_file #16420

Conversation

Eric-Arellano commented Aug 5, 2022 • edited

jsirois commented Aug 5, 2022

jsirois left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `[python].resolves_to_constraints_file` #16420

Add `[python].resolves_to_constraints_file` #16420

Eric-Arellano commented Aug 5, 2022 •

edited