refactor command line target spec resolution and check that all target roots exist #6480

cosmicexplorer · 2018-09-10T12:21:38Z

Problem

This should fail regardless of the order of the command-line target specs (the target named a does not exist):

> ./pants test tests/python/pants_test/option:a
# ...
ResolveError: "a" was not found in namespace "tests/python/pants_test/option". Did you mean one of:
          :options_integration
          :testing
> ./pants test tests/python/pants_test/option:{testing,a}
# ...
 ===== 158 passed, 1 xfailed in 2.77 seconds ======
                     
                   tests/python/pants_test/option:testing                                          .....   SUCCESS
23:25:48 00:12     [junit]
23:25:48 00:12     [go]
23:25:48 00:12     [node]
               Waiting for background workers to finish.
23:25:49 00:13   [complete]
               SUCCESS

Solution

Move AddressFamily and Address resolution logic into the Spec class and its subclasses instead of doing a massive if chain.
Introduce the _MappedSpecs datatype holding the target Specs, and the AddressFamilys covering the directories indicated by the Specs.
Move all of the address resolution logic into _MappedSpecs for clarity, and replace all if type(obj) is SingleAddress chains with abstract methods implemented in Spec subclasses.
Add test_fails_on_nonexistent_specs() to test the issue described in the Problem section.

stuhood

Thanks!

stuhood · 2018-09-10T15:41:36Z

src/python/pants/base/specs.py

+
+  class AddressResolutionError(Exception): pass
+
+  def all_address_target_pairs(self, address_families):


Rather than overriding concrete methods, it would be preferable to extract this as an optional helper as you did with address_families_for_dir, that some abstract method can optionally call.

This is great! Done in c4d2edc!

stuhood · 2018-09-10T15:47:36Z

src/python/pants/engine/build_files.py

-              addresses.append(a)
-              included.add(a)
-    return matched
+  yield MappedSpecs(address_families, specs)


While splitting this rule into two makes testing easier, it's basically pure overhead at runtime.

The method is called ~once per run though, so not a big deal. But if it's possible to remove the testing boilerplate by just adding a test helper method instead, that would be preferable.

The rules were merged back together in a694488, and a helper method was created in e47a710!

stuhood · 2018-09-10T15:51:33Z

src/python/pants/engine/build_files.py

+  def __new__(cls, address_families, specs):
+    return super(MappedSpecs, cls).__new__(cls, tuple(address_families), specs)
+
+  @memoized_property


I don't think there is really any advantage to memoizing these on the class, but the downside is that all of these memoized properties will survive on the object between runs, because the MappedSpecs object is itself memoized.

If you were to undo the @rule split, it would be a non-issue, because the MappedSpecs object would be a local-only construct. But as it stands, you're holding on to a lot of stuff that could/should be ephemeral.

I'm not too attached to the @rule split (lol). I'll try merging them back, I don't think it's necessary for this to work.

Btw, this was a very helpful explanation re: memoization. Thanks.

The rules have been merged back together into one, maintaining the MappedSpecs object, but not having it anywhere inside the rule graph (and therefore memoized/etc).

Thanks, that's great. If you need to run it through CI again, could rename it to _MappedSpecs to highlight that it's not public.

cosmicexplorer · 2018-09-11T08:35:40Z

There's one remaining non-spurious CI failure, which I will address tomorrow.

stuhood

Thanks, looks good! Just nits.

stuhood · 2018-09-11T14:59:23Z

src/python/pants/engine/build_files.py

+  def __new__(cls, address_families, specs):
+    return super(MappedSpecs, cls).__new__(cls, tuple(address_families), specs)
+
+  @memoized_property


Thanks, that's great. If you need to run it through CI again, could rename it to _MappedSpecs to highlight that it's not public.

stuhood · 2018-09-11T15:01:34Z

tests/python/pants_test/engine/test_build_files.py

+
+  def setUp(self):
+    self._default_address_mapper = AddressMapper(JsonParser(TestTable()))
+    self._default_snapshot = Snapshot(DirectoryDigest('xx', 2),


Rather than making these properties, prefer methods: that makes them easier to template later: ie self._snapshot() vs self._snapshot(with_more_stuff=True).

Done in 5801d95! Let me know if these need any explanatory comments or docstrings.

Github isn't letting me reply to your other comment, but I also made _MappedSpecs private in 11bd1c6.

…ed_specs" This reverts commit 81db14f7f38bb83ebfcb2f0f48e2a9f03dab2791.

…hods added to Spec

…atements

cosmicexplorer · 2018-09-12T03:02:12Z

The remaining failure was because we were dropping all the matching addresses into a frozenset and of course totally screwing up the order (which matters for e.g. calculating paths). Changing that to an OrderedSet made the test pass. This also shows that the paths integration testing does indeed cover more than the unit tests do, so I expanded the TODO in that file to point to this PR.

Eric-Arellano

Very cool! Thanks for this!

src/python/pants/base/specs.py

Eric-Arellano · 2018-09-12T15:58:47Z

src/python/pants/engine/build_files.py

@@ -201,108 +202,99 @@ def _hydrate(item_type, spec_path, **kwargs):
  return item


+class _MappedSpecs(datatype([
+    ('address_families', tuple),


We don’t have a way to say a tuple of what, do we?

With type hints it’s Tuple[Foo, Foo, Bar]

At a minimum, would be helpful to add a comment for what’s in the tuple.

Not yet about a tuple of what -- I would love to modify Collection in objects.py to make these more clear, and to type check the elements of the collection.

(for now I'll add a comment)

Added a comment in e991a34 -- you'll note it's List[AddressFamily], the tuple part is an implementation detail -- lists aren't hashable, and this class used to be the output of an @rule, so it was hashed.

Now that I think of it, I should just change the type to list.

Ok, I just tried that, and it turns out that @memoized_property is what requires inputs to be hashable, so I'm fine with leaving it as tuple, but that's an implementation detail that could maybe be fixed at some point. Because tuple is used elsewhere in the codebase for the same reason, I'm fine with this for now.

Eric-Arellano · 2018-09-12T16:00:52Z

src/python/pants/engine/build_files.py

+    return [re.compile(pattern) for pattern in set(self.specs.exclude_patterns or [])]
+
+  def _excluded_by_pattern(self, address):
+    return any(p.search(address.spec) is not None for p in self._exclude_compiled_regexps)


Nice! Very Pythonic 🐍

This was just stolen from elsewhere in the file so it is indeed pythonic but not quite my idea.

Eric-Arellano · 2018-09-12T16:05:00Z

src/python/pants/engine/build_files.py

+    try:
+      addr_families_for_spec = spec.matching_address_families(self._address_family_by_directory)
+    except Spec.AddressFamilyResolutionError as e:
+      raise ResolveError(e)


iirc, this will lose a lot of the stack trace from the original exception. We get more descriptive debugging if we use raise from syntax.

Using the Future library,

from future.utils import raise_from raise_from(ResolveError(e), e)

Although I see after writing this that you’re already passing the original exception. So maybe this isn’t necessary?

I think raise_from looks like it makes sense here and will use it.

Done in a146c99!

jsirois · 2018-09-12T16:53:42Z

Plenty of feedback / reviewers on this one so I'll bow out.

…tion errors

### Problem Due to some unexpected behaviour of `@memoized_property`, #6480 made spec parsing accidentally quadratic. See #6550 for a general overview of the problem. ### Solution Because `@memoized_property` was still a good strategy for the purposes of compiled regexes and compiled tag matchers, this extracts the inputs to those properties to a datatype `SpecsMatcher`, which will have a much smaller instance. It then removes `_MappedSpecs` by inlining its two remaining methods into `addresses_from_specs` to avoid other unnecessary memoization. ### Result For: ``` time ./pants --changed-diffspec='59d1632aa7bd4d7713d5ac18ae401adc7631e070^..59d1632' --changed-include-dependees=transitive list | wc -l ``` ``` # before #6480 real 0m7.574s user 0m8.323s sys 0m2.848s # after #6480 real 0m17.996s user 0m18.529s sys 0m3.000s # after this patch real 0m7.603s user 0m8.233s sys 0m2.786s ``` #6550 discusses auditing other usages of `@memoized_property`.

cosmicexplorer requested review from stuhood, jsirois, ity and CMLivingston September 10, 2018 12:21

stuhood reviewed Sep 10, 2018

View reviewed changes

cosmicexplorer force-pushed the fix-command-line-target-spec-validation branch from c867eff to e47a710 Compare September 11, 2018 08:07

stuhood approved these changes Sep 11, 2018

View reviewed changes

cosmicexplorer added 16 commits September 11, 2018 19:29

add failing test

54e5db8

move address resolution into Spec subclasses

24aa897

add some docstrings

5935481

flesh out more docstrings

bed5fca

fix some docstrings to have the same updated contentc

636e7fd

expand a ???

b0a0fa0

remove a final TODO

654a671

rename addresses_from_address_families -> addresses_from_mapped_specs

798fce7

only raise an AddressResolutionError for DescendantAddresses

d2d5628

Revert "rename addresses_from_address_families -> addresses_from_mapp…

33ea5a6

…ed_specs" This reverts commit 81db14f7f38bb83ebfcb2f0f48e2a9f03dab2791.

provide an abstractmethod and a classmethod default impl for both met…

c554209

…hods added to Spec

merge the two @rules back into one and revert test changes

8c8e40f

move shared test logic into a helper method

e2f9e52

add a comment

2629154

add make_glob_patterns() abstractmethod to remove more type switch st…

3c3c31a

…atements

make _MappedSpecs private

11bd1c6

cosmicexplorer force-pushed the fix-command-line-target-spec-validation branch from ea343af to 5801d95 Compare September 12, 2018 02:34

instead of default values, use factory functions for test data

5801d95

cosmicexplorer added 2 commits September 11, 2018 20:03

make paths integration test pass and explain why it is still needed

c309c7d

add self (how did i miss this? did i not run tests? a mystery)

6f51c0f

Eric-Arellano approved these changes Sep 12, 2018

View reviewed changes

jsirois removed their request for review September 12, 2018 16:53

cosmicexplorer added 3 commits September 12, 2018 23:54

add type annotation

e991a34

use raise_from to increase context where possible with address resolu…

a146c99

…tion errors

make _SingleAddressResolutionError private

d6f66ab

cosmicexplorer merged commit d6a9490 into pantsbuild:master Sep 16, 2018

stuhood mentioned this pull request Sep 24, 2018

Remove usage of @memoized_property on MappedSpecs #6551

Merged

stuhood mentioned this pull request Sep 26, 2018

AddressLookupError raised when using --tag option #6561

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor command line target spec resolution and check that all target roots exist #6480

refactor command line target spec resolution and check that all target roots exist #6480

cosmicexplorer commented Sep 10, 2018 •

edited

stuhood left a comment

stuhood Sep 10, 2018

cosmicexplorer Sep 11, 2018

stuhood Sep 10, 2018

cosmicexplorer Sep 11, 2018

stuhood Sep 10, 2018

cosmicexplorer Sep 11, 2018

cosmicexplorer Sep 11, 2018

cosmicexplorer Sep 11, 2018

stuhood Sep 11, 2018

cosmicexplorer commented Sep 11, 2018

stuhood left a comment

stuhood Sep 11, 2018

stuhood Sep 11, 2018

cosmicexplorer Sep 12, 2018

cosmicexplorer Sep 12, 2018

cosmicexplorer commented Sep 12, 2018

Eric-Arellano left a comment

Eric-Arellano Sep 12, 2018

cosmicexplorer Sep 12, 2018

cosmicexplorer Sep 12, 2018

cosmicexplorer Sep 13, 2018

cosmicexplorer Sep 13, 2018 •

edited

Eric-Arellano Sep 12, 2018

cosmicexplorer Sep 12, 2018

Eric-Arellano Sep 12, 2018

cosmicexplorer Sep 12, 2018

cosmicexplorer Sep 13, 2018

jsirois commented Sep 12, 2018


		class AddressResolutionError(Exception): pass

		def all_address_target_pairs(self, address_families):

refactor command line target spec resolution and check that all target roots exist #6480

refactor command line target spec resolution and check that all target roots exist #6480

Conversation

cosmicexplorer commented Sep 10, 2018 • edited

Problem

Solution

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Sep 11, 2018

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer commented Sep 12, 2018

Eric-Arellano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cosmicexplorer Sep 13, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois commented Sep 12, 2018

cosmicexplorer commented Sep 10, 2018 •

edited

cosmicexplorer Sep 13, 2018 •

edited