Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flaky product graph construction: Invalid requirement, parse error at "u''" #6861

Open
wisechengyi opened this issue Dec 4, 2018 · 18 comments

Comments

Projects
None yet
7 participants
@wisechengyi
Copy link
Contributor

commented Dec 4, 2018

https://api.travis-ci.org/v3/job/463150773/log.txt

�[Ktravis_fold:start:BackendTests
�[K�[34m[=== 05:35 Running internal backend python tests ===]�[0m
timestamp: 2018-12-04T04:51:24.356140
Exception caught: (pants.build_graph.address_lookup_error.AddressLookupError)
  File ".bootstrap/_pex/pex.py", line 349, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File ".bootstrap/_pex/pex.py", line 281, in _wrap_coverage
    return runner(*args)
  File ".bootstrap/_pex/pex.py", line 312, in _wrap_profiling
    return runner(*args)
  File ".bootstrap/_pex/pex.py", line 394, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File ".bootstrap/_pex/pex.py", line 506, in execute_entry
    return runner(entry_point)
  File ".bootstrap/_pex/pex.py", line 528, in execute_pkg_resources
    return runner()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 71, in main
    PantsLoader.run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 67, in run
    cls.load_and_execute(entrypoint)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_loader.py", line 60, in load_and_execute
    entrypoint_main()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/pants_runner.py", line 62, in run
    return runner.run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 159, in run
    self._run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 227, in _run
    goal_runner_result = self._maybe_run_v1(run_tracker, reporting)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/local_pants_runner.py", line 176, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/goal_runner.py", line 125, in create
    goals, context = self._setup_context()
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/bin/goal_runner.py", line 95, in _setup_context
    self._root_dir
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/init/engine_initializer.py", line 226, in create_build_graph
    for _ in graph.inject_roots_closure(target_roots):
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 217, in inject_roots_closure
    for address in self._inject_specs(target_roots.specs):
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 270, in _inject_specs
    subjects)
  File "/opt/pyenv/versions/2.7.14/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/legacy/graph.py", line 237, in _resolve_context
    'Build graph construction failed: {} {}'.format(type(e).__name__, str(e))

Exception message: Build graph construction failed: ExecutionError 1 Exception encountered:
Computing Select(Specs(dependencies<=tuple>=(DescendantAddresses(directory=u'pants-plugins/tests/python'),), matcher<=SpecsMatcher>=SpecsMatcher(tags<=tuple>=(), exclude_patterns<=tuple>=())), =TransitiveHydratedTargets)
  Computing Task(transitive_hydrated_targets, Specs(dependencies<=tuple>=(DescendantAddresses(directory=u'pants-plugins/tests/python'),), matcher<=SpecsMatcher>=SpecsMatcher(tags<=tuple>=(), exclude_patterns<=tuple>=())), =TransitiveHydratedTargets, true)
    Computing Task(transitive_hydrated_target, pants-plugins/tests/python/internal_backend_test/sitegen:sitegen, =TransitiveHydratedTarget, true)
      Computing Task(transitive_hydrated_target, 3rdparty/python:future, =TransitiveHydratedTarget, true)
        Computing Task(hydrate_target, 3rdparty/python:future, =HydratedTarget, true)
          Computing Task(hydrate_struct, 3rdparty/python:future, =(symbol table types), true)
            Computing Task(resolve_unhydrated_struct, 3rdparty/python:future, =UnhydratedStruct, true)
              Computing Task(parse_address_family, Dir(path=3rdparty/python), =AddressFamily, true)
                Throw(Failed to parse 3rdparty/python/BUILD:
Invalid requirement, parse error at "u''")
                  Traceback (most recent call last):
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/native.py", line 523, in extern_generator_send
                      res = c.from_value(func[0]).send(c.from_value(arg[0]))
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/build_files.py", line 61, in parse_address_family
                      address_mapper.parser))
                    File "/home/travis/.pex/code/65c2938b2de17351c60a770eb88d6bc1f1272701/pants/engine/mapper.py", line 55, in parse
                      raise MappingError('Failed to parse {}:\n{}'.format(filepath, e))
                  MappingError: Failed to parse 3rdparty/python/BUILD:
                  Invalid requirement, parse error at "u''"
@jsirois

This comment has been minimized.

Copy link
Member

commented Dec 5, 2018

Noting that the linked log does not correspond to the pasted error.

The pasted error happens before any test runs during the build graph setup for the whole pants run of:

./pants.pex test.pytest pants-plugins/tests/python:: -- -v --duration=3

This would seem to point not to a flaky test, but a flaky bit of the production system which is perhaps more alarming. Re-labeling as a bug.

@jsirois jsirois added bug and removed flaky-test labels Dec 5, 2018

@jsirois jsirois changed the title Flaky test: Invalid requirement, parse error at "u''" Flaky production graph construction: Invalid requirement, parse error at "u''" Dec 5, 2018

@stuhood stuhood changed the title Flaky production graph construction: Invalid requirement, parse error at "u''" Flaky product graph construction: Invalid requirement, parse error at "u''" Jan 3, 2019

@stuhood stuhood added the flaky-test label Jan 3, 2019

@stuhood

This comment has been minimized.

Copy link
Member

commented Jan 9, 2019

The actual underlying exception stacktrace is hidden by the raised MappingError. Trying out a nice and dumb repro here: #7052.

@stuhood

This comment has been minimized.

Copy link
Member

commented Jan 24, 2019

Seen again in master.

Related to the dumb repro attempt: it's possible that wrapping-and-re-raising as MappingError is no longer necessary, and is now just obfuscating errors like these. I'm going to let the CI run on #7052 run to completion, and see whether it fails any tests. If not, will clean it up to land.

@yaup

This comment has been minimized.

Copy link
Contributor

commented Mar 13, 2019

This is happening to us as well, and seems to be more common in the recent weeks (3 in the last 7 days, which is why I started paying attention), though I don't have more specific statistics.

Have yet to find a pattern on which commands cause them (the last three that I saw were ./pants list --changed... and ./pants binary (twice)). The only pattern I've found so far is that these happen on our build boxes (which are ubuntu EC2 instances or containers) while I haven't seen any reports on local dev machines (which are osx)

@cosmicexplorer

This comment has been minimized.

Copy link
Contributor

commented Mar 13, 2019

If this is something that can be reliably recognized in some way (we can do some hacky thing looking at the exception message), we could consider adding an option to either provide a specific error message saying "THIS IS A FLAKY ERROR, SEE #6861", or to automatically restart after dumping a lot of warnings (and only retry once, probably).

@stuhood

This comment has been minimized.

Copy link
Member

commented Apr 5, 2019

Hm... have we had a recent repro of this one? Wondering whether #7379 was related somehow.

@yaup

This comment has been minimized.

Copy link
Contributor

commented Apr 5, 2019

I don't think I've seen this recently, but I'm not sure how long it's been since I last saw this, probably a week or two?

edit: derp we haven't changed anything regarding pants versions on our side so I'm not sure why I thought anything is supposed to be different. Saw it again today but it is interesting that there were days where it happens a lot and days where it just doesn't.

@haneul

This comment has been minimized.

Copy link

commented Apr 16, 2019

Is it fixed in recent versions? I hit this issue in 0.14.0

@jsirois

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Seen on master on 5/21/2019 in 'Unit tests (Py2.7 PEX)' shard:

$ ./build-support/bin/ci.sh -2lp
[=== 00:00 CI BEGINS ===]
[=== 00:00 Setting interpreter constraints to ['CPython==2.7.*'] ===]
[=== 00:00 Running internal backend python tests ===]
timestamp: 2019-05-22T04:26:01.002407
Exception caught: (pants.build_graph.address_lookup_error.AddressLookupError)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 352, in execute
    exit_code = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 284, in _wrap_coverage
    return runner(*args)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 315, in _wrap_profiling
    return runner(*args)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 397, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 495, in execute_entry
    return runner(entry_point)
  File "/home/travis/build/pantsbuild/pants/pants.pex/.bootstrap/pex/pex.py", line 510, in execute_pkg_resources
    return runner()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 85, in main
    PantsLoader.run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 81, in run
    cls.load_and_execute(entrypoint)
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_loader.py", line 74, in load_and_execute
    entrypoint_main()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/pants_runner.py", line 88, in run
    return runner.run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 231, in run
    self._run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 292, in _run
    goal_runner_result = self._maybe_run_v1()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/local_pants_runner.py", line 259, in _maybe_run_v1
    return goal_runner_factory.create().run()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/goal_runner.py", line 126, in create
    goals, context = self._setup_context()
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/bin/goal_runner.py", line 96, in _setup_context
    self._root_dir
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/init/engine_initializer.py", line 226, in create_build_graph
    for _ in graph.inject_roots_closure(target_roots):
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 195, in inject_roots_closure
    for address in self._inject_specs(target_roots.specs):
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 247, in _inject_specs
    [specs])
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/legacy/graph.py", line 214, in _resolve_context
    'Build graph construction failed: {} {}'.format(type(e).__name__, str(e))
Exception message: Build graph construction failed: ExecutionError 1 Exception encountered:
Computing Select(Specs(dependencies<Exactly(tuple)>=(DescendantAddresses(directory=u'pants-plugins/src/python'), DescendantAddresses(directory=u'pants-plugins/tests/python')), matcher<Exactly(SpecsMatcher)>=SpecsMatcher(tags<Exactly(tuple)>=(), exclude_patterns<Exactly(tuple)>=())), TransitiveHydratedTargets)
  Computing Task(transitive_hydrated_targets(), Specs(dependencies<Exactly(tuple)>=(DescendantAddresses(directory=u'pants-plugins/src/python'), DescendantAddresses(directory=u'pants-plugins/tests/python')), matcher<Exactly(SpecsMatcher)>=SpecsMatcher(tags<Exactly(tuple)>=(), exclude_patterns<Exactly(tuple)>=())), TransitiveHydratedTargets, true)
    Computing Task(transitive_hydrated_target(), pants-plugins/src/python/internal_backend/utilities:plugin, TransitiveHydratedTarget, true)
      Computing Task(transitive_hydrated_target(), 3rdparty/python/twitter/commons:twitter.common.collections, TransitiveHydratedTarget, true)
        Computing Task(hydrate_target(), 3rdparty/python/twitter/commons:twitter.common.collections, HydratedTarget, true)
          Computing Task(hydrate_struct(), 3rdparty/python/twitter/commons:twitter.common.collections, HydratedStruct, true)
            Computing Task(parse_address_family(), Dir(path=3rdparty/python/twitter/commons), AddressFamily, true)
              Throw(Failed to parse 3rdparty/python/twitter/commons/BUILD:
Invalid requirement, parse error at "u''")
                Traceback (most recent call last):
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/native.py", line 428, in extern_generator_send
                    res = c.from_value(func[0]).send(c.from_value(arg[0]))
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/build_files.py", line 61, in parse_address_family
                    address_mapper.parser))
                  File "/home/travis/.pex/code/dd7cf54dd4e3e7da08b555be754a9a43094a3ba6/pants/engine/mapper.py", line 55, in parse
                    raise MappingError('Failed to parse {}:\n{}'.format(filepath, e))
                MappingError: Failed to parse 3rdparty/python/twitter/commons/BUILD:
                Invalid requirement, parse error at "u''"
Internal backend python test failure
The command "./build-support/bin/ci.sh -2lp" exited with 1.
@jsirois

This comment has been minimized.

Copy link
Member

commented May 27, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2019

@yaup @haneul could you both please check if your pants.ini has the entry plugins? You can check either by opening it up, or cat pants.ini | grep 'plugins'.

Twitter removed their plugins entry today and as a result saw a crazy amount of this particular failure happening. We're unsure if we never had seen the error before, but we at least had never seen it that severely. I'm wondering if a missing plugins is the cause of this all..your help checking would be highly appreciated.

@yaup

This comment has been minimized.

Copy link
Contributor

commented Jun 12, 2019

@Eric-Arellano we don't have any plugins installed via the plugins directive right now (we have some custom ones we built that's added via backend_packages.

@wisechengyi

This comment has been minimized.

Copy link
Contributor Author

commented Jun 21, 2019

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

@yaup also, what Pants version are you using? Are you using the ./pants script we distribute here, and if so, how recently have you updated it? I'm trying to see if you've encountered this problem when running Pants with Python 3, as every reported usage so far is when running with Python 2.

Another way to check this is if you have ever seen the error without u"", but just "".

@yaup

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

@Eric-Arellano We are using pants 1.14. I haven't updated the ./pants script since taking over about a year ago, so it should still be running on python 2.

I don't recall seeing just ""

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

Thanks for the quick response! That means that, yes, you are running Pants with Python 2.

I suspect this won't be an issue once you upgrade to >= 1.15.0 and upgrade the ./pants script (see https://www.pantsbuild.org/notes-1.15.x.html#heading__101 for instructions on how to update). I'm going to try to reproduce this for Twitter's codebase right now, which now runs Pants with Python 3 and doesn't have the plugins entry in pants.ini, and will see if I can get this to reproduce.

@yaup

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

Hmmm I see. I had 1.15 upgrades in the pipelines when I handed this work off recently, so I suppose we'll see how that goes. Thanks!

@Eric-Arellano

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

As a temporary workaround, I believe that you can add plugins: [] to your GLOBAL section of pants.ini. I'm not sure if that will actually fix it, but it seems very likely and it won't hurt anything beyond an extra line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.