Enable pantsd in Travis #7440

blorente · 2019-03-26T13:51:10Z

This is a WIP to iterate on the blockers to enable pantsd in Travis.

blorente · 2019-03-26T13:58:35Z

Removing the Draft qualifier to trigger travis, but this is still not ready for review.

tests/python/pants_test/backend/project_info/tasks/test_idea_plugin_integration.py

Also enable shutdown-after-use. Add a decorator to disable pantsd in integration tests that fail under pantsd. Blacklist a few more tests (which appear to be flaky, perhaps... gulp), and mark pantsd integration tests as hermetic. Only apply pantsd on integration test shards. Blacklist some other stuff Fix test_stats_local_json_file Add more reasoning for blacklisted tests Skip some more tests Add some colour to the hermetic errors Also skip test_v2_list Hacky way of making hermetic tests run Import fixes Skip some weird interactions with the hermetic hacks Pointless Commit to trigger travis Make stuff non-hermetic Explicitly disable pantsd in the test coordinator Add some color to some of the failures

… setup

blorente · 2019-04-09T19:03:03Z

I'm pretty happy with the state of this, removing the WIP tag, will blacklist more tomorrow if travis is not green. Some things to note:

I tried to separate the concepts of "the test coordinator is running pantsd" and "the integration tests spawn pantsd". This PR implements the latter, but whenever the former is necessary (we turn pantsd on by default), it should be isolated and should not require changes to this code.
The way I read the env var TRAVIS_AND_PANTS_INTEGRATION_TESTS_SHOULD_HAVE_PANTSD is a bit wonky, I'm totally open to suggestions.

stuhood

Thanks!

tests/python/pants_test/pants_run_integration_test.py

stuhood · 2019-04-09T19:47:16Z

tests/python/pants_test/pants_run_integration_test.py

@@ -284,6 +299,13 @@ def run_pants_with_workdir_without_waiting(self, command, workdir, config=None,
                   '--cache-bootstrap-read', '--cache-bootstrap-write'
                   ])

+    if self.should_configure_pantsd():
+      args.append("--enable-pantsd=True")
+      args.append("--no-shutdown-pantsd-after-run")


I thought that our intent right now was to actually enable --shutdown-pantsd-after-run... but I think that you also determined that because of the temporary workdir, pantsd will shutdown after each run anyway?

So we don't want to shutdown after every run inside the tests, because some of them rely on pantsd living run after run (although they don't rely on this codepath). We do want to set it (probably) in the overpants, the test coordinator, via travis env var. But this should be done at the same time as we enable pantsd by default, and shouldn't affect these changes at all.

I would argue that those tests should be marked hermetic and explicitly set the relevant values. But this works too!

illicitonion

Looks great :) Thanks!

tests/python/pants_test/pantsd/pantsd_integration_test_base.py

Eric-Arellano

Exciting!

build-support/travis/travis.yml.mustache

tests/python/pants_test/backend/jvm/tasks/jvm_compile/java/test_cache_compile_integration.py

tests/python/pants_test/engine/legacy/test_build_ignore_integration.py

tests/python/pants_test/pants_run_integration_test.py

Eric-Arellano

Thanks for the renames! Beyond fixing the stage entry, this lgtm.

build-support/travis/travis.yml.mustache

stuhood · 2019-04-15T23:36:44Z

Hooray! Thanks @blorente .

**Problem** The cron job was not configured to bootstrap the 3.6 version of pants, and therefore when we added the "3.6 + pantsd" shards in #7440, these don't work. **Solution** To add the 3.6 bootstrapping shards to the cron job. **Result** The cron job should now work, maybe?

**Problem** `travis-ci.sh` was removed in ef7f664 (#7548), but the changes to the cron shard in #7440 were not rebased to reflect this update.

blorente marked this pull request as ready for review March 26, 2019 13:58

blorente force-pushed the blorente/pantsd-in-travis branch 4 times, most recently from ccb8c3e to 56ecae1 Compare March 28, 2019 17:52

stuhood mentioned this pull request Apr 3, 2019

[Pantsd] Enable pantsd in Travis #7320

Closed

11 tasks

stuhood reviewed Apr 3, 2019

View reviewed changes

tests/python/pants_test/backend/project_info/tasks/test_idea_plugin_integration.py Outdated Show resolved Hide resolved

blorente force-pushed the blorente/pantsd-in-travis branch from d9818fb to b4f01ff Compare April 8, 2019 14:19

stuhood and others added 6 commits April 9, 2019 12:02

Whitelist test_idea_plugin_integration

a22de22

Startup point for a better solution

2f7170e

Un-blacklist some tests

6265447

Un-blacklist some tests

4bbb39e

Move all blacklisting to a classmethod

c803106

blorente force-pushed the blorente/pantsd-in-travis branch from b4f01ff to c803106 Compare April 9, 2019 14:19

Borja Lorente added 5 commits April 9, 2019 15:34

Add very specific env var to allow travis to interact with the daemon…

157902f

… setup

Remove leftover annotation

1302ccc

Modify integration shards to also include pantsd

65932d8

Actually read the env var

991f7c4

Actual structure for shards

0492f1b

blorente force-pushed the blorente/pantsd-in-travis branch from 0c1ae1c to 0492f1b Compare April 9, 2019 18:57

blorente changed the title ~~[WIP] Enable pantsd in Travis~~ Enable pantsd in Travis Apr 9, 2019

blorente requested review from illicitonion, cosmicexplorer, Eric-Arellano, benjyw and stuhood April 9, 2019 19:03

stuhood approved these changes Apr 9, 2019

View reviewed changes

blorente mentioned this pull request Apr 10, 2019

[pantsd] Make PantsDaemonIntegrationTestBase hermetic. #7525

Closed

Don't duplicate hermetic()'s behaviour

b00235d

illicitonion approved these changes Apr 10, 2019

View reviewed changes

tests/python/pants_test/pantsd/pantsd_integration_test_base.py Outdated Show resolved Hide resolved

Borja Lorente added 2 commits April 10, 2019 12:07

Blacklist another test

609cacd

Re-add deleted line

a3a9c5a

Eric-Arellano reviewed Apr 10, 2019

View reviewed changes

Address comments

18f59e2

Eric-Arellano reviewed Apr 11, 2019

View reviewed changes

build-support/travis/travis.yml.mustache Outdated Show resolved Hide resolved

build-support/travis/travis.yml.mustache Outdated Show resolved Hide resolved

Make test_cron not a list item

4c6a329

blorente merged commit 08fe94a into pantsbuild:master Apr 15, 2019

blorente mentioned this pull request Apr 16, 2019

Bootstrap Python 3.6 shards in cron. #7579

Merged

blorente added a commit that referenced this pull request Apr 17, 2019

Use ci.sh instead of travis-ci.sh (#7582)

ff01126

**Problem** `travis-ci.sh` was removed in ef7f664 (#7548), but the changes to the cron shard in #7440 were not rebased to reflect this update.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable pantsd in Travis #7440

Enable pantsd in Travis #7440

blorente commented Mar 26, 2019

blorente commented Mar 26, 2019

blorente commented Apr 9, 2019

stuhood left a comment

stuhood Apr 9, 2019

blorente Apr 9, 2019

stuhood Apr 9, 2019 •

edited

Loading

illicitonion left a comment

Eric-Arellano left a comment

Eric-Arellano left a comment

stuhood commented Apr 15, 2019

Enable pantsd in Travis #7440

Enable pantsd in Travis #7440

Conversation

blorente commented Mar 26, 2019

blorente commented Mar 26, 2019

blorente commented Apr 9, 2019

stuhood left a comment

Choose a reason for hiding this comment

stuhood Apr 9, 2019

Choose a reason for hiding this comment

blorente Apr 9, 2019

Choose a reason for hiding this comment

stuhood Apr 9, 2019 • edited Loading

Choose a reason for hiding this comment

illicitonion left a comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

stuhood commented Apr 15, 2019

stuhood Apr 9, 2019 •

edited

Loading