New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document Pants features for Organizations #5673

Merged
merged 4 commits into from Aug 10, 2018

Conversation

Projects
None yet
4 participants
@traviscrawford
Copy link
Member

traviscrawford commented Apr 6, 2018

Problem

Organizations that are interested in Pants do not have a simple way of learning what features are available for larger organizations using Pants, nor a place to learn best-practices for how to setup their site.

Solution

Here we create a new docs page focused on organizations, especially targeted at Developer Experience or equivalent teams who are setting up Pants for their org.

Result

Organizations should be better able to setup Pants at their site as they have a simple way to understand which features to setup.

@traviscrawford

This comment has been minimized.

Copy link
Member

traviscrawford commented Apr 6, 2018

I'd love to use this or a future PR to document other features targeted at Organizations using Pants, such as the build cache. If you have any ideas to include in this or future PRs please let me know. I think there's value in this as-is, but it's also a prompt for a conversation about how best to do some of these things (e.g.: recommended CI script).

@traviscrawford traviscrawford force-pushed the traviscrawford:travis/org-docs branch from f9ae6c8 to 8eea2ee Apr 6, 2018

@wisechengyi
Copy link
Contributor

wisechengyi left a comment

Thanks @traviscrawford! Looks good with a few nits

# Pants for Organisations

Pants works well for organizations with a large number of developers working in a multi-tenant repo.
Typically, repos becomes slower over time as more source and test files are added because most

This comment has been minimized.

@wisechengyi

wisechengyi Apr 8, 2018

Contributor

becomes -> become

Now consider a backend developer who only works on the backend service. When building the backend
service, only the code required by the backend service is built, regardless of what other code
is in the repo. The ability to build "slices" of the repo fundamental to enabling multiple users
and teams to work together in a single repo. Note how only the backend code would be build when

This comment has been minimized.

@wisechengyi

wisechengyi Apr 8, 2018

Contributor

build -> built


Now consider a backend developer who only works on the backend service. When building the backend
service, only the code required by the backend service is built, regardless of what other code
is in the repo. The ability to build "slices" of the repo fundamental to enabling multiple users

This comment has been minimized.

@wisechengyi

wisechengyi Apr 8, 2018

Contributor

The ability to build "slices" of the repo is fundamental

Most organizations require test pass before merging a change to master. Pants enables fast CI by
only running tests _affected_ by a change. That is, only targets that transitively depend on a
changed target have their tests run. In practice, a change to `README.md` might not trigger *any*
tests, a change to a leaf-node in the dependency graph would only trigger it's direct tests, and

This comment has been minimized.

@wisechengyi

wisechengyi Apr 8, 2018

Contributor

leaf-node might be a little overloaded these days because people have different conventions. How about

top-level node such as `org.pantsbuild.fe.service`

to keep the language consistent with low-level node?

@stuhood

stuhood approved these changes Apr 9, 2018

Copy link
Member

stuhood left a comment

Thanks a lot Travis!


Consider a frontend developer that only works on the frontend service. With most build systems, the
developer would need to compile everything every time, slowing them down over time as more and more
code is added to repo. With Pants, the frontend developer only needs to compile the code directly

This comment has been minimized.

@stuhood

stuhood Apr 9, 2018

Member

added to the repo

set -o
set -e
# Do not use the build cache when running tests to ensure no historical cruft

This comment has been minimized.

@stuhood

stuhood Apr 9, 2018

Member

This flag does not disable the buildcache: rather, it disables zinc incremental compilation, which is/has-always-been error prone compared to clean builds.

In fact, by disabling incremental compilation, you actually increase usage of the buildcache, because pants does not cache zinc incremental builds (for the reasons above), but it does cache clean builds.

This comment has been minimized.

@traviscrawford

traviscrawford Apr 9, 2018

Member

Thanks for the clarification, will update.

Do you have any thoughts on recommending ./pants test ::, and how it pairs with --test-junit-test-shard as how to CI? This would be cleanest, though in practice I don't do that anymore because there were some code that didn't build together, which is it's own issue.

This comment has been minimized.

@stuhood

stuhood Apr 10, 2018

Member

Do you have any thoughts on recommending ./pants test ::, and how it pairs with --test-junit-test-shard as how to CI?

In practice (when an organization becomes large enough) running ./pants test :: puts too tight a constraint on which things are compatible.

Our goal with work on remoting is to make ./pants test :: feasible again, and to do what you would expect (ie: enough ivy/coursier resolves to build all roots with a stable set of 3rdparty deps), but it at least 6-9 months out.

Travis Crawford added some commits Apr 10, 2018

Travis Crawford
@traviscrawford

This comment has been minimized.

Copy link
Member

traviscrawford commented Apr 10, 2018

Updated, thanks for the suggestions!

export PANTS_COMPILE_ZINC_INCREMENTAL=false
changed=$(./pants --changed-parent=origin/master list)
dependees=$(./pants dependees --dependees-transitive --dependees-closed $changed)

This comment has been minimized.

@stuhood

stuhood Apr 10, 2018

Member

You can merge the ./pants --changed=.. list, dependees, and filter by doing:

./pants --changed-parent=origin/master --changed-include-dependees=transitive filter --type=-jvm_binary

...and then finish with minimize.

This comment has been minimized.

@stuhood

stuhood Apr 10, 2018

Member

(and minimizing last is useful, because if something was only covered by a jvm_binary, you would still want to compile it in CI).

@baroquebobcat
Copy link
Contributor

baroquebobcat left a comment

Nice! Thanks.

minimized=$(./pants minimize $dependees)
./pants filter --filter-type=-jvm_binary $minimized | sort > minimized.txt
for target in $(cat minimized.txt); do
./pants test $target

This comment has been minimized.

@stuhood

stuhood Apr 10, 2018

Member

You might also consider suggesting ./pants lint test $target

@stuhood stuhood force-pushed the pantsbuild:master branch from b6bb42d to 9e2fdb5 May 11, 2018

@stuhood stuhood merged commit 4c31167 into pantsbuild:master Aug 10, 2018

1 check failed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
@stuhood

This comment has been minimized.

Copy link
Member

stuhood commented Aug 10, 2018

The relevant shards in travis were green... went ahead and merged.

CMLivingston pushed a commit to CMLivingston/pants that referenced this pull request Aug 27, 2018

Document Pants features for Organizations (pantsbuild#5673)
### Problem

Organizations that are interested in Pants do not have a simple way of learning what features are available for larger organizations using Pants, nor a place to learn best-practices for how to setup their site.

### Solution

Here we create a new docs page focused on organizations, especially targeted at Developer Experience or equivalent teams who are setting up Pants for their org.

### Result

Organizations should be better able to setup Pants at their site as they have a simple way to understand which features to setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment