Specifying deps outside of BUILD files (aka "dep inference")

[nsaechao]

The canonical way of specifying dependencies in Pants is through the BUILD file. Currently this limits the entry to barrier for wiring in new tools/plugins that natively has its own way of tracking dependencies. Such is the case for Node.js dependency management through NPM/Yarn (#6332). There can be an argument that BUILD should be the source of truth, but I feel that is impractical given the resources it may take to support something like node.js package management. There should be a way to infer dependencies outside of BUILD files that is easily extensible to other formats.

[nsaechao]

+@stuhood

[stuhood]

Thanks Ny... will get to this this week.

[stuhood]

I believe that #4535 blocks this, which is itself blocked by #6170. I'm going to focus on landing #6170, and then implementing #4535. The actual work involved in this issue once those are fixed is fairly small.

[nsaechao]

In the current state of node source-level dependencies, users have to specify dependencies both in the package.json and the BUILD file. Although it is a minor inconvenience, the process is prone to user error and increases the barrier to utilizing source dependencies with node targets. Adding something like dependency inference is not an impediment to migrating to source dependencies with node targets. However, I do believe that adding something like dep inference earlier would help smooth user transition to using JS source deps and help build trust with the new dependency specification.

[stuhood]

Refreshed the branch here: master...twitter:stuhood/dep-inference .

The largest blockers for landing this afaict (labeled with TODOs in the branch), are:

• Shipping the extract_imports binary to http://github.com/pantsbuild/binaries and fetching it using the DownloadFile intrinsic in a portable way (which maybe @illicitonion could help us spec out as a followup to Require a Digest in UrlToFetch #6672...?)
• Add or move a global option for "inference roots" to replace the SourceRoots concept in the patch (the name for which is overloaded with a related but less restrictive pattern-based concept: https://www.pantsbuild.org/setup_repo.html#source-roots)
• Look into whether there is a more principled way to accomplish a SourcesField union without waiting for Design v2 Target API #4535 (or alternatively, lean in to start Design v2 Target API #4535)

[stuhood]

Shipping the extract_imports binary to http://github.com/pantsbuild/binaries and fetching it using the DownloadFile intrinsic in a portable way (which maybe @illicitonion could help us spec out as a followup to #6672...?)

I think that doing this in a more principled is going to require getting Options incorporated (#5869). Given that, I think that I might just continue to work on Options this week rather than really focusing here.

[stuhood]

Status update: the last remaining rough edge for this is #4535, but it's not a blocker. Enough of the community wants this that we can land whatever support we need to and then maintain it even as we move forward with #4535.

[Eric-Arellano]

@stuhood closed?

[stuhood]

Heck yes. Thank you @Eric-Arellano.