New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose rules from language backends with an application to python_dist() creation #5747

Merged
merged 24 commits into from Apr 28, 2018

Conversation

Projects
None yet
4 participants
@cosmicexplorer
Copy link
Contributor

cosmicexplorer commented Apr 25, 2018

Problem

It's not possible to use rules defined outside of src/python/pants/engine in the rest of Pants.

Solution

Modify load_backend() in extension_loader.py to understand a rules entrypoint. If a backend has a method named rules() defined in its register.py, that method will be invoked to get a list of rules to make the scheduler with.

Result

We can now use rules defined in language backends. I created a register.py in the native backend so that we can test that this works (it's consumed in BuildLocalPythonDistributions). I have a diff that expands the native backend which depends on this functionality.

Note: Extending this to cover plugins as well is trivial (make the exact same change, but to load_plugins() instead).

@stuhood
Copy link
Member

stuhood left a comment

This looks great... thanks!

@@ -71,6 +71,8 @@ def register_bootstrap_options(cls, register):
default=['pants.backend.graph_info',
'pants.backend.python',
'pants.backend.jvm',
'pants.backend.native',
'pants.backend.python',

This comment has been minimized.

@stuhood

stuhood Apr 25, 2018

Member

python is already in the list.

This comment has been minimized.

@cosmicexplorer

cosmicexplorer Apr 25, 2018

Contributor

Fixed in 241692f! Thanks.

@@ -103,6 +103,9 @@ def _copy_sources(self, dist_tgt, dist_target_dir):
src_relative_to_target_base)
shutil.copyfile(abs_src_path, src_rel_to_results_dir)

def _request_single(self, product, subject):
return self.context._scheduler.product_request(product, [subject])[0]

This comment has been minimized.

@stuhood

stuhood Apr 25, 2018

Member

This technically is not supposed to be exposed yet... should leave a comment here pointing to #5440.

This comment has been minimized.

@cosmicexplorer

cosmicexplorer Apr 25, 2018

Contributor

Left a comment pointing to #4769 after offline discussion in 241692f, let me know if that's what you were thinking.

@baroquebobcat
Copy link
Contributor

baroquebobcat left a comment

This is really cool! I'd like to see some more tests of the loading functionality

@@ -169,11 +174,14 @@ def setup_legacy_graph(pants_ignore_patterns,

# Create a Scheduler containing graph and filesystem tasks, with no installed goals. The
# LegacyBuildGraph will explicitly request the products it needs.
if not rules:
rules = []

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 25, 2018

Contributor

This param defaulting feels like it should live above the comment about creating the scheduler. Maybe after if not build_file_aliases

This comment has been minimized.

@cosmicexplorer

cosmicexplorer Apr 26, 2018

Contributor

Moved to after if not build_file_aliases in 4cb2a18!

"""

if not isinstance(rules, Iterable):
raise TypeError('The rules must be an iterable, given {}'.format(rules))

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 25, 2018

Contributor

Might be good to explicitly use the repr by changing {} to {!r}.

This comment has been minimized.

@cosmicexplorer
from pants.util.osutil import get_normalized_os_name


class Platform(datatype('Platform', ['normed_os_name'])):

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 25, 2018

Contributor

maybe use normalized_os_name?

This comment has been minimized.

@cosmicexplorer
@@ -144,4 +144,8 @@ def invoke_entrypoint(name):
if subsystems:
build_configuration.register_subsystems(subsystems)

rules = invoke_entrypoint('rules')
if rules:
build_configuration.register_rules(rules)

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 25, 2018

Contributor

I'd like to see some unit tests for this in test_extension_loader.py

This comment has been minimized.

@stuhood

stuhood Apr 25, 2018

Member

It's only 3 lines long... seems like overkill?

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 25, 2018

Contributor

Fair. My thought was that as this is a plugin-writer facing change, it should have tests. If we're currently thinking of it as being experimental, then it's fine without. It'd be nice if that were noted somehow though.

This comment has been minimized.

@stuhood

stuhood Apr 25, 2018

Member

It's definitely covered by tests, in that the rest of this change relies on it working... it just doesn't seem worth a unit test. It's definitely not code with high cyclomatic complexity, for example.

This comment has been minimized.

@cosmicexplorer

cosmicexplorer Apr 26, 2018

Contributor

I totally agree that it's important to test anything that goes out to plugin writers, but as far as I am aware this change only affects backends within the pants codebase. I did however write an extremely small test anyway (see here). The natural extension of this PR is to do the same for plugins, and I absolutely think that change be paired with testprojects using it and integration tests as well, probably, but for this PR I think passing the python dist testing that exists is sufficient to prove it works.

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 26, 2018

Contributor

Thanks!

@cosmicexplorer

This comment has been minimized.

Copy link
Contributor

cosmicexplorer commented Apr 26, 2018

CI is failing at compiling native code and the error message isn't visible because pex decodes the stdout/stderr to ascii in their Installer class (which is the right thing to do there I think?), but that throws. I'm 99% sure I know what the failure is regardless and will fix that.

@cosmicexplorer

This comment has been minimized.

Copy link
Contributor

cosmicexplorer commented Apr 26, 2018

Ok, so setting aside the above, CI is also failing when building the engine for either platform with (on osx):

[=== 05:49 Installing and testing package pantsbuild.pants-1.7.0.dev1+3e5b23a3 ===]
Exception caught: (<type 'exceptions.ValueError'>)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 69, in main
    PantsLoader.run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 65, in run
    cls.load_and_execute(entrypoint)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_loader.py", line 58, in load_and_execute
    entrypoint_main()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 39, in main
    PantsRunner(exiter, start_time=start_time).run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 53, in run
    return runner.run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 49, in run
    self._run()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 95, in _run
    self._exiter).setup()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 201, in setup
    goals, context = self._setup_context()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 172, in _setup_context
    self._global_options.subproject_roots
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 114, in _init_graph
    include_trace_on_error=self._options.for_global_scope().print_exception_stacktrace
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/bin/engine_initializer.py", line 188, in setup_legacy_graph
    scheduler = LocalScheduler(workdir, dict(), tasks, project_tree, native, include_trace_on_error=include_trace_on_error)
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/engine/scheduler.py", line 339, in __init__
    self._scheduler.assert_ruleset_valid()
  File "/var/folders/vy/rcv48w3j4w79llzf_x6qnvw40000gn/T/pants.XXXXX.7MLVcUnZ/lib/python2.7/site-packages/pants/engine/scheduler.py", line 129, in assert_ruleset_valid
    raise ValueError(str(value))
Exception message: Rules with errors: 1
  (SetupPyInvocationEnvironment, (Select(Platform), Select(NativeToolchain)), get_setup_py_env):
    no matches for Select(NativeToolchain) with subject types: PathGlobs, Address, Specs, BuildFileAddress, BuildFileAddresses, ExecuteProcessRequest
    no matches for Select(Platform) with subject types: Specs, BuildFileAddress, ExecuteProcessRequest, BuildFileAddresses, Address, PathGlobs
'pants list src::' failed in venv!
Dry run release failed.
The command "./pants --version && ./build-support/bin/release.sh -n" exited with 1

This rule is not exhibiting this issue locally (on osx) in the python dist integration test, for example (which completes successfully), but when I run that command above locally (on osx) I get the same error (not ./pants list src::, that works fine, but the release script). Why might this ruleset be marked invalid here and not elsewhere? Platform is provided as a SingletonRule and NativeToolchain as a RootRule in src/python/pants/backend/native/register.py.

@baroquebobcat
Copy link
Contributor

baroquebobcat left a comment

Thanks for the changes. Looks good, modulo passing CI.

from pants.util.osutil import get_normalized_os_name


class Platform(datatype('Platform', ['normalized_os_name'])):

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 26, 2018

Contributor

I think this should have a docstring, if you've got a good idea for one handy.

This comment has been minimized.

@cosmicexplorer

cosmicexplorer Apr 26, 2018

Contributor

I do -- this datatype was ripped from my real dev branch here which has a ton more changes (and Platform isn't defined in register.py, for example), so I was a little hasty. I want to leave the docstring off for now because I'm going to move this class out of register.py for sure immediately after this pr gets merged (and keeping the docstring out makes it less legitimate).

Does that make sense? I can totally add the docstring too with great ease, just wanted to get your thoughts on my thoughts.

This comment has been minimized.

@baroquebobcat

baroquebobcat Apr 26, 2018

Contributor

That works for me.

cosmicexplorer added some commits Apr 25, 2018

@cosmicexplorer cosmicexplorer force-pushed the cosmicexplorer:expose-rules-from-language-backends branch from 1418e08 to 94f68f5 Apr 27, 2018

cosmicexplorer added some commits Apr 27, 2018

@wisechengyi wisechengyi merged commit f9b2ed7 into pantsbuild:master Apr 28, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment