Suggest missing dependencies from target's transitive dependencies #4171

peiyuwang · 2017-01-09T18:53:37Z

Problem

Switching from non-strict_deps to strict-deps may result missing deps compile errors if deps used are from target's transitive deps but not declared as direct deps.

Solution

This PR extracts the class names that are reported as not found from failure compile log and locate them in target's transitive classpath.

This approach is on a best-effort bsis:

Compiler may not always report the not found class name, even though the error is due to missing deps. Work to improve scalac coverage is undergoing
Newly introduced deps that are not in the transitive deps are not covered.

Result

./pants compile testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist --compile-zinc-suggest-missing-deps
...
                       [info] Compiling 1 Java source to /Users/peiyu/github/pants/.pants.d/compile/zinc/66ac1c9326be/testprojects.src.java.org.pantsbuild.testproject.missingjardepswhitelist.missingjardepswhitelist/current/classes...
                       [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:4:1: package com.google.common.io does not exist
                       [error] import com.google.common.io.Closer;
                       [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:8:1: cannot find symbol
                       [error]   symbol:   class Closer
                       [error]   location: class org.pantsbuild.testproject.missingjardepswhitelist.MissingJarDepsWhitelist
                       [error]     Closer c = Closer.create();
                       [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:8:1: cannot find symbol
                       [error]   symbol:   variable Closer
                       [error]   location: class org.pantsbuild.testproject.missingjardepswhitelist.MissingJarDepsWhitelist
                       [error]     Closer c = Closer.create();
                       [error] Compile failed at Jan 10, 2017 11:23:09 AM [0.058s]
                       
11:23:09 00:03         [missing-deps-suggest]
                       
                       Found the following deps from target's transitive dependencies that provide the missing classes:
                         com.google.common.io.Closer: 3rdparty:guava
                       
                       If the above information is correct, please add the following to the dependencies of (testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist):
                         3rdparty:guava
                       
                   compile(testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist) failed: Zinc compile failed.

stuhood

Looks good! A few minor issues.

stuhood · 2017-01-10T00:41:31Z

src/python/pants/backend/jvm/tasks/jvm_compile/class_not_found_error_patterns.py

+import re
+
+
+_CLASS_NOT_FOUND_ERROR_PATTERNS = [


Would it be crazy to have these in a pants list option, so that users could add/remove them via configuration?

Also, having these as module-level constants will cause the regexes to be compiled during pants' startup... if you're not going to move them to an option, you should still make them members of a class so that they will be compiled only if they will be used.

not crazy at all! since the patterns aren't going to be complete, versions of compiler may change, it's more flexible to allow override from config.

changed.

stuhood · 2017-01-10T00:42:43Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

@@ -163,6 +165,10 @@ def register_options(cls, register):
             help='Controls whether unused deps are checked, and whether they cause warnings or '
                  'errors.')

+    register('--suggest-missing-deps', advanced=False, type=bool,
+             help='Suggest missing dependencies best-effort from target\'s transitive deps '


s/best-effort/on a best-effort basis/

stuhood · 2017-01-10T00:46:22Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

+                       log_file, settings, fatal_warnings, zinc_file_manager,
+                       javac_plugins_to_exclude)
+        except TaskError:
+          def find_failed_compile_log(workunits):


Defining this inside the except block makes this more challenging to read. Would prefer to see the function defined outside (class level, or otherwise) so that the except is small.

moved _find_failed_compile_log to outside except block

stuhood · 2017-01-10T00:48:26Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

+            for workunit in workunits:
+              for output_name, outpath in workunit.output_paths().items():
+                # Locate failure log from child compilation workunit's stdout
+                if (workunit.name == self.name() and output_name == 'stdout'


Matching self.name is a bit strange... a comment on why this is the appropriate thing to match would be good. Or maybe having an explicit property that matches the tool name?

Added comments

stuhood · 2017-01-10T00:49:31Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

+
+      if missing_dep_suggestions:
+        self.context.log.info('Found the following deps from target\'s transitive '
+                              'dependencies that contain the not found classes:')


s/that contain the not found classes/that provide the missing classes/

stuhood · 2017-01-10T00:50:50Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

+        )
+        self.context.log.info(suggestion_msg)
+
+      if no_suggestions:


IMO, I don't think it makes sense to log the case where you don't find anything (that seems obvious enough). Maybe you could do it at debug level though?

I think there is still value. when there is no suggestion found, there are two possibilities, either error extraction pattern is incomplete or classpath is incomplete. would be good to know which one of the two causes the miss.

change it to debug level.

stuhood · 2017-01-10T00:55:20Z

src/python/pants/backend/jvm/tasks/jvm_dependency_analyzer.py

+    """Search which targets from `target`'s transitive dependencies contain `classname`."""
+    return self._targets_by_class(tuple(target.closure())).get(classname, set())
+
+  @memoized_method


This is a very expensive thing to memoize, afaict. Remember that memoizing a method treats the arguments to the method as a cache key. So you're using a very large (unsorted) tuple as a cache key here.

Instead, I think you'd want to break up the memoization so that you have per-target memoization of classnames: ie, mark a method that takes a single target memoized_method instead.

changed to memoize at target level. Thanks!

stuhood

Looks good, except for one big typo.

stuhood · 2017-01-10T16:53:16Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

@@ -133,7 +138,7 @@ def register_options(cls, register):
             help='Extra compiler args to use when warnings are disabled.')

    register('--fatal-warnings-enabled-args', advanced=True, type=list, fingerprint=True,
-             default=list(cls.get_fatal_warnings_enabled_args_default()),
+             default=CLASS_NOT_FOUND_ERROR_PATTERNS,


xx: wrong option.

Fixed. :( late night edits, remember there was sth odd. Thanks for catching this

stuhood · 2017-01-10T16:55:01Z

src/python/pants/backend/jvm/tasks/jvm_compile/jvm_compile.py

+          raise
+
+  def _find_failed_compile_log(self, compile_workunit):
+    '''One of the compile child workunits actually calls compiler, locate its stdout.'''


" rather than '

stuhood

Thanks Peiyu!

…antsbuild#4171) ### Problem Switching from `non-strict_deps` to `strict-deps` may result missing deps compile errors if deps used are from target's transitive deps but not declared as direct deps. ### Solution This PR extracts the class names that are reported as `not found` from failure compile log and locate them in target's transitive classpath. This approach is on a best-effort bsis: * Compiler may not always report the not found class name, even though the error is due to missing deps. Work to improve scalac coverage is undergoing * Newly introduced deps that are not in the transitive deps are not covered. ### Result ``` ./pants compile testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist --compile-zinc-suggest-missing-deps ... [info] Compiling 1 Java source to /Users/peiyu/github/pants/.pants.d/compile/zinc/66ac1c9326be/testprojects.src.java.org.pantsbuild.testproject.missingjardepswhitelist.missingjardepswhitelist/current/classes... [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:4:1: package com.google.common.io does not exist [error] import com.google.common.io.Closer; [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:8:1: cannot find symbol [error] symbol: class Closer [error] location: class org.pantsbuild.testproject.missingjardepswhitelist.MissingJarDepsWhitelist [error] Closer c = Closer.create(); [error] /Users/peiyu/github/pants/testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist/MissingJarDepsWhitelist.java:8:1: cannot find symbol [error] symbol: variable Closer [error] location: class org.pantsbuild.testproject.missingjardepswhitelist.MissingJarDepsWhitelist [error] Closer c = Closer.create(); [error] Compile failed at Jan 10, 2017 11:23:09 AM [0.058s] 11:23:09 00:03 [missing-deps-suggest] Found the following deps from target's transitive dependencies that provide the missing classes: com.google.common.io.Closer: 3rdparty:guava If the above information is correct, please add the following to the dependencies of (testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist): 3rdparty:guava compile(testprojects/src/java/org/pantsbuild/testproject/missingjardepswhitelist:missingjardepswhitelist) failed: Zinc compile failed. ```

peiyuwang added 11 commits January 5, 2017 16:44

Prepare to extract stdout from children workunit.outputs

81549fc

ready to use the output

b001908

Adding missing_dependency_finder and tests

d2d9ada

Minor cosmetics

2b7ef5b

MissingDependencyFinder still rough but works

e427785

Print suggestion message

01b0226

Handle --colors

6a65e78

Fix the unable to find deps case

f9f8ae9

More polish

7a2cf30

Added integration test

cfe3c75

minor

f1d3c54

peiyuwang requested review from stuhood, ericzundel, mateor and JieGhost January 9, 2017 18:53

peiyuwang mentioned this pull request Jan 9, 2017

Suggest jvm missing dependencies #4140

Closed

Fix inappropriately initialized dep_analyzer

fd39ca7

stuhood reviewed Jan 10, 2017

View reviewed changes

peiyuwang added 5 commits January 9, 2017 19:33

Merge branch 'master' into peiyu/suggest-missing-deps

a3d1373

move patters to cli option

baeeadf

Addressing feedbacks

0f2acea

memorize at target level

a911710

properly check regex contains

833c136

stuhood reviewed Jan 10, 2017

View reviewed changes

Fix unintended changes

c5884b8

stuhood approved these changes Jan 10, 2017

View reviewed changes

peiyuwang added 2 commits January 10, 2017 13:24

zinc error might be stderr

e00df24

handle synthetic target

c007c80

stuhood approved these changes Jan 10, 2017

View reviewed changes

peiyuwang merged commit 3d9f891 into pantsbuild:master Jan 10, 2017

peiyuwang deleted the peiyu/suggest-missing-deps branch January 18, 2017 04:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Suggest missing dependencies from target's transitive dependencies #4171

Suggest missing dependencies from target's transitive dependencies #4171

peiyuwang commented Jan 9, 2017 •

edited

stuhood left a comment

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood left a comment

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood Jan 10, 2017

peiyuwang Jan 10, 2017

stuhood left a comment

		import re


		_CLASS_NOT_FOUND_ERROR_PATTERNS = [

Suggest missing dependencies from target's transitive dependencies #4171

Suggest missing dependencies from target's transitive dependencies #4171

Conversation

peiyuwang commented Jan 9, 2017 • edited

Problem

Solution

Result

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

peiyuwang commented Jan 9, 2017 •

edited