json-report: one file per dependency instead of multiple; use m2 coords #782

baroquebobcat · 2018-02-16T01:02:17Z

When classifiers are used as part of dependency specifications, it's important to be able to select just the classified artifact. Unfortunately, in the current json, dependencies don't specify classifiers, so it isn't possible to just get one of the artifacts for a dependency when a classifier is required.

This patch introduces maven style artifact prefixes in order to include classifier and packaging information in the coordinates. By doing that, we can use them as keys in dependency lists more easily and it allows consumers of the json to treat those dependency keys as mostly opaque ids rather than having to parse them.

Addresses #743

…dinates When classifiers are used as part of dependency specifications, it's important to be able to select just the classified artifact. Unfortunately, in the current json, dependencies don't specify classifiers, so it isn't possible to just get one of the artifacts for a dependency when a classifier is required. This patch introduces maven style artifact prefixes in order to include classifier and packaging information in the coordinates. By doing that, we can use them as keys in dependency lists more easily and it allows consumers of the json to treat those dependency keys as mostly opaque ids rather than having to parse them.

This is a multi-tool change. The coursier side is here: coursier/coursier#782 This fixes the issue with coursier resolution where transitive dependencies that have classifiers can't be differentiated. Ie, if you have a classifier on a dependency, you get all the jars even if they have a different classifier. Additionally this updates the m2 coordinate string representation to bring it inline with maven's coordinate language. This is because I've chosen that language to use in the coursier change as well

alexarchambault · 2018-02-18T23:30:40Z

@baroquebobcat The CI error seems legit… contains is called on an Option[String], not on the contained String (I guess you want something like .exists(_.contains("…")))

baroquebobcat · 2018-02-20T18:07:08Z

Yeah. Thanks for looking at it.

alexarchambault · 2018-02-23T10:15:36Z

@baroquebobcat New CI failure seems legit too.

wisechengyi · 2018-02-23T20:58:53Z

Turns out I left something when splitting the tests #788, so CI should pass after that.

…ifier

wisechengyi

Thanks! Looks good overall.

wisechengyi · 2018-02-27T23:49:18Z

cli/src/test/scala-2.12/coursier/cli/util/JsonReportTest.scala

+    )
+
+    assert(
+      report == "{\"conflict_resolution\":{},\"dependencies\":[],\"version\":\"0.1.0\"}")


Any particular reason string based assertion is used rather than parsing the json report then assert on structured data? https://github.com/coursier/coursier/blob/master/cli/src/test/scala-2.12/coursier/cli/CliFetchIntegrationTest.scala#L18

I think the assertion failure message is a bit clearer with the string, and the string is the actual output, so it feels like the right thing to test against. That said, it could be a bit too fragile.

…so that the dependency listings can be corrected

wisechengyi · 2018-03-05T01:59:50Z

This error consistently in windows CI with this change:
https://github.com/coursier/coursier/pull/782/files#diff-86042d8fbf74d324dff94d39c3760dfbR1073

C:\Users\appveyor\AppData\Local\Temp\1\sbt_a021fee1\scala-sources-javadoc-jars\build.sbt:35)
[00:32:20] [info] 	at $66918475b6504a4b75df$$anonfun$$sbtdef$1.apply(C:\Users\appveyor\AppData\Local\Temp\1\sbt_a021fee1\scala-sources-javadoc-jars\build.sbt:40)
[00:32:20] [info] 	at $66918475b6504a4b75df$$anonfun$$sbtdef$1.apply(C:\Users\appveyor\AppData\Local\Temp\1\sbt_a021fee1\scala-sources-javadoc-jars\build.sbt:8)
[00:32:20] [info] 	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
[00:32:20] [info] 	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
[00:32:20] [info] 	at sbt.std.Transform$$anon$4.work(System.scala:63)
[00:32:20] [info] 	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
[00:32:20] [info] 	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
[00:32:20] [info] 	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[00:32:20] [info] 	at sbt.Execute.work(Execute.scala:235)
[00:32:20] [info] 	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
[00:32:20] [info] 	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
[00:32:20] [info] 	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
[00:32:20] [info] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
[00:32:20] [info] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[00:32:20] [info] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[00:32:20] [info] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[00:32:20] [info] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[00:32:20] [info] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[00:32:20] [info] 	at java.lang.Thread.run(Thread.java:748)
[00:32:20] [info] [error] (*:updateClassifiersCheck) java.lang.AssertionError: assertion failed: scala-library javadoc not found
[00:32:20] [info] [error] Total time: 2 s, completed Mar 3, 2018 12:38:34 AM

I also verified it by singling it out at #750

The test should be covered in travis as well, do you have any clue why this would be case, @alexarchambault?

alexarchambault · 2018-03-05T08:49:53Z

It's possibly https://github.com/coursier/coursier/pull/782/commits/afe30721a00cc9790edc1289d58240d8eade6704… From what I remember, the scripted test that fails corresponds to things that are a bit hackish (but with non reg tests). I'll have a look at it.

wisechengyi · 2018-03-05T22:48:19Z

@alexarchambault the link is invalid btw

coursier/coursier#782

baroquebobcat · 2018-03-06T22:24:27Z

It looks like I've got the CI issue resolved. It seems that the change I made to Resolution broke some assumptions. I've reverted that and instead do the dep flattening in Helper.

wisechengyi · 2018-03-07T19:56:49Z

Thanks. We are still vetting this change against our internal repo. will merge if all checks out

…#5475) This is a multi-tool change. The coursier side is here: coursier/coursier#782 This fixes the issue with coursier resolution where transitive dependencies that have classifiers can't be differentiated. Ie, if you have a classifier on a dependency, you get all the jars even if they have a different classifier. Additionally this updates the m2 coordinate string representation to bring it inline with maven's coordinate language. This is because I've chosen that language to use in the coursier change as well. I've bumped ivy and coursier's implementation version to invalidate cached artifacts that used the old format.

baroquebobcat mentioned this pull request Feb 16, 2018

[coursier/m2-coords] update coursier json parsing; use maven's coords pantsbuild/pants#5475

Merged

baroquebobcat added 2 commits February 20, 2018 12:01

Fix compilation error due to String -> Option[String] change

1c4cdff

Merge branch 'master' into nhoward/json_entry_per_file

427a5ae

baroquebobcat added 5 commits February 23, 2018 15:39

Merge branch 'master' into nhoward/json_entry_per_file

60b2350

Merge branch 'master' into nhoward/json_entry_per_file

5632f26

fix fetch tests

e08f4c5

Add unit test for JsonReport, enforce sorting

14020cb

Resolution.scala: dependencyArtifacts0 now will return deps per class…

afe3072

…ifier

wisechengyi reviewed Feb 27, 2018

View reviewed changes

baroquebobcat added 2 commits March 1, 2018 16:10

pass around the override classifiers when generating the json report …

f54aa03

…so that the dependency listings can be corrected

Merge branch 'master' into nhoward/json_entry_per_file

3b703cf

revert Resolution change, and move it to Helper

34602bb

baroquebobcat added a commit to baroquebobcat/pants that referenced this pull request Mar 6, 2018

Update coursier jar to align with the associated PR 782

62f17cf

coursier/coursier#782

Merge branch 'master' into nhoward/json_entry_per_file

e943c80

alexarchambault approved these changes Mar 7, 2018

View reviewed changes

Merge branch 'master' into nhoward/json_entry_per_file

37594ba

wisechengyi merged commit d0b4686 into coursier:master Mar 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

json-report: one file per dependency instead of multiple; use m2 coords #782

json-report: one file per dependency instead of multiple; use m2 coords #782

baroquebobcat commented Feb 16, 2018 •

edited by wisechengyi

Loading

alexarchambault commented Feb 18, 2018

baroquebobcat commented Feb 20, 2018

alexarchambault commented Feb 23, 2018

wisechengyi commented Feb 23, 2018

wisechengyi left a comment •

edited

Loading

wisechengyi Feb 27, 2018

baroquebobcat Feb 28, 2018

wisechengyi commented Mar 5, 2018

alexarchambault commented Mar 5, 2018

wisechengyi commented Mar 5, 2018

baroquebobcat commented Mar 6, 2018

wisechengyi commented Mar 7, 2018 •

edited

Loading

json-report: one file per dependency instead of multiple; use m2 coords #782

json-report: one file per dependency instead of multiple; use m2 coords #782

Conversation

baroquebobcat commented Feb 16, 2018 • edited by wisechengyi Loading

alexarchambault commented Feb 18, 2018

baroquebobcat commented Feb 20, 2018

alexarchambault commented Feb 23, 2018

wisechengyi commented Feb 23, 2018

wisechengyi left a comment • edited Loading

Choose a reason for hiding this comment

wisechengyi Feb 27, 2018

Choose a reason for hiding this comment

baroquebobcat Feb 28, 2018

Choose a reason for hiding this comment

wisechengyi commented Mar 5, 2018

alexarchambault commented Mar 5, 2018

wisechengyi commented Mar 5, 2018

baroquebobcat commented Mar 6, 2018

wisechengyi commented Mar 7, 2018 • edited Loading

baroquebobcat commented Feb 16, 2018 •

edited by wisechengyi

Loading

wisechengyi left a comment •

edited

Loading

wisechengyi commented Mar 7, 2018 •

edited

Loading