Cache publishing in sbt

[jvican]

problem

publishLocal and publish do not cache.

sbt:sbtRoot> actionsProj/publishLocal
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT-sources.jar ...
[info] Done packaging.
[info] Wrote /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT.pom
[info] Main Scala API documentation to /data/rw/code/scala/sbt/main-actions/target/scala-2.12/api...
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT.jar ...
[info] Done packaging.
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:41:15: parameter value doc in method javadoc is never used
[warn]               doc: JavaTools,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:42:15: parameter value log in method javadoc is never used
[warn]               log: Logger,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:43:15: parameter value reporter in method javadoc is never used
[warn]               reporter: Reporter,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Sync.scala:70:13: pattern var target in value $anonfun is never used; `target@_' suppresses this warning
[warn]       case (target, srcs) =>
[warn]             ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Sync.scala:31:13: parameter value outStyle in method apply is never used
[warn]             outStyle: FileInfo.Style = FileInfo.exists)
[warn]             ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/TestResultLogger.scala:132:18: pattern var name in value $anonfun is never used; `name@_' suppresses this warning
[warn]                 (name, testEvent)) =>
[warn]                  ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Tests.scala:156:14: pattern var a in value $anonfun is never used; `a@_' suppresses this warning
[warn]         case a: Argument              => // now handled by whatever constructs `runners`
[warn]              ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Tests.scala:295:18: parameter value tags in method makeSerial is never used
[warn]                  tags: Seq[(Tag, Int)]): Task[List[(String, SuiteResult)]] = {
[warn]                  ^
model contains 43 documentable templates
[warn] 8 warnings found
[info] Main Scala API documentation successful.
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[info] :: delivering :: org.scala-sbt#actions_2.12;1.0.2-SNAPSHOT :: 1.0.2-SNAPSHOT :: integration :: Sat Sep 16 01:05:26 CEST 2017
[info] 	delivering ivy file to /data/rw/code/scala/sbt/main-actions/target/scala-2.12/ivy-1.0.2-SNAPSHOT.xml
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/poms/actions_2.12.pom
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/jars/actions_2.12.jar
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/srcs/actions_2.12-sources.jar
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/docs/actions_2.12-javadoc.jar
[info] 	published ivy to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/ivys/ivy.xml
[success] Total time: 7 s, completed Sep 16, 2017 1:05:27 AM
sbt:sbtRoot> actionsProj/publishLocal
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
Warning: Unknown Scalafmt version 1.2.0; using 1.0 interface
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT-sources.jar ...
[info] Done packaging.
[info] Wrote /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT.pom
[info] Main Scala API documentation to /data/rw/code/scala/sbt/main-actions/target/scala-2.12/api...
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT.jar ...
[info] Done packaging.
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:41:15: parameter value doc in method javadoc is never used
[warn]               doc: JavaTools,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:42:15: parameter value log in method javadoc is never used
[warn]               log: Logger,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Doc.scala:43:15: parameter value reporter in method javadoc is never used
[warn]               reporter: Reporter,
[warn]               ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Sync.scala:70:13: pattern var target in value $anonfun is never used; `target@_' suppresses this warning
[warn]       case (target, srcs) =>
[warn]             ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Sync.scala:31:13: parameter value outStyle in method apply is never used
[warn]             outStyle: FileInfo.Style = FileInfo.exists)
[warn]             ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/TestResultLogger.scala:132:18: pattern var name in value $anonfun is never used; `name@_' suppresses this warning
[warn]                 (name, testEvent)) =>
[warn]                  ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Tests.scala:156:14: pattern var a in value $anonfun is never used; `a@_' suppresses this warning
[warn]         case a: Argument              => // now handled by whatever constructs `runners`
[warn]              ^
[warn] /data/rw/code/scala/sbt/main-actions/src/main/scala/sbt/Tests.scala:295:18: parameter value tags in method makeSerial is never used
[warn]                  tags: Seq[(Tag, Int)]): Task[List[(String, SuiteResult)]] = {
[warn]                  ^
model contains 43 documentable templates
[warn] 8 warnings found
[info] Main Scala API documentation successful.
[info] Packaging /data/rw/code/scala/sbt/main-actions/target/scala-2.12/actions_2.12-1.0.2-SNAPSHOT-javadoc.jar ...
[info] Done packaging.
[info] :: delivering :: org.scala-sbt#actions_2.12;1.0.2-SNAPSHOT :: 1.0.2-SNAPSHOT :: integration :: Sat Sep 16 01:05:33 CEST 2017
[info] 	delivering ivy file to /data/rw/code/scala/sbt/main-actions/target/scala-2.12/ivy-1.0.2-SNAPSHOT.xml
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/poms/actions_2.12.pom
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/jars/actions_2.12.jar
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/srcs/actions_2.12-sources.jar
[info] 	published actions_2.12 to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/docs/actions_2.12-javadoc.jar
[info] 	published ivy to /home/jvican/.ivy2/local/org.scala-sbt/actions_2.12/1.0.2-SNAPSHOT/ivys/ivy.xml
[success] Total time: 5 s, completed Sep 16, 2017 1:05:34 AM
sbt:sbtRoot> 

Why is this an issue?

1. Users expect it to cache, in the same way that compile is cached.
2. Explicit tasks that try to replace this functionality, like scriptedUnpublished in Zinc, will disappear automatically as soon as we have caching at the publish level.

expectation

publishLocal will not publish again if there hasn't been any recompilation of the project.

sbt should be smarter than its users and avoid extra work.

notes

If we agree this is a good idea, we can discuss implementation details.

[eed3si9n]

publish could have all sorts of other side effects on the server-side, like generating meta data, so I don't know if we should mess with that.

I am open to caching publishLocal.

[jvican]

Alright, caching publishLocal sounds like a good tradeoff.

[dwijnand]

If you publish and then delete the published artefacts they should re-publish with publish. How would sbt know if something is published or not remotely? Resolve it during publish to verify?

I agree to no-op'ing publishLocal if it's deemed unnecessary. 👍

Your log output also reminds me of the "incremental scaladoc" feature request: sbt/zinc#387.

[jvican]

Resolve it during publish to verify

The idea I had in mind is that caching will only work if someone has done publish before and hasn't cleaned or removed the target directories manually. When an artifact is published, sbt does the following:

1. It creates a timestamp and write it in a file inside target. The timestamp will be gotten from the published JARs.
2. It records the local URIs where the jars have been published.
3. It records the local repository where it's publishing.

With this information, I believe sbt is capable of checking whether publish is redundant by:

1. Making sure that the local repository where we published before is the same.
2. Making sure that the local URIs that represent the published artifacts exist.
3. Making sure that the timestamps of the artifacts are the same and haven't been modified.

If any of these conditions are not met, sbt bails out and publishes again.

In this case, this would only apply to publishLocal, and not publish. Though it could work for both.