implement RemoteCache #5534
implement RemoteCache #5534
Conversation
|
Do I understand correctly that the generated code located in 'src_managed' will not be cached? |
| final val cachedTestClasifier = "cached-test" | ||
|
|
||
| def gitCommitId: String = | ||
| scala.sys.process.Process("git rev-parse --short HEAD").!!.trim |
There was a problem hiding this comment.
I think that this line introduce a dependency from the git command line tool.
You can possibly do the same using jgit:
git.getRepository().exactRef("refs/heads/master").getObjectId().abbreviate(7).name()It might be an additional dependency for Sbt itself, but I think it would be more "self-contained", what do you think?
There was a problem hiding this comment.
I'm of double mind here. On one hand you're right that it would be good to be self-contained. But on the other, every layer of J-anything could lag behind and introduce problem on its own. See for example pgp vs BouncyCastle. If ppl want JGit support they could use sbt-git and write
ThisBuild / git.gitRunner := com.typesafe.sbt.git.JGitRunner
ThisBuild / remoteCacheId := git.gitHeadCommit.valueor something to the effect?
There was a problem hiding this comment.
And I don't think it's a good idea to use git as a file cache.
The git commit id is the one for the source code, file cache will be a regular Maven repo.
There was a problem hiding this comment.
I understand your concerns, still, for exactly one call I would prefer to opt for the self-contained solution.
PGP is a core functionality and it make sense to use the standard implementation.
Here we are using it as a tool for getting the job done.
There was a problem hiding this comment.
Every dependency we bring into sbt is a long-term liability. here's the transitive dependencies for the current release:
$ cs resolve org.eclipse.jgit:org.eclipse.jgit:5.7.0.202003110725-r
com.googlecode.javaewah:JavaEWAH:1.1.7:default
com.jcraft:jsch:0.1.55:default
com.jcraft:jzlib:1.1.1:default
org.bouncycastle:bcpg-jdk15on:1.64:default
org.bouncycastle:bcpkix-jdk15on:1.64:default
org.bouncycastle:bcprov-jdk15on:1.64:default
org.eclipse.jgit:org.eclipse.jgit:5.7.0.202003110725-r:default
org.slf4j:slf4j-api:1.7.2:default
Note that (not really maintained) sbt-git uses a much older version of JGit. Keeping this in the plugin allows the dependencies to be updated without worrying about bincompat impact etc. Eventually when enough ppl start to use this and realize we need JGit, we can always add more later.
|
There is one thing like this com-lihaoyi/mill#726 |
It has not been cached. I am not sure if it's possible to cache that since the input for the code generation logic would be missing and it will be invalidated anyway. Also if the content hashing worked, it shouldn't matter as long as bit-by-bit identical code gets generated? |
| pushRemoteCache.in(Defaults.TaskZero) := (Def.task { | ||
| val s = streams.value | ||
| val config = pushRemoteCacheConfiguration.value | ||
| IvyActions.publish(ivyModule.value, config, s.log) |
There was a problem hiding this comment.
can this override with any other filestore?
There was a problem hiding this comment.
like maybe we can make use of something like https://www.seafile.com/home/ to keep things sync automatically.
There was a problem hiding this comment.
can this override with any other filestore?
Yes. In the blog post I used:
ThisBuild / pushRemoteCacheTo := Some(MavenCache("local-cache", file("/tmp/remote-cache")))
I wonder about the case of code generation from the static scheme that changes quite rarely (case of generation codecs for protocol buffers). The plugin tracks an earlier generated code (or, more accurately, it tracks the timestamp of changes the scheme) and doesn't generate/recompile code for every time. For now, we just copy the generation stuff from the previous job run at CI, but it looks like an ad-hoc workaround. Is it an edge case for the caching 'src_managed' stuff? |
This adds `pushRemoteCache`, `pushRemoteCacheTo`, `pullRemoteCache`, etc to implement cached compilation facility. In addition, the analysis file location is now made more clear.
In case there are a few local commits ahead of the remote cache, this would still grab the close point in the history to resume the build.
See http://eed3si9n.com/cached-compilation-for-sbt for details.
Ref sbt/zinc#712
This adds
pushRemoteCache,pushRemoteCacheTo,pullRemoteCache, etc to implement cached compilation facility.In addition, the analysis file location is now made more clear.