Unable to pin due to incorrect coursier cache path

[joprice]

I'm seeing the error Error while trying to parse the path of file in the coursier cache when trying to pin with the latest changes meant to improve performance b16b35f. I added a print statement and found that somehow, even though I'm on mac, my coursier cache is at $HOME .coursier/cache instead of the expected $HOME/Library/Caches/Coursier/v1. Is this some incorrect setup or something I have cached? I have no env vars related to coursier exported. It surprises me that failing to find a cached file when pinning would not be something the rules would attempt to recover from, but I assume I'm in a broken state that wasn't anticipated.

[jin]

cc @cheister

That somewhat means that your machine isn't going into the _is_macos() branch here. Do you mind trying a local copy of rules_jvm_external and confirming that?

I'll add this to the release blockers.

[joprice]

Yes I'll check that now

[joprice]

Looks like it's this line https://github.com/coursier/coursier/blob/f48c1c6b01ac5b720e66e06cf93587b21d030e8c/modules/paths/src/main/java/coursier/paths/CoursierPaths.java#L60. It uses the $HOME/.coursier folder if present. Maybe for backwards compat, but maybe also to allow using the same folder across different os'es? I moved my folder out of the way and then it used $HOME/Library/Caches/Coursier/v1 and was able to pin. So I guess that logic in get_coursier_cache_or_default needs to be extended to account for this?

[jin]

Looks like it, we should maintain backwards compat with existing directories. I think doing a test for the existence of the directory suffices? Similar to what we're doing for netrc:

https://github.com/bazelbuild/rules_jvm_external/blob/b16b35fa7160fc6d09752efb80baacb254bce801/coursier.bzl#L231

[joprice]

I hit another issue just now, which may be a bug in coursier: error:

Caused by: java.io.FileNotFoundException: $HOME/Library/Caches/Coursier/v1/.structure.lock (Operation not permitted)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
	at coursierapi.shaded.coursier.paths.CachePath.withStructureLock(CachePath.java:128)
	at coursierapi.shaded.coursier.cache.CacheLocks$.withStructureLock(CacheLocks.scala:21)
	at coursierapi.shaded.coursier.cache.CacheLocks$.withLockOr(CacheLocks.scala:38)
	at coursierapi.shaded.coursier.cache.FileCache.$anonfun$download$32(FileCache.scala:506)
	at coursierapi.shaded.scala.concurrent.Future$.$anonfun$apply$1(Future.scala:661)
	at coursierapi.shaded.scala.concurrent.impl.Promise$Transformation.run(Promise.scala:430)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

(The $HOME is redacted home directory)

Not sure if this is an existing issue. Was the coursier version changed recently as well?

[joprice]

Actually I think this has something to do with a code generator I'm using, which is now interacting in an interesting way with the coursier cache.

[cheister]

I think #419 captures the logic that coursier is using. @joprice can you check that it works for you if you don't have a $HOME/Library/Caches/Coursier/v1 directory?

[joprice]

Yes I can try that out later today.

[joprice]

Looks like it's working. I was able to move my cache directory back to .coursier it did not fail or redownload artifacts to the $HOME/Library/Caches location.

Also there's a prose version of the logic here that might be helpful to link to https://get-coursier.io/docs/cache.html#compatibility-with-former-versions, which also describes the reasoning, along with the versions where the change to cache locations took place.

[jin]

Merged #417. Thanks all!