Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Google Cloud Artifact Registry #1987

Open
MasseGuillaume opened this issue Mar 4, 2021 · 16 comments
Open

Feature Request: Google Cloud Artifact Registry #1987

MasseGuillaume opened this issue Mar 4, 2021 · 16 comments

Comments

@MasseGuillaume
Copy link
Contributor

I'm moving to https://cloud.google.com/artifact-registry

There is currently a plugin that works with the Ivy resolver:
https://github.com/suprafii/sbt-google-artifact-registry

I will take a look if I'm able to implement this. Any pointers would be appreciated.

@alexarchambault
Copy link
Member

alexarchambault commented Mar 4, 2021

Thanks for the suggestion, it'd be nice to support this!

Reading this page, it seems Artifact Registry supports both password-based authentication, and authentication based on their credential helper.

Password-based should work already. Would it work for you? See this page for how to pass credentials to the coursier command-line. This should also work from sbt I think, and sbt also accepts those its own way.

About credential helper based authentication, it seems it relies on custom URL schemes (like artifactregistry://LOCATION-maven.pkg.dev/PROJECT/REPOSITORY). It's possible to add support for those. This is illustrated in the tests. To make this work, things along those lines should work:

  • create a library with a coursier.cache.protocol.ArtifactregistryHandler class (beware the lower-case r in Artifactregistry), implementing java.net. URLStreamHandlerFactory, returning java.net. URLConnections that use the credential helper under the hood (using Google's libraries?)
  • publish that library locally, as io.github.masseguillaume::coursier-gcar:0.1.0-SNAPSHOT say
  • test it with the coursier JVM CLI, like
$ cs launch coursier:2.0.12 io.github.masseguillaume::coursier-gcar:0.1.0-SNAPSHOT -- \
    resolve \
      -r '!artifactregistry://LOCATION-maven.pkg.dev/PROJECT/REPOSITORY' \
      org:name:ver

(This uses cs, the native coursier CLI, to launch coursier, the JVM-based one, adding your custom library in its class path. The -r !… options ignores default repositories, via the !, then add a artifactregistry:// one.)

@MasseGuillaume
Copy link
Contributor Author

Great I will take a look at this during the weekend. It does not look to hard to implement. Thanks for the pointers. Also, how does this feature land in sbt? I know there is this project: https://github.com/coursier/sbt-coursier

@MasseGuillaume
Copy link
Contributor Author

MasseGuillaume commented Mar 7, 2021

Yup the method with credentials works fine for fetching artifacts, thanks for the pointer, we will use this until we have the proper authentication. I will see if I can implement the real deal now :).

For publishing artifacts, I still need https://github.com/suprafii/sbt-google-artifact-registry, since sbt does not do preemptive checks, but that's outside of the scope of Coursier.

@MasseGuillaume
Copy link
Contributor Author

I was able to get coursier to resolve the dependency with the method you suggested. Thanks a lot!

Now I'm a bit stuck with integrating this into my build. Two questions:

A) How do I add the library I created to my classpath? Is it only a libraryDependencies += ... in project/plugins.sbt?

B) How do I configure the resolver with this new protocol?

csrConfiguration := {
  val csr = csrConfiguration.value
  csr.withResolvers(csr.resolvers ++ Vector(
    ???
  ))
}

@alexarchambault
Copy link
Member

It ought to work with libraryDependencies += in project/plugins.sbt, then adding your artifactregistry:// repository to resolvers in build.sbt.

If ever it doesn't, maybe try also adding your class as lmcoursier.internal.shaded.coursier.cache.protocol.ArtifactregistryHandler (because of the shading of coursier in sbt).

@alexarchambault
Copy link
Member

It shouldn't be necessary to change csrConfiguration.

@MasseGuillaume
Copy link
Contributor Author

Hum ok I will take a look

adding:

ThisBuild / resolvers += "google" at "artifactregistry://us-maven.pkg.dev/<my..project>/maven"

result in:

 Error parsing Maven repository base artifactregistry://us-maven.pkg.dev/<my...project>/maven (unknown protocol: artifactregistry), ignoring it

https://github.com/coursier/sbt-coursier/blob/master/modules/lm-coursier/src/main/scala/lmcoursier/internal/Resolvers.scala#L45

This means it couldn't load the URLStreamHandlerFactory.

If I do: consoleProject, I can load the ArtifactregistryHandler.

new coursier.cache.protocol.ArtifactregistryHandler
// class loaded
// ^ I added this println in the initializer
// res4: coursier.cache.protocol.ArtifactregistryHandler = coursier.cache.protocol.ArtifactregistryHandler@b3de1

There is probably something funny with the classloader that make this not working:

clsOpt(Thread.currentThread().getContextClassLoader)
          .orElse(clsOpt(getClass.getClassLoader))

@MasseGuillaume
Copy link
Contributor Author

I built a local version of sbt/coursier/coursier shaded and I was able to add more debug logs.

You are correct about the shading, it's trying to load:
lmcoursier.internal.shaded.coursier.cache.protocol.ArtifactregistryHandler

I changed the package name and still no luck. It looks like it's not in the classloader. I'm continuing my investigation 🕵🏻‍♂️

@MasseGuillaume
Copy link
Contributor Author

MasseGuillaume commented Mar 7, 2021

There is indeed something fishy with the classloader/dependency shading.

As a sanity check, I added ArtifactregistryHandler in coursier.

Added: modules/cache/jvm/src/main/java/coursier/cache/protocol/ArtifactregistryHandler.java

+    libs += Deps.artifactregistry,
+    shadedModules += Deps.artifactregistry.module,
+    shadingRules += ShadingRule.moveUnder("com.google", "coursier.cache.shaded"),

and I re-published coursier and sbt-lm-coursier. Then end result was java.lang.NoClassDefFoundError: com/google/cloud/artifactregistry/auth/CredentialProvider.

Edit: Hum maybe because it's a java file?
unzip -l /home/gui/.ivy2/local/io.get-coursier/coursier-cache_2.12/6.6.6-SNAPSHOT/jars/coursier-cache_2.12.jar | grep google is empty

Edit2: Yup I only look for Scala files. This looks like a rabbit hole, I would need to shade all the dependencies. But at least I validated that there is indeed something weird with the classpath.

@MasseGuillaume
Copy link
Contributor Author

I found a way to make it work. If I build the sbt launcher with my dependency, it's available in the classpath and you can download:

console
[GUI] handlerFor: https
[GUI] handlerFor: artifactregistry
[GUI] handlerFor: artifactregistry
openConnection: artifactregistry://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar
https://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar
[info] Fetching artifacts of 
https://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar
[GUI] handlerFor: artifactregistry
openConnection: artifactregistry://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar.sha1
https://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar.sha1
artifactregistry://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-sca…
  100.0% [##########] 20.8 MiB (7.1 MiB / s)
https://us-maven.pkg.dev/<my-project>/maven/ai/automat/shopify-graphql-scala_2.12/0.4.0/shopify-graphql-scala_2.12-0.4.0.jar.sha1
[info] Fetched artifacts of 
[error] stack trace is suppressed; run last update for the full output
[error] (update) java.net.MalformedURLException: unknown protocol: artifactregistry
[error] Total time: 4 s, completed Mar 7, 2021 3:25:28 PM
sbt:yo> last

However, it fails at:

sbt:yo> last
[debug] > Exec(console, Some(0982643e-f8d4-4bc6-bbf0-405b62b51c8b), Some(CommandSource(console0)))
[debug] Evaluating tasks: Compile / console
[debug] Running task... Cancel: Signal, check cycles: false, forcegc: true
[error] java.net.MalformedURLException: unknown protocol: artifactregistry
[error]         at java.net.URL.<init>(URL.java:618)
[error]         at java.net.URL.<init>(URL.java:508)
[error]         at java.net.URL.<init>(URL.java:457)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$artifact$1(SbtUpdateReport.scala:73)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$caching$2(SbtUpdateReport.scala:25)
[error]         at scala.Option.getOrElse(Option.scala:189)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$caching$1(SbtUpdateReport.scala:24)
[error]         at lmcoursier.internal.SbtUpdateReport$$anonfun$1.applyOrElse(SbtUpdateReport.scala:82)
[error]         at lmcoursier.internal.SbtUpdateReport$$anonfun$1.applyOrElse(SbtUpdateReport.scala:80)
[error]         at scala.PartialFunction.$anonfun$runWith$1$adapted(PartialFunction.scala:145)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:943)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:943)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
[error]         at scala.collection.TraversableLike.collect(TraversableLike.scala:407)
[error]         at scala.collection.TraversableLike.collect$(TraversableLike.scala:405)
[error]         at scala.collection.AbstractTraversable.collect(Traversable.scala:108)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$moduleReport$1(SbtUpdateReport.scala:80)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$caching$2(SbtUpdateReport.scala:25)
[error]         at scala.Option.getOrElse(Option.scala:189)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$caching$1(SbtUpdateReport.scala:24)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$moduleReports$30(SbtUpdateReport.scala:283)
[error]         at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:943)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:943)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
[error]         at scala.collection.TraversableLike.map(TraversableLike.scala:286)
[error]         at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[error]         at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error]         at lmcoursier.internal.SbtUpdateReport$.moduleReports(SbtUpdateReport.scala:251)
[error]         at lmcoursier.internal.SbtUpdateReport$.$anonfun$apply$1(SbtUpdateReport.scala:316)
[error]         at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[error]         at scala.collection.Iterator.foreach(Iterator.scala:943)
[error]         at scala.collection.Iterator.foreach$(Iterator.scala:943)
[error]         at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
[error]         at scala.collection.IterableLike.foreach(IterableLike.scala:74)
[error]         at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
[error]         at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
[error]         at scala.collection.TraversableLike.map(TraversableLike.scala:286)
[error]         at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[error]         at scala.collection.AbstractTraversable.map(Traversable.scala:108)
[error]         at lmcoursier.internal.SbtUpdateReport$.apply(SbtUpdateReport.scala:303)
[error]         at lmcoursier.internal.UpdateRun$.$anonfun$update$1(UpdateRun.scala:86)
[error]         at lmcoursier.internal.Lock$.maybeSynchronized(Lock.scala:8)
[error]         at lmcoursier.internal.UpdateRun$.update(UpdateRun.scala:59)
[error]         at lmcoursier.CoursierDependencyResolution.$anonfun$update$37(CoursierDependencyResolution.scala:225)
[error]         at scala.util.Either.map(Either.scala:353)
[error]         at lmcoursier.CoursierDependencyResolution.$anonfun$update$36(CoursierDependencyResolution.scala:222)
[error]         at scala.util.Either.flatMap(Either.scala:341)
[error]         at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:220)
[error]         at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60)
[error]         at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:59)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:133)
[error]         at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:73)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$20(LibraryManagement.scala:146)
[error]         at scala.util.control.Exception$Catch.apply(Exception.scala:228)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:146)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:127)
[error]         at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219)
[error]         at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:160)
[error]         at sbt.Classpaths$.$anonfun$updateTask0$1(Defaults.scala:3608)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]         at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]         at sbt.Execute.work(Execute.scala:291)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)

It's because it's trying to create a new URL without setURLStreamHandlerFactory:

private val artifact = caching[(Module, Map[String, String], Publication, Artifact), sbt.librarymanagement.Artifact] {
    case (module, extraProperties, pub, artifact) =>
      sbt.librarymanagement.Artifact(pub.name)
        .withType(pub.`type`.value)
        .withExtension(pub.ext.value)
        .withClassifier(
          Some(pub.classifier)
            .filter(_.nonEmpty)
            .orElse(MavenAttributes.typeDefaultClassifierOpt(pub.`type`))
            .map(_.value)
        )
        // .withConfigurations(Vector())
        .withUrl(Some(new URL(artifact.url)))
        .withExtraAttributes(module.attributes ++ extraProperties)
  }

Like we say in Quebecois: On n'est pas sorti du bois!

@MasseGuillaume
Copy link
Contributor Author

Almost there, I found the solution for the issue above when creating an URL with a custom protocol. The only issue pending is to figure out why I can't get the classloader to expose my URLStreamHandlerFactory.

@MasseGuillaume
Copy link
Contributor Author

MasseGuillaume commented Mar 8, 2021

Hey, finally, I propose what they did in scalafix where they have: https://github.com/scalacenter/sbt-scalafix/blob/1eeafe2df0c86a3cf4d90137cc3d2c00adb8b407/src/main/scala/scalafix/sbt/ScalafixPlugin.scala#L75

csrProtocolHandlerDependencies: SettingKey[Seq[ModuleID]]

What do you think?

Edit:

Here is my game plan:

  1. Allow passing explicit classloader for CacheUrl.url #1995 CacheUrl.url takes ClassLoader as optional parameter
  2. Add support for custom protocols sbt-coursier#327 Load dependencies from csrProtocolHandlerDependencies and create a class loader
  3. Add support for custom protocol with coursier sbt/sbt#6375 expose csrProtocolHandlerDependencies to the end user

987Nabil added a commit to 987Nabil/scala-steward that referenced this issue Jun 8, 2022
This can be used to add custom url handlers to coursier, so
scala-steward could for example reach s3 buckets, google cloud storage
or artifactregistry.
See https://get-coursier.io/docs/extra.html#extra-protocols and
coursier/coursier#1987
987Nabil added a commit to 987Nabil/scala-steward that referenced this issue Jun 8, 2022
This can be used to add custom url handlers to coursier, so
scala-steward could for example reach s3 buckets, google cloud storage
or artifactregistry.
See https://get-coursier.io/docs/extra.html#extra-protocols and
coursier/coursier#1987
987Nabil added a commit to 987Nabil/scala-steward that referenced this issue Jun 8, 2022
This can be used to add custom url handlers to coursier, so
scala-steward could for example reach s3 buckets, google cloud storage
or artifactregistry.
See https://get-coursier.io/docs/extra.html#extra-protocols and
coursier/coursier#1987
@peterrosell
Copy link

@MasseGuillaume
We're looking into moving to GCP Artifact Registry and we're using Coursier with sbt today.
I see that you made a great job on this issue, but it's been stuck for a year. Any updates on this? Did you find a workaround or run sbt with useCoursier := false

@peterrosell
Copy link

To answer my own question. I just found and tested another plugin, sbt-gcs-resolver and was able to publish to GCP Artifact Registry. https://index.scala-lang.org/abdolence/sbt-gcs-resolver

@MasseGuillaume
Copy link
Contributor Author

Yup the crux of this plugin is here: https://github.com/abdolence/sbt-gcs-resolver/blob/master/sbt-gcs-plugin/src/main/scala/org/latestbit/sbt/gcs/GcsUrlHandlerFactory.scala#L52 It globally set a URL handler.

At the time I wrote all those PR, I was unaware it was possible to solve it this way, I was trying to pass a URL handler until it reaches coursier URL resolver. It's still doable, but the global URL handler has the advantage of being available for all coursiers/sbt versions.

@reddaly
Copy link

reddaly commented Aug 25, 2023

More on Coursier's missing support:

It doesn't seem too difficult to update the credential parsing to support passing a configuration like the following:

simple.username=simple
simple.password=SiMpLe
simple.host=artifacts.simple.com
simple.realm=simple realm
simple.headers[0].name=Auth
simple.headers[0].value=Bearer aklsdfa;lkhn234k5lnlkasd

Followups:

  • I'm not sure what code uses this to make the HTTP requests yet.
  • Most examples I see of artifact registry use the "artifactregistry://" URL scheme. I believe there is an "https://" scheme that can be used for these repositories as an alternative. This might be relevant if there is an issue getting coursier to accept "artifactregistry://" URLs.

987Nabil added a commit to 987Nabil/scala-steward that referenced this issue Feb 29, 2024
This can be used to add custom url handlers to coursier, so
scala-steward could for example reach s3 buckets, google cloud storage
or artifactregistry.
See https://get-coursier.io/docs/extra.html#extra-protocols and
coursier/coursier#1987
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants