New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for shorter dependency descriptors #324

Closed
alexarchambault opened this Issue Aug 13, 2016 · 12 comments

Comments

Projects
None yet
3 participants
@alexarchambault
Member

alexarchambault commented Aug 13, 2016

Briefly discussed at Scala Up North with @MasseGuillaume.

It would be cool to allow users to specify a dependency like typelevel/cats, for "org.typelevel" %% "cats" % "latest.release" (with support for latest.release to be added to coursier too). The scala index could be used to get the module org and name "org.typelevel" %% "cats" from typelevel/cats - for public repositories at least (they need to be published on Central too, I guess).

In particular, that would allow to seamlessly launch applications like

$ coursier launch lihaoyi/Ammonite

Support for only the project name (just cats or Ammonite) could be added too, like

$ coursier launch Ammonite

Lastly, for projects publishing several modules, this kind of syntaxes could be supported:

  • http4s/http4s->http4s-circe for org.http4s:http4s-circe_2.11:latest.release (http4s/http4s resolved to "org.http4s" %% ... with the scala index, and the part after -> being interpreted as a specific module of it),
  • or http4s->http4s-circe for org.http4s:http4s-circe_2.11:latest.release (like before, with just http4s resolved to "org.http4s" %% ... too),
    which would allow things like
$ coursier fetch http4s/http4s->http4s-circe
$ coursier fetch http4s->http4s-circe
@alexarchambault

This comment has been minimized.

Show comment
Hide comment
@alexarchambault

alexarchambault Aug 13, 2016

Member

To handle the same syntax for private repositories (e.g. in corporate settings), we could either:

  • get those infos directly from repositories (as suggested by @djspiewak), or
  • crawl a repository and get those infos from its metadata.

Getting infos from repositories would require launching SBT, and have it output the relevant things (org / name for each published sub-project). Which may be a bit cumbersome, as it requires launching SBT for each project we want to index.

Crawling a repository may only work for Maven repos (via the SCM infos of the POMs), not for Ivy ones.

As getting those infos is a bit cumbersome no matter what, these could fetched only from time to time, and made available to possible users, by putting those in a single JAR and publishing it to a repository. That way, users would just have to reference the repository and module of the index. This could possibly be made so by some config files under ~/.coursier. These JAR with the infos would be cached like any JAR, and could be updated from time-to-time via the already existing TTL mechanism say.

Member

alexarchambault commented Aug 13, 2016

To handle the same syntax for private repositories (e.g. in corporate settings), we could either:

  • get those infos directly from repositories (as suggested by @djspiewak), or
  • crawl a repository and get those infos from its metadata.

Getting infos from repositories would require launching SBT, and have it output the relevant things (org / name for each published sub-project). Which may be a bit cumbersome, as it requires launching SBT for each project we want to index.

Crawling a repository may only work for Maven repos (via the SCM infos of the POMs), not for Ivy ones.

As getting those infos is a bit cumbersome no matter what, these could fetched only from time to time, and made available to possible users, by putting those in a single JAR and publishing it to a repository. That way, users would just have to reference the repository and module of the index. This could possibly be made so by some config files under ~/.coursier. These JAR with the infos would be cached like any JAR, and could be updated from time-to-time via the already existing TTL mechanism say.

@alexarchambault alexarchambault added this to the 1.0.0 milestone Aug 23, 2016

@alexarchambault alexarchambault referenced this issue Nov 9, 2016

Merged

Stuff #385

@alexarchambault alexarchambault modified the milestones: 1.1.0, 1.0.0 Nov 30, 2016

@alexarchambault alexarchambault modified the milestone: 1.1.0 Feb 28, 2017

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Apr 23, 2017

Contributor

Instead of trying to automatically figure out maven coordinates with for example scaladex, would it be possible to support configuration for aliases? For example,

$ cat ~/.coursier/coursier.conf
aliases = [
  {
    name = "scalafmt",
    groupId = "com.geirsson",
    artifactId = "scalafmt-cli",
    main = "org.scalafmt.cli.Cli",
    jvmOptions = "-Xss8m"
  }
]

Then you could run

sudo coursier bootstrap --standalone scalafmt -o /usr/local/bin/scalafmt
# OR with hypothetical install/upgrade commands
coursier install scalafmt # puts it in ~/local/bin or wherever you want according to coursier.conf
coursier upgrade scalafmt # if artifact is on maven, use latest version in maven-metadata.xml
coursier upgrade scalafmt 1.0 # manually provide version, this is unnecessarily complex with homebrew

I imagine that it might be possible to create a decentralized system for sharing coursier.conf aliases in a similar way as homebrew works.

coursier add-alias olafurpg/scalafmt
# fetches from olafurpg/coursier-scalafmt,
# similar to homebrew tap olafurpg/scalafmt

There could be a centrally managed collection of aliases in coursier/aliases. However, I believe the decentralized model would more scalable. It's annoying to wait for PR approval when you're publishing your first cli app.

PS. I have given up on supporting homebrew for scalafmt scalameta/scalafmt@0cf3b79 I really believe coursier offers a much more better way to share and consume cli apps on the JVM. I'm happy to help contribute on this front.

Contributor

olafurpg commented Apr 23, 2017

Instead of trying to automatically figure out maven coordinates with for example scaladex, would it be possible to support configuration for aliases? For example,

$ cat ~/.coursier/coursier.conf
aliases = [
  {
    name = "scalafmt",
    groupId = "com.geirsson",
    artifactId = "scalafmt-cli",
    main = "org.scalafmt.cli.Cli",
    jvmOptions = "-Xss8m"
  }
]

Then you could run

sudo coursier bootstrap --standalone scalafmt -o /usr/local/bin/scalafmt
# OR with hypothetical install/upgrade commands
coursier install scalafmt # puts it in ~/local/bin or wherever you want according to coursier.conf
coursier upgrade scalafmt # if artifact is on maven, use latest version in maven-metadata.xml
coursier upgrade scalafmt 1.0 # manually provide version, this is unnecessarily complex with homebrew

I imagine that it might be possible to create a decentralized system for sharing coursier.conf aliases in a similar way as homebrew works.

coursier add-alias olafurpg/scalafmt
# fetches from olafurpg/coursier-scalafmt,
# similar to homebrew tap olafurpg/scalafmt

There could be a centrally managed collection of aliases in coursier/aliases. However, I believe the decentralized model would more scalable. It's annoying to wait for PR approval when you're publishing your first cli app.

PS. I have given up on supporting homebrew for scalafmt scalameta/scalafmt@0cf3b79 I really believe coursier offers a much more better way to share and consume cli apps on the JVM. I'm happy to help contribute on this front.

@alexarchambault

This comment has been minimized.

Show comment
Hide comment
@alexarchambault

alexarchambault Apr 24, 2017

Member

Something along those lines should totally be done (after 1.0 final is out). This is the kind of things that were initially discussed at Scala Up North - @djspiewak was interested in that kind of way of handling aliases IIRC.

brew-like repositories look fine. This could also handle private repositories (both GitHub and Nexus ones, for the aliases and the artifacts).

The scaladex support could be given a lower priority and / or be triggered by a --scaladex flag. (The scaladex handler could also be made more interactive at the same time, like offering to choose among the scaladex search results...)

Member

alexarchambault commented Apr 24, 2017

Something along those lines should totally be done (after 1.0 final is out). This is the kind of things that were initially discussed at Scala Up North - @djspiewak was interested in that kind of way of handling aliases IIRC.

brew-like repositories look fine. This could also handle private repositories (both GitHub and Nexus ones, for the aliases and the artifacts).

The scaladex support could be given a lower priority and / or be triggered by a --scaladex flag. (The scaladex handler could also be made more interactive at the same time, like offering to choose among the scaladex search results...)

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Apr 29, 2017

Contributor

The scaladex support could be given a lower priority and / or be triggered by a --scaladex flag. (The scaladex handler could also be made more interactive at the same time, like offering to choose among the scaladex search results...)

I agree.

I was thinking about the syntax of the configuration. It might be beneficial to support something lightweight that handles the most common cases (published to maven, no custom jvm options) and something more verbose but flexible. Example

aliases {
  scalafmt = "org:artifact:version@com.example.Main" # lightweight
  scalafmt-nightly { # verbose, but more flexible
    org = "com.foo"
    artifact = "bar"
    resolver = "bintray:scalameta/maven"
    password = ${?NEXUS_PW}
  }
}
Contributor

olafurpg commented Apr 29, 2017

The scaladex support could be given a lower priority and / or be triggered by a --scaladex flag. (The scaladex handler could also be made more interactive at the same time, like offering to choose among the scaladex search results...)

I agree.

I was thinking about the syntax of the configuration. It might be beneficial to support something lightweight that handles the most common cases (published to maven, no custom jvm options) and something more verbose but flexible. Example

aliases {
  scalafmt = "org:artifact:version@com.example.Main" # lightweight
  scalafmt-nightly { # verbose, but more flexible
    org = "com.foo"
    artifact = "bar"
    resolver = "bintray:scalameta/maven"
    password = ${?NEXUS_PW}
  }
}
@alexarchambault

This comment has been minimized.

Show comment
Hide comment
@alexarchambault

alexarchambault May 13, 2017

Member

Initial issue fixed by #385.

Member

alexarchambault commented May 13, 2017

Initial issue fixed by #385.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Jul 3, 2017

Contributor

Would it be possible to host the aliases/metadata in the original repo? That is, when users run coursier install scalameta/scalafmt that would fetch necessary metadata from a .coursier file in the scalameta/scalafmt repo. We are experimenting with a similar approach in https://github.com/densh/scala-native-cli (installing from .scalanative files)

Contributor

olafurpg commented Jul 3, 2017

Would it be possible to host the aliases/metadata in the original repo? That is, when users run coursier install scalameta/scalafmt that would fetch necessary metadata from a .coursier file in the scalameta/scalafmt repo. We are experimenting with a similar approach in https://github.com/densh/scala-native-cli (installing from .scalanative files)

@alexarchambault

This comment has been minimized.

Show comment
Hide comment
@alexarchambault

alexarchambault Jul 3, 2017

Member

@olafurpg I guess that could be added, yeah. The scaladex support works this way (fetches / caches an intermediate file, that contains the coordinates of the actual dependency).

Member

alexarchambault commented Jul 3, 2017

@olafurpg I guess that could be added, yeah. The scaladex support works this way (fetches / caches an intermediate file, that contains the coordinates of the actual dependency).

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Jul 3, 2017

Contributor

It would be nice as the tooling author to have the ability to specify the exact version, org, artifact, main class and optionally jvm options. coursier install scalafmt currently fetches scalafx and then fails to find a main class.

So far, however, it's been working fine to generate the full coordinates in CI and let users copy-paste the installation command from the docs http://scalameta.org/scalafmt/#Coursier

Contributor

olafurpg commented Jul 3, 2017

It would be nice as the tooling author to have the ability to specify the exact version, org, artifact, main class and optionally jvm options. coursier install scalafmt currently fetches scalafx and then fails to find a main class.

So far, however, it's been working fine to generate the full coordinates in CI and let users copy-paste the installation command from the docs http://scalameta.org/scalafmt/#Coursier

@alexarchambault

This comment has been minimized.

Show comment
Hide comment
@alexarchambault

alexarchambault Jul 3, 2017

Member

coursier launch scalafmt fetches scalafx because it's the first entry that the scaladex returns:

$ curl https://index.scala-lang.org/api/search\?q\=scalafmt\&target\=JVM\&scalaVersion\=2.11.11
[{"organization":"scalafx","repository":"scalafx",
...
Member

alexarchambault commented Jul 3, 2017

coursier launch scalafmt fetches scalafx because it's the first entry that the scaladex returns:

$ curl https://index.scala-lang.org/api/search\?q\=scalafmt\&target\=JVM\&scalaVersion\=2.11.11
[{"organization":"scalafx","repository":"scalafx",
...
@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Jul 3, 2017

Contributor

Yeah, I know 😞 The search relevance will hopefully get better soon scalacenter/scaladex#389 (comment)

Even if scaladex search relevance improves, I'm not sure if it will be able to handle tricker cases like custom main like here http://scalameta.org/scalafmt/#Nailgun

Contributor

olafurpg commented Jul 3, 2017

Yeah, I know 😞 The search relevance will hopefully get better soon scalacenter/scaladex#389 (comment)

Even if scaladex search relevance improves, I'm not sure if it will be able to handle tricker cases like custom main like here http://scalameta.org/scalafmt/#Nailgun

@MasseGuillaume

This comment has been minimized.

Show comment
Hide comment
@MasseGuillaume

MasseGuillaume Jul 3, 2017

Contributor

weird this was supposed to be fixed: https://index.scala-lang.org/search?q=scalafmt.

let me see what's wrong.

I like olaf's approach too. the only issue I see is this can get out of sync. one can release a new version an forget to update the meta file.

Contributor

MasseGuillaume commented Jul 3, 2017

weird this was supposed to be fixed: https://index.scala-lang.org/search?q=scalafmt.

let me see what's wrong.

I like olaf's approach too. the only issue I see is this can get out of sync. one can release a new version an forget to update the meta file.

@olafurpg

This comment has been minimized.

Show comment
Hide comment
@olafurpg

olafurpg Jul 3, 2017

Contributor

The meta file can be generated on sbt load to keep it in sync.

Automatically finding the right artifact from scaladex is quite challenging, for example in scalafmt it should be the scalafmt-cli module (out of ~3-4 modules) and main org.scalafmt.cli.Cli (out of two mains, can't be auto-detected). I admit it's a bad idea to have two mains, but it would be nice that if it wasn't a problem.

Contributor

olafurpg commented Jul 3, 2017

The meta file can be generated on sbt load to keep it in sync.

Automatically finding the right artifact from scaladex is quite challenging, for example in scalafmt it should be the scalafmt-cli module (out of ~3-4 modules) and main org.scalafmt.cli.Cli (out of two mains, can't be auto-detected). I admit it's a bad idea to have two mains, but it would be nice that if it wasn't a problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment