New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git source dependencies don't work on Windows #895

Closed
jodyfanning opened this Issue Oct 2, 2013 · 17 comments

Comments

Projects
None yet
10 participants
@jodyfanning

jodyfanning commented Oct 2, 2013

Source dependencies from Git (at least) don't work on Windows because the relevant parent directories in staging are not created and Git will not clone the repository. This does work fine on Linux and MacOS, so this is a platform specific issue.

When fetching source dependencies they are put to staging under some hash named directory. The problem is that this hash named directory is not created first.

Using the source dependency example from the documentation

...
fatal: could not create work tree dir 'C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0\sbt-assembly'.: No such file or directory
[error] Nonzero exit code (128): git clone git://github.com/sbt/sbt-assembly C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0\sbt-assembly

If the directory is created manually

PS C:\Users\jodyfan> mkdir C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0

Then it works

...
Cloning into 'C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0\sbt-assembly'...

I worked around it by making a custom resolver that creates the directory before returning the normal Git resolver, but the git resolver itself should create this directory before performing the clone.

Speaking of which, the Build Loaders documentation is wrong. There is no method RetrieveUnit.temporary().

@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Oct 2, 2013

Member

Thanks for the report. A pull request would be welcome. You've already identified the problem and came up with a workaround. Applying it to the sbt codebase should be straightforward, but let me know if you run into any problems.

Member

harrah commented Oct 2, 2013

Thanks for the report. A pull request would be welcome. You've already identified the problem and came up with a workaround. Applying it to the sbt codebase should be straightforward, but let me know if you run into any problems.

@jodyfanning

This comment has been minimized.

Show comment
Hide comment
@jodyfanning

jodyfanning Oct 4, 2013

Sure. It might take a week or two.

jodyfanning commented Oct 4, 2013

Sure. It might take a week or two.

@harrah

This comment has been minimized.

Show comment
Hide comment
@harrah

harrah Oct 5, 2013

Member

Great, that works, thanks!

Member

harrah commented Oct 5, 2013

Great, that works, thanks!

@Blaisorblade

This comment has been minimized.

Show comment
Hide comment
@Blaisorblade

Blaisorblade Nov 27, 2013

Contributor
  1. Can you post the code for your custom resolver? I'm using SBT for a course, and ~30% of my students ran into the same issue. (I didn't know that using source dependencies might be such a stupid idea). As a workaround, having the custom resolver is much better than giving setup instructions.
  2. If you create C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0\sbt-assembly instead of C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0 (or the equivalent in my case), apparently SBT will assume the checkout already happened, without warnings (except maybe the first time, but I only observed the later runs). That's very fragile, and made for a short but tense debugging experience: to fix the problem, you have to guess that the student made a mistake creating a directory — I could only find it because I was lucky. One is certainly not supposed to mess with the cache, but shouldn't SBT doublecheck if everything is right? I mean, there's no source whatsoever in that folder.
Contributor

Blaisorblade commented Nov 27, 2013

  1. Can you post the code for your custom resolver? I'm using SBT for a course, and ~30% of my students ran into the same issue. (I didn't know that using source dependencies might be such a stupid idea). As a workaround, having the custom resolver is much better than giving setup instructions.
  2. If you create C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0\sbt-assembly instead of C:\Users\jodyfan\.sbt\0.13\staging\65e4501327cb11085ce0 (or the equivalent in my case), apparently SBT will assume the checkout already happened, without warnings (except maybe the first time, but I only observed the later runs). That's very fragile, and made for a short but tense debugging experience: to fix the problem, you have to guess that the student made a mistake creating a directory — I could only find it because I was lucky. One is certainly not supposed to mess with the cache, but shouldn't SBT doublecheck if everything is right? I mean, there's no source whatsoever in that folder.
@jodyfanning

This comment has been minimized.

Show comment
Hide comment
@jodyfanning

jodyfanning Nov 28, 2013

Yes, sorry. I haven't had much time to devote to doing a patch. But anyway, this is what I did for our Playframework 2.x project.

So for importing the source plug-in you have a project/project/Plugins.scala. And there I override the resolver. In our case it was for importing sbt-jasmine-plugin.

import java.io.File
import sbt._
import Keys._

object Plugins extends Build {
  lazy val plugins = Project("plugins", file("."))
    .dependsOn(uri("git://github.com/guardian/sbt-jasmine-plugin.git#0.8"))

  // Return our new resolver by default
  override def buildLoaders =
    BuildLoader.resolve(gitResolver) ::
      Nil

  // Define a new build resolver to wrap the original git one
  def gitResolver(info: BuildLoader.ResolveInfo): Option[() => File] =
    if (info.uri.getScheme != "git")
      None
    else {
      // Use a subdirectory of the staging directory for the new plugin build.
      // The subdirectory name is derived from a hash of the URI,
      // and so identical URIs will resolve to the same directory.
      val hashDir = new File(info.staging, Hash.halfHashString(info.uri.normalize.toASCIIString))
      hashDir.mkdirs()

      // Return the original git resolver that will do the actual work.
      Resolvers.git(info)
    }
}

jodyfanning commented Nov 28, 2013

Yes, sorry. I haven't had much time to devote to doing a patch. But anyway, this is what I did for our Playframework 2.x project.

So for importing the source plug-in you have a project/project/Plugins.scala. And there I override the resolver. In our case it was for importing sbt-jasmine-plugin.

import java.io.File
import sbt._
import Keys._

object Plugins extends Build {
  lazy val plugins = Project("plugins", file("."))
    .dependsOn(uri("git://github.com/guardian/sbt-jasmine-plugin.git#0.8"))

  // Return our new resolver by default
  override def buildLoaders =
    BuildLoader.resolve(gitResolver) ::
      Nil

  // Define a new build resolver to wrap the original git one
  def gitResolver(info: BuildLoader.ResolveInfo): Option[() => File] =
    if (info.uri.getScheme != "git")
      None
    else {
      // Use a subdirectory of the staging directory for the new plugin build.
      // The subdirectory name is derived from a hash of the URI,
      // and so identical URIs will resolve to the same directory.
      val hashDir = new File(info.staging, Hash.halfHashString(info.uri.normalize.toASCIIString))
      hashDir.mkdirs()

      // Return the original git resolver that will do the actual work.
      Resolvers.git(info)
    }
}
@lihaoyi

This comment has been minimized.

Show comment
Hide comment
@lihaoyi

lihaoyi Dec 13, 2013

Contributor

+1 to this issue for biting me too

Contributor

lihaoyi commented Dec 13, 2013

+1 to this issue for biting me too

@Blaisorblade

This comment has been minimized.

Show comment
Hide comment
@Blaisorblade

Blaisorblade Dec 13, 2013

Contributor

Yes, sorry. I haven't had much time to devote to doing a patch.

Don't worry, and thanks for posting your solution!

Contributor

Blaisorblade commented Dec 13, 2013

Yes, sorry. I haven't had much time to devote to doing a patch.

Don't worry, and thanks for posting your solution!

@enijns

This comment has been minimized.

Show comment
Hide comment
@enijns

enijns commented Feb 27, 2014

+1

@jsuereth

This comment has been minimized.

Show comment
Hide comment
@jsuereth

jsuereth Feb 27, 2014

Member

So this workaround looks good. While I want to pull in some revamped project resolvers soon, would you mind making a pull request with that change? I'd like to include that in 0.13.2 if we can. If I have time after delving into this ivy madness, I'll see if I can pull it in.

Thanks!

  • Josh
Member

jsuereth commented Feb 27, 2014

So this workaround looks good. While I want to pull in some revamped project resolvers soon, would you mind making a pull request with that change? I'd like to include that in 0.13.2 if we can. If I have time after delving into this ivy madness, I'll see if I can pull it in.

Thanks!

  • Josh

@jsuereth jsuereth added this to the 0.13.2 milestone Feb 27, 2014

@jsuereth jsuereth removed the Documentation label Mar 3, 2014

@jsuereth jsuereth modified the milestones: 0.13.3, 0.13.2 Mar 3, 2014

Blaisorblade referenced this issue in Blaisorblade/scalameter Mar 4, 2014

Fix axel22#23
This undoes the adverse effects of effd715.
@q42jaap

This comment has been minimized.

Show comment
Hide comment
@q42jaap

q42jaap Mar 19, 2014

I don't know how this has been fixed, but for me sbt 0.13 doesn't seem to be working with git:// addresses, so I changed it to ssh://
Also, the workaround code creates the full directory for which my git executable refuses to clone, so I'm posting it here as a reference, maybe it helps somebody:

import RichURI.fromURI

  // Define a new build resolver to wrap the original git one
  def gitResolver(info: BuildLoader.ResolveInfo): Option[() => File] = {
    if (info.uri.getScheme != "git" && info.uri.getScheme != "ssh")
      None
    else {
      val uri = info.uri.withoutMarkerScheme.copy(scheme = "git")

      // Use a subdirectory of the staging directory for the new plugin build.
      // The subdirectory name is derived from a hash of the URI,
      // and so identical URIs will resolve to the same directory.
      val hashDir = sbt.Resolvers.uniqueSubdirectoryFor(uri, info.staging)
      hashDir.getParentFile.mkdirs()

      // Return the original git resolver that will do the actual work.
      Resolvers.git(info)
    }
  }

q42jaap commented Mar 19, 2014

I don't know how this has been fixed, but for me sbt 0.13 doesn't seem to be working with git:// addresses, so I changed it to ssh://
Also, the workaround code creates the full directory for which my git executable refuses to clone, so I'm posting it here as a reference, maybe it helps somebody:

import RichURI.fromURI

  // Define a new build resolver to wrap the original git one
  def gitResolver(info: BuildLoader.ResolveInfo): Option[() => File] = {
    if (info.uri.getScheme != "git" && info.uri.getScheme != "ssh")
      None
    else {
      val uri = info.uri.withoutMarkerScheme.copy(scheme = "git")

      // Use a subdirectory of the staging directory for the new plugin build.
      // The subdirectory name is derived from a hash of the URI,
      // and so identical URIs will resolve to the same directory.
      val hashDir = sbt.Resolvers.uniqueSubdirectoryFor(uri, info.staging)
      hashDir.getParentFile.mkdirs()

      // Return the original git resolver that will do the actual work.
      Resolvers.git(info)
    }
  }
@Blaisorblade

This comment has been minimized.

Show comment
Hide comment
@Blaisorblade

Blaisorblade Mar 19, 2014

Contributor

Thanks!

I don't know how this has been fixed, but for me sbt 0.13 doesn't seem to be working with git:// addresses, so I changed it to ssh://

I got extremely confused by that, but it seems your code does the opposite thing. And indeed, SBT's handling does not handle ssh: URLs. But should it? I see why Git can assume that SSH URLs are Git-related, but arguably SBT can't.
However, we could make this clause:

case Path(path) if path.endsWith(".git") => Resolvers.git(info)

also strip the ssh:// prefix, and maybe this is needed if your URL ends in .git (does it?)

Contributor

Blaisorblade commented Mar 19, 2014

Thanks!

I don't know how this has been fixed, but for me sbt 0.13 doesn't seem to be working with git:// addresses, so I changed it to ssh://

I got extremely confused by that, but it seems your code does the opposite thing. And indeed, SBT's handling does not handle ssh: URLs. But should it? I see why Git can assume that SSH URLs are Git-related, but arguably SBT can't.
However, we could make this clause:

case Path(path) if path.endsWith(".git") => Resolvers.git(info)

also strip the ssh:// prefix, and maybe this is needed if your URL ends in .git (does it?)

@q42jaap

This comment has been minimized.

Show comment
Hide comment
@q42jaap

q42jaap Mar 25, 2014

I couldn't get git:// to work, so I had to use ssh://

The git command is executed with ssh:// which works for me.

q42jaap commented Mar 25, 2014

I couldn't get git:// to work, so I had to use ssh://

The git command is executed with ssh:// which works for me.

@eed3si9n eed3si9n modified the milestones: 0.13.5, 0.13.3 Mar 31, 2014

@pommedeterresautee

This comment has been minimized.

Show comment
Hide comment
@pommedeterresautee

pommedeterresautee Apr 4, 2014

Hi,

I am sorry, I am very new to SBT. I have the same issue, and my office laptop is Windows. I am not sure how I am supposed to use the workaround from jodyfanning.

I create the project/project/Plugins.scala file, and then? I am using a build.sbt file. Do I need to switch to a Scala build one? Do I need a specific call to a function?

Thank you a lot for your help,
Regards,
Michaël

pommedeterresautee commented Apr 4, 2014

Hi,

I am sorry, I am very new to SBT. I have the same issue, and my office laptop is Windows. I am not sure how I am supposed to use the workaround from jodyfanning.

I create the project/project/Plugins.scala file, and then? I am using a build.sbt file. Do I need to switch to a Scala build one? Do I need a specific call to a function?

Thank you a lot for your help,
Regards,
Michaël

@jsuereth jsuereth modified the milestones: 0.13.6, 0.13.5 Apr 4, 2014

@andrewdbate

This comment has been minimized.

Show comment
Hide comment
@andrewdbate

andrewdbate May 30, 2014

I can confirm that this is also a blocker for me. For some reason, the posted patches aren't working, but I'm not sure why. If I manually create the directory, then it works.

andrewdbate commented May 30, 2014

I can confirm that this is also a blocker for me. For some reason, the posted patches aren't working, but I'm not sure why. If I manually create the directory, then it works.

@pommedeterresautee

This comment has been minimized.

Show comment
Hide comment
@pommedeterresautee

pommedeterresautee Sep 18, 2014

Great news, it is now fixed with Git 1.9.4!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Directories are now correctly created by Git. I have tested on my project.

-> This bug can be closed <-

May be an information in the ReadMe would be useful (like please use Git version > 1.9.4 on Windows).

pommedeterresautee commented Sep 18, 2014

Great news, it is now fixed with Git 1.9.4!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Directories are now correctly created by Git. I have tested on my project.

-> This bug can be closed <-

May be an information in the ReadMe would be useful (like please use Git version > 1.9.4 on Windows).

@Blaisorblade

This comment has been minimized.

Show comment
Hide comment
@Blaisorblade

Blaisorblade Sep 18, 2014

Contributor

Cool that Git saved us! Having the info when you get the failure would be even better, but I don't have time for that. If anybody can do a PR for the README/docs/wherever is useful, that'd be great.

Contributor

Blaisorblade commented Sep 18, 2014

Cool that Git saved us! Having the info when you get the failure would be even better, but I don't have time for that. If anybody can do a PR for the README/docs/wherever is useful, that'd be great.

@eed3si9n

This comment has been minimized.

Show comment
Hide comment
@eed3si9n

eed3si9n Feb 2, 2018

Member

Closing this since it's no longer a problem?

Member

eed3si9n commented Feb 2, 2018

Closing this since it's no longer a problem?

@eed3si9n eed3si9n closed this Feb 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment