Can't use Microsoft Azure DevOps Maven Repository as Resolver #5492

mantovani · 2020-04-01T21:24:15Z

steps

Add a Microsoft Azure DevOps Maven repository as resolver with Coursier throws:

[warn]         Note: Unresolved dependencies path:
[error] stack trace is suppressed; run last update for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading com.foo:bar_2.11:1.0.0
[error]   Not found
[error]   Not found
[error]   not found: /Users/mantovani/.ivy2/local/com.foo/bar_2.11/1.0.0/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/com/foo/bar_2.11/1.0.0/bar_2.11-1.0.0.pom
[error]   checksum format error: /Users/mantovani/Library/Caches/Coursier/v1/https/pkgs.dev.azure.com/Foo-backend/Foo/_packaging/Foo-releases/maven/v1/com/foo/bar_2.11/1.0.0/bar_2.11-1.0.0.pom.sha1

ThisBuild / name := "baz"
ThisBuild / organization := "com.foo"
lazy val scala211 = "2.11.12"

crossScalaVersions := Seq(scala211)
ThisBuild / scalaVersion := scala211

libraryDependencies ++= Seq(
  "foo" %% "bar" % 0.1 % "provided"
)

ThisBuild / useCoursier := true

resolvers +=   "nabu-releases" at "https://pkgs.dev.azure.com/foo/Foo/_packaging/foo-releases/maven/v1"
credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

sbt version

[info] This is sbt 1.3.9

sbt:foo> show credentials
[info] * FileCredentials("/Users/mantovani/.sbt/sonatype_credential")
[info] * FileCredentials("/Users/mantovani/.sbt/.credentials")

cat /Users/mantovani/.sbt/.credentials

realm=foo
host=pkgs.dev.azure.com
user=foo
password=xxx

The checksum actually has a HTML from Azure:

cat /Users/mantovani/Library/Caches/Coursier/v1/https/pkgs.dev.azure.com/nabu-backend/Nabu-Bots/_packaging/nabu-releases/maven/v1/com/modak/spark-profiler_2.11/1.0.6-2.4.0/spark-profiler_2.11-1.0.6-2.4.0.pom.sha1


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">




<html lang="en-US">
<head><title>

            Azure DevOps Services | Sign In

</title><meta http-equiv="X-UA-Compatible" content="IE=11;&#32;IE=10;&#32;IE=9;&#32;IE=8" />
    <link rel="SHORTCUT ICON" href="/favicon.ico"/>

    <link data-bundlelength="508502" data-bundlename="commoncss" data-highcontrast="/_static/tfs/M166_20200303.4/_cssbundles/HighContrast/vss-bundle-commoncss-vJ2fST3DWywo5reQpe4kluM0dIKSswAv-7hLhXVoyoEU=" data-includedstyles="jQueryUI-Modified;Core;Splitter;PivotView" href="/_static/tfs/M166_20200303.4/_cssbundles/Default/vss-bundle-commoncss-vRG44Xj08_ItEwhXa3JU14uvR-wUjc7PMTHxSYSSCaHw=" rel="stylesheet" />
<link data-bundlelength="116152" data-bundlename="viewcss" data-highcontrast="/_static/tfs/M166_20200303.4/_cssbundles/HighContrast/vss-bundle-viewcss-vdlgmvK3r7d2NJT2HxmrnSW5uGherQCq6dVLcVJGo_uE=" data-includedstyles="VSS.Controls" href="/_static/tfs/M166_20200303.4/_cssbundles/Default/vss-bundle-viewcss-vu75e1r3CYt-i_9HTv9byA67VrRg063mttS0GgT3kje8=" rel="stylesheet" />

If I turn off CoursierThisBuild / useCoursier := false I have the following error from ivy:

sbt:foo> test
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases
[error] Unable to find credentials for [https://pkgsprodcus1.pkgs.visualstudio.com/ @ pkgs.dev.azure.com].
[error]   Is one of these realms misspelled for host [pkgs.dev.azure.com]:
[error]   * Foo-releases

The text was updated successfully, but these errors were encountered:

eed3si9n · 2020-04-05T19:50:36Z

For the record, I think the publishing side will be fixed by sbt/ivy#36.
The resolving side would need to be handled by Coursier, so this should be reported to coursier/coursier.

mantovani · 2020-04-06T10:10:57Z

@eed3si9n I open a ticket with as well Coursier coursier/coursier#1649

mantovani · 2020-04-16T08:20:36Z

In order to make it work with sbt 1.3.10 you have to create a null realm credential and disable Coursier like:

ThisBuild / useCoursier := false
credentials += Credentials(null, "my.artifact.repo.net", "admin", "****")

If you do use a file just let realm blank.

gilandose · 2020-04-27T21:24:25Z

@mantovani checked out the url response headers and the realm is reported as
www-authenticate →Basic realm="https://pkgsprodeus21.pkgs.visualstudio.com/" not foo

As stated in sbt#5492 and sbt#2366 some artifact hosting services (at least Azure Artifacts, seems to affect GCP as well) do not offer a stable HTTP Auth realm that can be safely set in SBT credentials, hence the need to support null or empty Credentials. The Ivy (publishing) side of the issue was fixed in sbt/ivy#36 As to the resolving side, This commit is only part of the solution as it just prevents an NPE and does not consider if coursier itself will fall back to finding credentials with a null realm that matches the server hostname. Related-to: sbt#5492 Related-to: sbt#2366 Signed-off-by: Erwan Queffelec <erwan.queffelec@gmail.com>

3rwww1 · 2020-05-03T14:38:14Z

I attempted to tackle this issue and found that even if the HTTP realm is an Option in sbt/coursierint there was still room for an NPE, #5526 seems to fix it.

I'm still unsure it needs any work on upstream coursier as stated in coursier/coursier#1649, see my latest comment in #5483... There might be an User Agent issue here as well. I tried logging UA from coursier HTTP requests, it sets the UA header with an empty value, I'll try to set it to Apache Maven or unsetting it to see what happens.

agolovenko · 2021-02-01T10:43:33Z

Still have this problem for sbt > 1.2.8

markoutso · 2021-05-28T21:48:35Z

The problem seems to be in the default coursier configuration.

csrConfiguration := {
    val conf = csrConfiguration.value
    val resolvers = csrResolvers.value
    val adoRepoHost = "pkgs.dev.azure.com"
    val adoCredentialsOpt = credentials.value.collectFirst { case creds: DirectCredentials if creds.host == adoRepoHost => creds }
    val newConfOpt = adoCredentialsOpt.map { adoCredentials =>
      val auths =
        resolvers
          .collect {
            case repo: MavenRepository if repo.root.startsWith(s"https://$adoRepoHost/") =>
              repo.name -> Authentication(adoCredentials.userName, adoCredentials.passwd)
          }
      auths.foldLeft(conf) { case (conf, (repoId, auth)) => conf.addRepositoryAuthentication(repoId, auth) }
    }
    newConfOpt.getOrElse(conf)
  }

This snippet manually adds authentication information to resolution requests towards Azure Artifacts servers.
After that the artifacts are resolved as usual.

The checksum error occurs because Azure Artifacts sometimes returns an HTTP 203 response.
The payload of the response is not the original pom.xml though.
It is an html page telling you to login somewhere somehow.

Coursier treats 2XX response codes as successes so even if you disable the checksum checking you still get some SAX exception with no stacktrace in the error log.

SattaiLanfear · 2021-07-23T20:49:15Z

I'm dealing with this problem myself. Unfortunately I tried the snippet above with sbt 1.5.5, and it didn't seem to help.

I do notice that my organization has a "(orgname).pkgs.visualstudio.com" host instead, and swapped it out as the repo host, but no joy.

spockz · 2021-08-17T10:17:36Z

I'm dealing with this problem myself. Unfortunately I tried the snippet above with sbt 1.5.5, and it didn't seem to help.

I'm dealing with right now as well. But for me update works and updateClassifiers attempts to download the same POMs as update and then tries to download it again and it will fail with the unauthorized html in the pom/sha.

edit:
We have resolved this issue by applying the csrConfiguration to the updateClassifiers task as well as it is specifically determined in Defaults.scala.

.settings(
  csrConfiguration := { ..
  },
  updateClassifiers / csrConfiguration := csrConfiguration.value
}

alexandru · 2021-09-30T12:09:49Z

Hello,

This issue is still happening for us, and it's really annoying.

What fixes it for us is including this sbt-coursier version:

addSbtPlugin("io.get-coursier"  % "sbt-coursier" % "1.0.3")

Before sbt-coursier version 1.0.3 we were doing this:

coursierChecksums := Nil,
coursierArtifactsChecksums := Nil,

But depending on sbt-coursier creates other problems and I'd rather not have it in plugins.sbt.

alexandru · 2021-10-19T13:40:46Z

To add extra details, this solution DOES NOT work:

// Does not work for us
.settings(
  csrConfiguration := {
    // ...
  },
  updateClassifiers / csrConfiguration := csrConfiguration.value
}

alexandru · 2021-10-19T13:46:50Z

One more — adding the sbt-coursier plugin fixes the problem, but there's an extra problem ...

The .sha1 and .pom get downloaded from Azure DevOps on every sbt update. This makes the update, a normally slow operation, to be even slower. In a multi-project setup, I'm seeing update take 35+ seconds.

purijatin · 2021-12-24T14:16:46Z

This is still a problem for us. Unable to publish jar to Azure DevOps from SBT>

Any alternative ways to deal with it?

Roymprog · 2022-01-10T13:20:03Z

.settings(
csrConfiguration := { ..
},
updateClassifiers / csrConfiguration := csrConfiguration.value
}

Using the above I'm able to successfully download artefacts from an ADO Maven repository.

edit:
We have resolved this issue by applying the csrConfiguration to the updateClassifiers task as well as it is specifically determined in Defaults.scala.

The addition to the updateClassifiers key is necessary, but somehow blocks sources from being fetched when running sbt updateClassifiers

EDIT: Fetching sources was fixed by adding the following to the Coursier config:

.withClassifiers(Vector("sources"))
.withHasClassifiers(true)

spockz · 2022-01-10T20:53:59Z

The addition to the updateClassifiers key is necessary, but somehow blocks sources from being fetched when running sbt updateClassifiers

That is interesting because that is exactly how we use it to make it able to download sources. Without it it lacks the proper settings.

purijatin · 2022-01-11T07:41:01Z

The below worked for me in the interim: https://github.com/arktekk/sbt-aether-deploy

# Use 0.26.0 for JDK8. I think the versions above that use JDK11
addSbtPlugin("no.arktekk.sbt" % "aether-deploy" % "0.26.0")

Able to push using the above plugin and following settings:


    publishMavenStyle := true,
    publishTo := {
      val url = "https://pkgs.dev.azure.com/msazure/One/_packaging/AIOps-Core-Model/maven/v1/"
      Some("releases".at(url))
    },
    credentials += Credentials(Path.userHome / ".sbt" / ".credentials")

schragnasher · 2022-12-20T14:33:24Z

I am having issues publishing to an azure feed, but different. I get

[error] (publish) java.net.ProtocolException: Server redirected too many times (20)

I can publish a regular maven project just fine, but sbt is giving me problems.

ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.13.10"
ThisBuild / versionScheme := Some("early-semver")

lazy val root = (project in file("."))
  .settings(
    name := "untitled",
    publishMavenStyle := true,
    publishTo := Some("tsmaven" at "https://[SELF HOSTED ADDRESS]/[ORG]/_packaging/tsmaven/maven/v1"),
    credentials += Credentials (Path.userHome / ".sbt" / "azureArtifactsFeed.credentials")
  )

KeerthiYandaOS · 2022-12-22T16:37:45Z

Can we get help on this, please? Still unable to publish the Azure DevOps Maven repository and seeing the below error:
[error] (publish) java.net.ProtocolException: Server redirected too many times (20)

alexandru · 2023-01-17T11:14:41Z

This still keeps happening.

The solution seems to be to set the realm to null. Since we're using a credentials file, we had to resort to doing this something like this:

ThisBuild / credentials += {
  // https://github.com/sbt/sbt/issues/5492#issuecomment-614492826
  val file = Path(sys.env.getOrElse("SBT_CREDENTIALS_PATH", Path.userHome.toString)) / ".sbt" / ".credentials"
  val props = new java.util.Properties()
  props.load(new java.io.FileInputStream(file))
  Credentials(
    realm = null,
    host = props.getProperty("host"),
    userName = props.getProperty("user"),
    passwd = props.getProperty("password")
  )
}

schragnasher · 2023-01-31T15:30:32Z

This still keeps happening.

The solution seems to be to set the realm to null. Since we're using a credentials file, we had to resort to doing this something like this:

ThisBuild / credentials += {
  // https://github.com/sbt/sbt/issues/5492#issuecomment-614492826
  val file = Path(sys.env.getOrElse("SBT_CREDENTIALS_PATH", Path.userHome.toString)) / ".sbt" / ".credentials"
  val props = new java.util.Properties()
  props.load(new java.io.FileInputStream(file))
  Credentials(
    realm = null,
    host = props.getProperty("host"),
    userName = props.getProperty("user"),
    passwd = props.getProperty("password")
  )
}

This does not help with my issue, I am using a null/empty realm in the creds file. Still get too many redirects, and aether deploy continually give unauthorized even when i knew the creds are good from publishing other maven projects.

mantovani added the Bug label Apr 1, 2020

eed3si9n added the area/library_management library management label Apr 1, 2020

eed3si9n added the Blocked on Upstream label Apr 5, 2020

mantovani mentioned this issue Apr 6, 2020

Can't use Microsoft Azure DevOps Maven Repository as Resolver coursier/coursier#1649

Open

3rwww1 mentioned this issue May 3, 2020

Fix coursierint NPE when credential realm is null #5526

Merged

dwijnand mentioned this issue Jul 17, 2020

Set explicit HTTP Accept header to avoid text/html sbt/ivy#40

Merged

max-empirys mentioned this issue Mar 21, 2023

Fetch artifact on Maven with credentials #7181

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't use Microsoft Azure DevOps Maven Repository as Resolver #5492

Can't use Microsoft Azure DevOps Maven Repository as Resolver #5492

mantovani commented Apr 1, 2020 •

edited by eed3si9n

eed3si9n commented Apr 5, 2020

mantovani commented Apr 6, 2020

mantovani commented Apr 16, 2020

gilandose commented Apr 27, 2020

3rwww1 commented May 3, 2020

agolovenko commented Feb 1, 2021

markoutso commented May 28, 2021

SattaiLanfear commented Jul 23, 2021

spockz commented Aug 17, 2021 •

edited

alexandru commented Sep 30, 2021

alexandru commented Oct 19, 2021

alexandru commented Oct 19, 2021

purijatin commented Dec 24, 2021

Roymprog commented Jan 10, 2022 •

edited

spockz commented Jan 10, 2022

purijatin commented Jan 11, 2022

schragnasher commented Dec 20, 2022

KeerthiYandaOS commented Dec 22, 2022

alexandru commented Jan 17, 2023

schragnasher commented Jan 31, 2023

Can't use Microsoft Azure DevOps Maven Repository as Resolver #5492

Can't use Microsoft Azure DevOps Maven Repository as Resolver #5492

Comments

mantovani commented Apr 1, 2020 • edited by eed3si9n

steps

eed3si9n commented Apr 5, 2020

mantovani commented Apr 6, 2020

mantovani commented Apr 16, 2020

gilandose commented Apr 27, 2020

3rwww1 commented May 3, 2020

agolovenko commented Feb 1, 2021

markoutso commented May 28, 2021

SattaiLanfear commented Jul 23, 2021

spockz commented Aug 17, 2021 • edited

alexandru commented Sep 30, 2021

alexandru commented Oct 19, 2021

alexandru commented Oct 19, 2021

purijatin commented Dec 24, 2021

Roymprog commented Jan 10, 2022 • edited

spockz commented Jan 10, 2022

purijatin commented Jan 11, 2022

schragnasher commented Dec 20, 2022

KeerthiYandaOS commented Dec 22, 2022

alexandru commented Jan 17, 2023

schragnasher commented Jan 31, 2023

mantovani commented Apr 1, 2020 •

edited by eed3si9n

spockz commented Aug 17, 2021 •

edited

Roymprog commented Jan 10, 2022 •

edited