Response status code does not indicate success: 500 (Internal Server Error)

[nunzo-dev]

Hello,

I would like to contact you because we have a new problem. On azure devops during the task "artifactory nuget restore" we encounter an error:

"Response status code does not indicate success: 500 (Internal Server Error)."

Which makes the pipeline fail.

This seems to be related to the update of the task to 1.12.0 a few hours ago

[eyalbe4]

@nunzo-dev,
Thanks for reporting this issue.
Will you please share the entire task log indicating this issue?

[eyalbe4]

@nunzo-dev,
I can see the following error in the log you shared -

2021-04-30T07:05:06.2105940Z NuGet.Protocol.Core.Types.FatalProtocolException: The feed 'JFrogCli [https://XXX.jfrog.io/XXX/api/nuget/v3/tax-all-nuget]' lists package 'LumenWorksCsvReader.4.0.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. ---> NuGet.Protocol.PackageNotFoundProtocolException: Unable to find package 'LumenWorksCsvReader.4.0.0'.
2021-04-30T07:05:06.2107423Z at NuGet.Protocol.FindPackagesByIdNupkgDownloader.d__6.MoveNext()

Recently, JFrog CLI started using the NuGet V3 endpoing by default. Notice the /v3/ part in all the download URLs in the build log. It appears that there's an issue with fetching this package through the v3 endpoint.

Maybe we should add support for reverting to the /v2 endpoint - please let me know what you think.

As a temporary workaround, you can force the Azure DevOps extension to use an older version for JFrog CLI. You can try using version 1.41.2. To do this, you'll need to use the Artifactory Tools Installer task, and set 1.41.2 as the customer version. Note that you'll need to configure an Artifactory repository to fetch the customer version. You can read more about the steps to od this here

Let me know what you think.

[nunzo-dev]

Hello,

We cant change the jfrog-cli version with the "Artifactory Tools Installer" task because the artifactory nuget task automatically download the last version and dont use the version of the Artifactory Tools Installer task ..

And this is not at us to decide what u have to do ..

Please just make a correction very quickly, we have some releases that depend on these pipelines and it is very very critical on our side !

[eyalbe4]

@nunzo-dev,
Indeed in the past the Artifactory tasks always downloaded the latest jfrog-cli version, regardless of the Artifactory Tools Installer configuration.
This has been fixed in version 1.12.0.

[nunzo-dev]

I can confirm that the workaround works.
With version 1.42.2 our pipelines work

[eyalbe4]

Glad to hear this @nunzo-dev!
We'll add a "use nuget v2" option to the "Artifactory NuGet" task. The "jfrog rt nuget" command supports this already. Hopefully this will allow you to upgrade the jfrog-cli version in the future.

[bcouavoux]

Hello,
Im working with @nunzo-dev on this subject.
I dont understand your answer.
We have more than 500 pipelines that use the artifactory tasks. We cant modify all our pipelines and add manually on option on them. This is again a temporary solution that we dont want and that we cant applied on all our project.
If the NuGet V3 endpoint has problem, so revert it by default to the v2 ?

[eyalbe4]

I understand the challenge @bcouavoux and we'd like to assist. Would exposing an environment variable that will be read by jfrog-cli and set v2 as the default help? You'll be able to set it on all of your build agents.
The move to nuget v3 came following many requests from users, who experienced issues with v2. Having v2 as the default may cause issues, which end up getting resolved in v3.

[bcouavoux]

@eyalbe4 , we have several agents pool with a lot of agent and the problem is also present on Microsoft hosted agent where we dont have control on environement variable on it. We can't do what you suggest.

There is something that i don't understand.

Is this a problem with the configuration on our side for v3 nuget compatibility or is the problem is on your side ?

• If the problem is on our side, please explain how we can integrate the v3 compatibility.
• If not, you have to revert to v2 endpoint and solve this endpoint problem on your side before update it to v3.

[bcouavoux]

Please re open the ticket, the answers given did not help us and we are in critical issue on our side

[eyalbe4]

The issue was closed by mistake @bcouavoux. Sorry about that.
I'm not sure why the package fails to be downloaded through the v3 endpoint, and to be honest, I'm not even sure that the nuget API version is the actual problem.
To get to the root cause of the issue, will you be able to try and run the build outside of Azure DevOps, using jfrog-cli only? You can use the latest release, and expect to receive the exact same error you received in your Azure DevOps pipeline.
Then, try running the same command, but with the --nuget-v2 command option. This can help determine the cause of the issue. Once that is determined, we can consider our options.

[jeroenvdbrink]

Just to hop on to this issue: I also suddenly have the same problem:

The feed 'JFrogCli [https://artifactory[redacted]/artifactory/api/nuget/v3/[redacted]]' lists package 'NuGet.Protocol.5.6.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again.
  Unable to find package 'NuGet.Protocol.5.6.0'.
NuGet.Protocol.Core.Types.FatalProtocolException: The feed 'JFrogCli [https://artifactory[redacted]/artifactory/api/nuget/v3/[redacted]]' lists package 'NuGet.Protocol.5.6.0' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. ---> NuGet.Protocol.PackageNotFoundProtocolException: Unable to find package 'NuGet.Protocol.5.6.0'.

This started happening a few hours ago, after the "JFrog Artifactory" Azure DevOps Server plugin was automatically updated from 1.11.8 to 1.12.0. It worked without problems a few hours before that. It has taken the better part of the day to figure out what was going wrong and find this GitHub issue. I can understand people wanting it to default to v3, but I don't have the time to currently update all pipelines or to try debugging the issue. But all our pipelines are currently failing. Isn't there a workaround that we can just revert the change and only reintroduce it when there is a fallback available (or when I have time to debug the issue in a separate test pipeline without blocking all developers).

[bcouavoux]

@eyalbe4,

We tested with jfrog-cli on local and we have same error than our pipeline and with the --nuget-v2 parameters, no error.

Jfroc-cli : 1.47.1
Nuget : 5.9.1

[eyalbe4]

All -
Thank you for sharing your feedback here.
We'll soon release a patch, setting v2 as the default for the Artifactory NuGet task in Azure DevOps.

[jeroenvdbrink]

Thank you!

For completeness, I can also confirm the old version works: I tested on a single test build agent, where I created a $(Agent.ToolsDirectory)/_jfrog/current folder and put jfrog.exe 1.46.3 in it. This works with no errors.

I'm curious what goes wrong on v3 and what we can do to fix this. But for now I'm glad with setting v2 as the default so the existing pipelines will work again on the existing build agents.

[eyalbe4]

This PR sets NuGet protocol v2 as the default, and allows selecting v3 - #250
I'll update here once a patch with this change is released.

[eyalbe4]

Version 1.12.1 with the above fix is released.
Looking forward to everyone's feedback.

[Thacai]

I seem to be getting these aswell with Artifactory NuGet push, even after the 1.12.1 "fix":

I did see the saml integration for normal user access, failed for like 10min earlier today (2. May), around 21:00 UTC+2, might be related

[eyalbe4]

@Thacai,
These 500 errors seem to be unrelated to the issue discussed here, because you're receiving them as a response to an upload of files to Artifactory. The "push" command in the Artifactory NuGet task does nothing more than executing a "jfrog rt upload" command.
As a troubleshooting step, I would try uploading any file using "jfrpg rt upload" to Artifactory from the terminal, making sure you upload the file to same repository. If you get the same error, you can take Azure DevOps out of the equation which should help narrow down the cause.

[jeroenvdbrink]

@eyalbe4 I tested the new 1.12.1 version on a couple of our pipelines and they all seem to work correctly now. Thank you for the quick response!

[nunzo-dev]

We are testing this version 1.12.1 and it solves the problem for some pipelines, we will stay in observation until we get more data

[Thacai]

@eyalbe4
maybe that's all it's suppoesed to do, but after i've pinned the cli version to 1.41.1 instead of automaticlly using 1.47.1, the issue has now gone away

[eyalbe4]

Happy to hear this @mightyjay and @nunzo-dev - thanks for confirming!
@Thacai - I suggest we do the tests proposed above. This seem to require deeper troubleshooting.

[RobiNino]

Hi @nunzo-dev ,
This seems related to #251 .
We are verifying now that the fix will solve your problem so we can release it.
Thanks

[RobiNino]

Hi @nunzo-dev ,
We just released extension version 1.12.2 that includes a fix to this issue.
We'd appreciate your feedback for this.
Thanks!

[nunzo-dev]

Hello RobiNino,

Indeed that solves the problem!

Thanks!

[RobiNino]

Thanks for the feedback @nunzo-dev ,
Sorry again for the inconvenience caused by the recent issues.