Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent errors during NuGet restore is corrupting the package cache #19882

Open
tmat opened this issue May 30, 2017 · 6 comments
Open

Intermittent errors during NuGet restore is corrupting the package cache #19882

tmat opened this issue May 30, 2017 · 6 comments

Comments

@tmat
Copy link
Member

@tmat tmat commented May 30, 2017

System.AggregateException: One or more errors occurred. ---> NuGet.Protocol.Core.Types.FatalProtocolException: An error occurred while retrieving package metadata for 'System.Runtime.4.3.0' from source 'C:\Users\dotnet-bot\.nuget\packages\'. ---> NuGet.Packaging.Core.PackagingException: An error occurred while retrieving package metadata for 'System.Runtime.4.3.0' from source 'C:\Users\dotnet-bot\.nuget\packages\'. ---> System.Xml.XmlException: '.', hexadecimal value 0x00, is an invalid character. Line 1, position 1.
@jasonmalinowski jasonmalinowski changed the title Intermittent nuget failure: NuGet.Protocol.Core.Types.FatalProtocolException: An error occurred while retrieving package metadata Intermittent errors during NuGet restore is corrupting the package cache Jun 1, 2017
@jasonmalinowski

This comment has been minimized.

Copy link
Member

@jasonmalinowski jasonmalinowski commented Jun 1, 2017

(going to use this as the parent bug to track instances of this)

jaredpar added a commit to jaredpar/roslyn that referenced this issue Jun 9, 2017
This change should help us pin down why we are seeing NuGet cache
corruption issues in our Jenkins job.  It adds a check to three places
in our script:

- Start
- Post restore
- End

Depending on when this check fails it should give us a better picture of
where the corruption is occuring.

dotnet#19882
jaredpar added a commit to jaredpar/roslyn that referenced this issue Jun 9, 2017
This change should help us pin down why we are seeing NuGet cache
corruption issues in our Jenkins job.  It adds a check to three places
in our script:

- Start
- Post restore
- End

Depending on when this check fails it should give us a better picture of
where the corruption is occuring.

dotnet#19882
@jmarolf

This comment has been minimized.

Copy link
Member

@jmarolf jmarolf commented Jun 13, 2017

found in dotnet/project-system#2429, results here

@davkean

This comment has been minimized.

Copy link
Member

@davkean davkean commented Jun 13, 2017

@jmarolf That's not corruption. That's someone importing a different version of MicroBuild package - which results in that error. Everyone that shares these images need to agree on version.

@jasonmalinowski

This comment has been minimized.

Copy link
Member

@jasonmalinowski jasonmalinowski commented Jun 23, 2017

Closing this as the root cause was identified. Our VMs were bugchecking, which invalidates the safeties NuGet puts in place to ensure a NuGet crash doesn't corrupt the cache. The file writes it does in order are fine...assuming your disk is eventually flushed. In the bugcheck case, you can see some writes but not others and bad things happen.

NuGet/Home#5438 was logged to track that issue, and the VMs have been patched to avoid the bugcheck.

@tmat

This comment has been minimized.

@jasonmalinowski

This comment has been minimized.

Copy link
Member

@jasonmalinowski jasonmalinowski commented Dec 3, 2019

@tmat Hmm, want to raise this with the engineering team? The prior root cause was the machine was bugchecking, and if there's a lot of bugchecking in the lab I assume we'd have some info on that (and would want that fixed....)

tmat added a commit that referenced this issue Dec 3, 2019
Workaround for #19882, NuGet/Home#5438
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.