Automated Repository Testing

[fearphage]

It's not uncommon for the (Debian?) repository to be broken (#2608).

Is there an automated (CI-able) way to confirm this is working so the owners don't have to find out from user-initiated bug reports?

[dagood]

As far as I know, this has only happened twice, and only on the Ubuntu 14.04 repository. I'm asking about this in my mail about the recent instance to the repository owners, though--other Microsoft products depend on these feeds so it shouldn't be on .NET Core to set up some sort of health check.

[dagood]

/cc @leecow

[fearphage]

As far as I know, this has only happened twice, and only on the Ubuntu 14.04 repository.

That's one time too many in my opinion. That's the purpose of (regression) tests. When you find a bug, you write a test case for it and then at a minimum, you'll know the next time it becomes broken.

[dagood]

Agreed, just contrasting vs. "not uncommon".

[dagood]

The repo owners have set up some monitoring and alerting to catch this. My impression is that the system is too complex for pre-publish testing to actually catch these things, so this is probably the best we can hope for right now. (The owners automatically knowing there's a problem rather than us having to ping them.) There are continuing conversations about how to improve the service.

[fearphage]

The repo owners have set up some monitoring and alerting to catch this.

It may be ineffective since the issue is back yet again. It seems like running an Ubuntu docker image with sudo apt-get update after the deploy and ensuring a successful (0) exit code would suffice.

Note: This doesn't feel fixed to me, but I'm unable to reopen the issue.

[dagood]

I'll be interested to hear from them whether the monitoring caught this. I don't know what response time we expect them to have when it is caught automatically.

[dagood]

Rolling back to last known good on error would be ideal, of course. 😕

[herebebeasties]

• Ubuntu repo maybe broken? microsoft/vscode#41094
• yum and apt repos are behind microsoft/vscode#45868
• Failed to fetch https://packages.microsoft.com/ubuntu/14.04/prod/dists/trusty/main/binary-amd64/Packages Hash Sum mismatch #2608
• Ubuntu dotnet install fails with "File has unexpected size"  #3221
• packages.microsoft.com Debian Packages does not verify microsoft/msphpsql#938

My impression is that the system is too complex for pre-publish testing to actually catch these things

If your InRelease PGP signed file is missing, or has an earlier timestamp than your Release file does, it's clearly not going to work. I have a complete lack of context around this process so may be wrong, but that seems easy enough to test/monitor. 😕

[herebebeasties]

Can we get this re-opened? Or another ticket made. It's an improvement if you have monitoring to catch when this happens, but the actual root cause clearly needs addressing.

[dagood]

Does an open issue help you in some way? (Linking to it from somewhere, etc.) I'm not opposed to having one, but there's no work we (.NET Core) can do since we rely on a shared Microsoft resource to operate the repository properly for a variety of teams. I don't have any visibility into the underlying problems.

[herebebeasties]

Not if its not in the right place to get fixed, obviously. It's not uncommon for "master" tickets like this one seems to have become to be held open while the thing they are dependent on is fixed, especially if this is the public-facing view of it all.

It's a great shame that there's no external visibility (or internal, you say) on something that breaks a ton of stuff across the globe whenever it happens, probably costing (tens of?) thousands of man hours to people using the Microsoft stack each time. Especially when it's recurring and clearly not fixed yet. Can't you escalate this with the right team or something?

[dagood]

Can't you escalate this with the right team or something?

@leecow has plans for this, I'll let him decide the best course of action for this (or a new) issue.

[leecow]

We do escalate when issues like this are encountered and I am planning a meeting with them to cover this, and other areas of concern with respect to SLA and validation.

[fearphage]

@leecow Any updates?

[leecow]

Meeting set for next Thurs.