Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is PowerShell Gallery server down? We are noticing an unusual delay to get response when we attempt to publish to PowerShell gallery. #132

Closed
arunswarnam opened this issue Oct 7, 2020 · 31 comments

Comments

@arunswarnam
Copy link

Even search through the console UI doesn't return expected results and times out.

@mkht
Copy link

mkht commented Oct 7, 2020

Same for me. The package downloads very slowly and times out.

@viralmodi
Copy link

I see that too.

@v-ronand
Copy link

v-ronand commented Oct 7, 2020

I am facing a similar issue. I am unable to install azure rm using powershell

@o-l-a-v
Copy link

o-l-a-v commented Oct 7, 2020

Still down, or partially at least. PowerShellGallery status does not reflect that.
image

@hbuckle
Copy link

hbuckle commented Oct 7, 2020

Definitely having problems

2020-10-07T09:22:59Z:     azure-arm: VERBOSE: Downloading module 'Az' with version '4.7.0' from the repository 'https://www.powershellgallery.com/api/v2'.
2020-10-07T09:23:02Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az'' for ''.
2020-10-07T09:24:09Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:24:53Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'&$skip=0&$top=40' for '2' more times
2020-10-07T09:26:29Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Advisor'' for ''.
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Advisor'&$skip=0&$top=40' for '2' more times
2020-10-07T09:28:10Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:28:15Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Aks'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.AnalysisServices'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.Accounts'' for ''.
2020-10-07T09:28:22Z:     azure-arm: VERBOSE: Searching repository 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'' for ''.
2020-10-07T09:31:13Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:31:13Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '2' more times
2020-10-07T09:32:53Z:     azure-arm: VERBOSE: A task was canceled.
2020-10-07T09:32:53Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '1' more times
2020-10-07T09:33:15Z:     azure-arm: VERBOSE: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2020-10-07T09:33:15Z: ==> azure-arm: Install-Package: C:\Program Files\PowerShell\Modules\PowerShellGet\2.2.5\PSModule.psm1:9711
2020-10-07T09:33:15Z:     azure-arm: VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2/FindPackagesById()?id='Az.ApiManagement'&$skip=0&$top=40' for '0' more times
2020-10-07T09:33:15Z: ==> azure-arm: Line |
2020-10-07T09:33:15Z: ==> azure-arm: 9711 |  … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-07T09:33:15Z: ==> azure-arm:      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-07T09:33:15Z: ==> azure-arm:      | One or more errors occurred. (One or more errors occurred. (Object reference not set to an instance of
2020-10-07T09:33:15Z: ==> azure-arm:      | an object.)) (Object reference not set to an instance of an object.)

@tunisiano187
Copy link

Yup, too

@GreenBlood
Copy link

Same here, its breaking one of our CI job.

@SebastianGoers
Copy link

Same here. installing modules from the Gallery takes ages and times out / failes eventually.

@NikolaiO
Copy link

NikolaiO commented Oct 7, 2020

https://psg-prod-eastus.azureedge.net/packages/awspowershell.netcore.4.0.5.nupkg

mirror works if you have local nuget repo/artifactory etc and you know the package names and versions

@NikolaiO
Copy link

NikolaiO commented Oct 7, 2020

also if your local repo thing is checking PSGallery - turn that off, else it'll be super slow...

@NikolaiO
Copy link

NikolaiO commented Oct 7, 2020

@alerickson
Copy link
Member

The gallery's been experiencing ongoing low availability. The issue should be mitigated now. I unfortunately don't have write permissions to this repo, but I've put in a PR to update the status. Feel free to comment here with updates on whether builds are still breaking. We're working on completely resolving the issue, but I'm hoping the scenario's improved for most.

@Skymogul
Copy link

Skymogul commented Oct 7, 2020

@alerickson The issue is definitely still ongoing. The gallery landing pages themselves appear fine but calls to download packages continue to fail.

@siromega
Copy link

siromega commented Oct 7, 2020

In my CI/CD pipeline in AzDO it took 55 minutes to import-module Az. I don’t think it is fixed.

@normj
Copy link

normj commented Oct 7, 2020

We are still seeing issues doing our AWS PowerShell release. The release normally takes about an hour for the 200 packages we have to push but the release job has been going on for about 3.5 hours and we look to be only about half way through.

@arunswarnam
Copy link
Author

Yes.. The availability is better than yesterday but the problem exists.

@alerickson
Copy link
Member

We were seeing significant improvements when we switched to our backup backend cloud service, but the issue started to appear again. We're still working on mitigation right now.

@NikolaiO
Copy link

NikolaiO commented Oct 8, 2020

hope the issue can be revealed and talked about technically - seems like it would be interesting, given how much this service is used. As a SRE/DevOps person keen to be made aware of a thing and learn something new.

@alerickson
Copy link
Member

I believe the issue is mitigated now, but we're monitoring to see if it presents itself again. @NikolaiO we'll get a post-mortem out on this issue as well.

@lawrencegripper
Copy link

I'm seeing a similar issue again here. Very long download times and connections hanging for an age.

@mikeharder
Copy link

mikeharder commented Oct 19, 2020

@alerickson: We are seeing very long download times again.

@MattJeanes
Copy link

I'm seeing very long delays installing Az modules from PSGallery today, delays of approximately 30 minutes or more before it works

@JanuszNowak
Copy link

Here the same having issues for over 7 days.

@alerickson
Copy link
Member

Unfortunately it might be another week until significant improvements start to show. Our database is processing a backlog of statistics right now, locking temporarily when running each batch. Today we've been hit by many malformed incoming urls as well. We're definitely prioritizing this issue and will continue to work on it until it's completely resolved.

@MattJeanes
Copy link

We've had a assortment of intermittent errors today, two of them being:

2020-10-23T11:04:19.0821567Z PackageManagement\Install-Package : Package 'Az.Accounts' failed to be installed because: End of Central Directory record could not be found.
2020-10-23T11:04:19.0825860Z At C:\program files\powershell\7\Modules\PowerShellGet\PSModule.psm1:9709 char:34
2020-10-23T11:04:19.0834946Z + … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-23T11:04:19.0835461Z +                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-23T11:04:19.0835904Z + CategoryInfo          : InvalidResult: (Az.Accounts:String) [Install-Package], Exception
2020-10-23T11:04:19.0837230Z + FullyQualifiedErrorId : Package '{0}' failed to be installed because: {1},Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage
2020-10-23T12:18:38.5223796Z PackageManagement\Install-Package : No match was found for the specified search criteria and module name 'Az.Storage'. Try Get-PSRepository to see all available registered module repositories.
2020-10-23T12:18:38.5228737Z At C:\program files\powershell\7\Modules\PowerShellGet\PSModule.psm1:9709 char:34
2020-10-23T12:18:38.5252604Z + … talledPackages = PackageManagement\Install-Package @PSBoundParameters
2020-10-23T12:18:38.5253090Z +                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2020-10-23T12:18:38.5253607Z + CategoryInfo          : ObjectNotFound: (Microsoft.PowerShel…lets.InstallPackage:InstallPackage) [Install-Package], Exception
2020-10-23T12:18:38.5254147Z + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

This is running Install-Module from Powershell Core 7 on an Azure Pipelines Agent, and it worked after retrying but this is very frustrating. Statistics generation should really not impact the core download service, hopefully this issue is resolved soon it's causing our deployments to become very unstable.

@mikeharder
Copy link

@MattJeanes: If you are using the Microsoft-hosted agents, you might be able to use the AzurePowerShell task which uses a pre-installed copy of the Az modules:

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-powershell?view=azure-devops

@MattJeanes
Copy link

@mikeharder Our use case requires Az CLI and when using that task it doesn't seem to have the Az modules loaded so have to install, which is a shame.

@mikeharder
Copy link

@MattJeanes: In that case you might be able to use a workaround like we did: https://github.com/Azure/azure-sdk-tools/pull/1137/files

@MattJeanes
Copy link

@mikeharder that's super useful, I'll definitely give that a go - thank you!

@staedter
Copy link

I believe the issue is mitigated now, but we're monitoring to see if it presents itself again. @NikolaiO we'll get a post-mortem out on this issue as well.

Was there ever a post-mortem released on this?

@SydneyhSmith
Copy link
Contributor

Blog post should be out covering this issue soon (at https://devblogs.microsoft.com/powershell/)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests