pub.dev is down (global outage) #4663

timsneath · 2021-03-25T02:19:47Z

Overview

At approximately 7pm, the pub infrastructure began to respond with a HTTP 502 Site Error message. The site infra is deployed to the us-central GCP region. This impacts core Flutter services, specifically:

The pub.dev website itself, which presents the error above;
Calls made by the flutter or pub commands (e.g. flutter upgrade or pub upgrade).

Please do not reply to this bug with "me too" or +1 messages, it makes it harder for folk to track. Thanks!

Updates

7:10pm Pacific
We're currently experiencing an outage on pub.dev, which appears to be related to a load balancer issue. We don't have an ETA for a resolution at this time; we're currently working to understand the issue.

7:59pm Pacific
The pub.dev site is still down. We have a Google on-call engineer currently investigating. We have not yet identified a root cause.

8:15pm Pacific
We apologize for the inconvenience. We're seeing load balancer errors and are escalating to the appropriate team. Still no ETA, unfortunately, since we're still haven't determined the root cause.

8:27pm Pacific
We have multiple Google Cloud engineers on-call investigating, but I'm sorry to report that we still don't have a root cause. We'll continue to post updates regularly. Thank you for your patience.

9:00pm Pacific
We are continuing to debug the problem. We have declared a Google escalated outage while we attempt to identify the root cause. Some folk have been successful using the Chinese mirror site at https://pub.flutter-io.cn.

9:20pm Pacific
Again, apologies.

9:35pm Pacific
We are currently exploring the theory that we have exceeded a quota, but that the error didn't show in the log. Paging an oncall team to try and increase the quota to see if this resolves. Again, this really sucks -- we recognize that it's a major inconvenience to you all, and we're feeling sick that we're down. Thank you for being patient with us :(

9:45pm Pacific
We have updated the quota and are resetting the VM instances, to see if we have successfully identified the root cause.

9:51pm Pacific
We are seeing evidence of partially restored service.

9:55pm Pacific
The pub service appears to be fully restored.

10:15pm Pacific
Here's what we think we know at this point in time. At some point within the last day or two, a change was made to the pub.dev landing page that includes a call to the YouTube API. There is a quota limit for YouTube calls that we didn't hit over the last few days, but today we hit it. Confounding the issue, the code was missing exception handling and the logging was inadequate or obfuscated sufficiently that we were unable to immediately spot the problem. The immediate resolution was to raise the quota temporarily to give us time to revert the original change.

At this time we think the issue is resolved, but we'll obviously be monitoring closely. Again, apologies on behalf of the Flutter & Dart teams for the disruption. We take this very seriously, and we will perform a full post-mortem and share the learnings and actions we'll take as a result of this.

The text was updated successfully, but these errors were encountered:

timsneath · 2021-03-25T02:20:33Z

Internal bug: https://buganizer.corp.google.com/issues/183665226

timsneath · 2021-03-25T02:24:01Z

No ETA to fix yet. We're figuring out what's gone wrong; we'll update this bug with status as we learn more.

GiteshDalal · 2021-03-25T02:44:17Z

Is there any back up website that we can point to while pub.dev is down?
This is a big blocker for business.

boyan01 · 2021-03-25T02:58:47Z

Is there any back up website that we can point to while pub.dev is down?
This is a big blocker for business.

~~If you can access pub.flutter-io.cn, then you can try to use this URL as a temporary solution.~~

~~export PUB_HOSTED_URL=https://pub.flutter-io.cn~~

~~ref: https://flutter.dev/community/china~~

Did not work.

timsneath · 2021-03-25T03:01:28Z

Updates will be posted at the top of this bug. To make it easier to follow along, please don't post "me too" or +1 comments.

LeoAiolia · 2021-03-25T03:29:37Z

Kpi is gone this month

BytesZero · 2021-03-25T03:34:38Z

I provide the following solutions, If you can't pack it, please try as follows.
If you have successfully run the project before, please use the second plan. The newly acquired package is not applicable. You need to wait for the website to be completely restored.

pub.dev

 export PUB_HOSTED_URL=~/.pub-cache/hosted/pub.dev

pub.flutter-io.cn （中国地区请使用下面的命令）

 export PUB_HOSTED_URL=~/.pub-cache/hosted/pub.flutter-io.cn

See effect

Running "flutter pub get" in example...                             0.7s

Please see below for new solutions #4663 (comment)

paurakhsharma · 2021-03-25T03:34:52Z

Is this the reason for me being stuck at this? This happened after I downgraded from 2.0.1 to 1.22.6. I even reinstalled flutter but still stuck at it.

Rquinz · 2021-03-25T03:36:20Z

Is this the reason for me being stuck at this? This happened after I downgraded from 2.0.1 to 1.22.6. I even reinstalled flutter but still stuck at it.

Yeah, it is

Rquinz · 2021-03-25T03:50:19Z

Does aliyun has any pub.dev mirrors? or something we can use meanwhile?

Saw this above. You can try this.

If you can access pub.flutter-io.cn, then you can try to use this URL as a temporary solution.

export PUB_HOSTED_URL=https://pub.flutter-io.cn

ref: https://flutter.dev/community/china

doesnt work for me though. goodluck

mingyouzhu · 2021-03-25T03:50:43Z

the "pub.flutter-io.cn" is accessible

sudoaccess · 2021-03-25T05:00:17Z

uuuuuuuppppppp

ensaryusuf · 2021-03-25T05:00:49Z

@timsneath What was the source of the problem?

kxviel · 2021-03-25T05:02:04Z

pub.dev seems to back, but pub get is aint working
UPDATE: pub get works now (india)

hjleesm · 2021-03-25T05:05:11Z

👍

Nanra · 2021-03-25T05:11:11Z

pub.dev is now accessible from Indonesia. Great 👍🏻✨

timsneath · 2021-03-25T05:19:07Z

Thank you all. I've posted a quick update at the top of this bug, but in summary services should now be resolved. We've identified the root cause and increased the quota as a short-term measure until we rollback the offending code.

timnew · 2021-03-25T06:09:07Z

@timsneath is there any status page for the pub.dev. like https://status.cloud.google.com for GCP?
And will it be any public incident report for issue today?

timsneath · 2021-03-25T06:21:12Z

Can't speak to the status page yet; we'll figure out the right mitigations during the post mortem. I'm not sure it would have helped us much: the issue page seemed fairly effective to communicate status. But interested to hear from others.

Yes, we'll share the post-mortem summary. It should make for fun reading :) We operate a blameless post-mortem policy at Google; it's all about learning lessons rather than finding scapegoats. Any failure is a system failure, and we try and learn how we can address the system causes.

lookiestudio · 2021-03-25T07:29:15Z

pub.dev is now accessible from Vietnam.

vgsrivathsan · 2021-03-25T08:20:37Z

pub get is showing socket error

BytesZero · 2021-03-25T08:48:29Z

pub get显示套接字错误

你被墙了

vgsrivathsan · 2021-03-25T09:08:23Z

pub get显示套接字错误

你被墙了

thanks!

themisir · 2021-03-25T16:56:28Z

Seriously, why didn't you cached YouTube calls for some period of time (eg: release cache a few times a day) in the beginning?
Just wondering, seriously.

isoos · 2021-03-25T17:06:57Z

Seriously, why didn't you cached YouTube calls for some period of time (eg: release cache a few times a day) in the beginning?
Just wondering, seriously.

We do cache them, here is the related code with history:
https://github.com/dart-lang/pub-dev/blob/master/app/lib/service/youtube/backend.dart

However, once the fetched failed with the quota limit, the error propagated up in the chain - until the isolate was killed, restarted and with the restart we started to fetch it again. We will redesign/refactor this and similar background task so we can make sure such failures will not be propagated in the future.

themisir · 2021-03-25T18:28:00Z

Seriously, why didn't you cached YouTube calls for some period of time (eg: release cache a few times a day) in the beginning?
Just wondering, seriously.

We do cache them, here is the related code with history:
https://github.com/dart-lang/pub-dev/blob/master/app/lib/service/youtube/backend.dart

However, once the fetched failed with the quota limit, the error propagated up in the chain - until the isolate was killed, restarted and with the restart we started to fetch it again. We will redesign/refactor this and similar background task so we can make sure such failures will not be propagated in the future.

Oh I get it. :D Since the previous fetch is failed and the cache was empty the restarted isolate tried to fetch new data again, and it failed & crashed the isolate and gce restarted it then the loop continued... Interesting failure.

Thanks for letting us know!

Levi-Lesches · 2021-03-25T18:32:19Z

@timsneath + Google team
Want to reiterate that -- thanks for the quick response and clear communication. Maintenance is one of those things that goes unnoticed until it's a bad thing, but it makes us appreciate that pub.dev is otherwise 100% reliable and easy-to-use. ❤️

jonasfj · 2021-04-27T08:18:49Z

Postmortem is referenced here: https://github.com/flutter/flutter/wiki/Postmortems

xi1570-krupeshanadkat · 2021-11-29T05:08:31Z

It seems it is down again (region South Asia - India)

Attached chrome devtools > Network tab screenshots for reference.

My network seems fine, rest of the stuff is opening correctly.

xi1570-krupeshanadkat · 2021-11-29T05:34:21Z

It seems it is down again (region South Asia - India)

Attached chrome devtools > Network tab screenshots for reference.

My network seems fine, rest of the stuff is opening correctly.

Looks like it is working now!

manglide · 2022-04-15T10:16:04Z

It's still not working. Can't access pub.dev

isoos · 2022-04-15T10:45:34Z

@manglide: please open a new issue next time, we don't monitor closed issues.

As pub.dev is working for me, it is possible that the problem is at your ISP's side. Please run this script and return back its output:
https://github.com/dart-lang/pub-dev/blob/master/app/bin/tools/check_domain_access.dart

manglide · 2022-04-15T13:09:18Z

Hi @isoos, thanks for your response. The issue is from my local dnsmasq configuration on mac. I have resolved it now and can access pub.dev.

Thanks once again.

luis901101 · 2023-04-13T17:54:22Z

Hi, is any problem with this again right now?

isoos · 2023-04-13T18:01:11Z

@luis901101 There is an outage right now, we are aware and trying to fix. Also: please don't comment on old threads.

iamchathu · 2023-12-12T16:16:14Z

Is there any public status page for pub.dev?

sigurdm · 2023-12-14T08:53:10Z

@iamchathu no we don't have such a page

timsneath added the P0 critical label Mar 25, 2021

timsneath pinned this issue Mar 25, 2021

This comment has been minimized.

Sign in to view

Levi-Lesches mentioned this issue Mar 25, 2021

https://pub.dev/ dart-lang/pub#2936

Closed

This comment has been minimized.

Sign in to view

timsneath changed the title ~~pub.dev appears to be down~~ pub.dev is down Mar 25, 2021

timsneath mentioned this issue Mar 25, 2021

[RESOLVED] pub.dev is down flutter/flutter#79017

Closed

This comment has been minimized.

Sign in to view

eseidelGoogle changed the title ~~pub.dev is down~~ pub.dev is down (global outage) Mar 25, 2021

This comment has been minimized.

Sign in to view

timsneath closed this as completed Mar 25, 2021

This was referenced Mar 26, 2021

Do not kill frontend isolate on uncaught exceptions. #4672

Merged

Review of Timers and proposal of cached_value.dart #4674

Merged

isoos mentioned this issue Apr 9, 2021

Re-enable Youtube playlist with cached value. #4713

Merged

pub.dev is down (global outage) #4663

pub.dev is down (global outage) #4663

Comments

timsneath commented Mar 25, 2021 • edited

Overview

Updates

timsneath commented Mar 25, 2021

timsneath commented Mar 25, 2021 • edited

This comment has been minimized.

This comment has been minimized.

GiteshDalal commented Mar 25, 2021 • edited

This comment has been minimized.

boyan01 commented Mar 25, 2021 • edited

timsneath commented Mar 25, 2021 • edited

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

LeoAiolia commented Mar 25, 2021

BytesZero commented Mar 25, 2021 • edited

paurakhsharma commented Mar 25, 2021

Rquinz commented Mar 25, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Rquinz commented Mar 25, 2021

mingyouzhu commented Mar 25, 2021

This comment has been minimized.

sudoaccess commented Mar 25, 2021

ensaryusuf commented Mar 25, 2021

kxviel commented Mar 25, 2021 • edited

hjleesm commented Mar 25, 2021

Nanra commented Mar 25, 2021

timsneath commented Mar 25, 2021

timnew commented Mar 25, 2021 • edited

timsneath commented Mar 25, 2021

lookiestudio commented Mar 25, 2021

vgsrivathsan commented Mar 25, 2021

BytesZero commented Mar 25, 2021

vgsrivathsan commented Mar 25, 2021

themisir commented Mar 25, 2021

isoos commented Mar 25, 2021

themisir commented Mar 25, 2021

Levi-Lesches commented Mar 25, 2021 • edited

jonasfj commented Apr 27, 2021

xi1570-krupeshanadkat commented Nov 29, 2021

xi1570-krupeshanadkat commented Nov 29, 2021 • edited

manglide commented Apr 15, 2022

isoos commented Apr 15, 2022

manglide commented Apr 15, 2022

luis901101 commented Apr 13, 2023

isoos commented Apr 13, 2023

iamchathu commented Dec 12, 2023

sigurdm commented Dec 14, 2023

timsneath commented Mar 25, 2021 •

edited

timsneath commented Mar 25, 2021 •

edited

GiteshDalal commented Mar 25, 2021 •

edited

boyan01 commented Mar 25, 2021 •

edited

timsneath commented Mar 25, 2021 •

edited

BytesZero commented Mar 25, 2021 •

edited

kxviel commented Mar 25, 2021 •

edited

timnew commented Mar 25, 2021 •

edited

Levi-Lesches commented Mar 25, 2021 •

edited

xi1570-krupeshanadkat commented Nov 29, 2021 •

edited