New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PLEASE READ: Builds are being queued slowly #3767

Closed
manishas opened this Issue Aug 8, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@manishas
Contributor

manishas commented Aug 8, 2017

UPDATE: The service is back up. There is still a delay in queueing builds, and we're looking into that right now.


We are currently experiencing a major service outage.

Status is posted at https://status.shippable.com

We're actively working on recovering as soon as we can. We'll post updates as we go along.

@thelamer

This comment has been minimized.

thelamer commented Aug 8, 2017

Can we get a timeline or any preliminary information regarding the outage ?? I need to know if we need to cutover.

@manishas

This comment has been minimized.

Contributor

manishas commented Aug 8, 2017

We will have an update in 30 minutes. Our DB is down because for some reason EC2 isn't recognizing the volumes.

@rhenretta

This comment has been minimized.

rhenretta commented Aug 8, 2017

Any new update?

@manishas

This comment has been minimized.

Contributor

manishas commented Aug 8, 2017

The service should be back up very soon, but we're still looking at what's causing slow queuing of builds. I'll update the title of this issue once the service is back up and use it to track the queuing issue.

@manishas

This comment has been minimized.

Contributor

manishas commented Aug 8, 2017

The service should be back up now. We're still looking into the queuing slowness.

@manishas manishas changed the title from PLEASE READ: Major outage to PLEASE READ: Builds are being queued slowly Aug 8, 2017

@manishas manishas added the resolved label Aug 9, 2017

@manishas

This comment has been minimized.

Contributor

manishas commented Aug 9, 2017

Our queue has caught up and builds should no longer be slow to trigger.

We are closely monitoring the situation. I will leave this open for a while to make sure nobody is running into it again.

@manishas

This comment has been minimized.

Contributor

manishas commented Aug 9, 2017

I am closing this since nobody has reported this for a while and our monitoring also shows that builds are being triggered as usual.

@manishas manishas closed this Aug 9, 2017

@manishas manishas removed the resolved label Aug 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment