Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow cancellation left pages deploy wedged #22

Closed
sfreilich opened this issue Mar 17, 2022 · 96 comments
Closed

Workflow cancellation left pages deploy wedged #22

sfreilich opened this issue Mar 17, 2022 · 96 comments
Labels
incident Service incident or outage

Comments

@sfreilich
Copy link

I recently had a problem where my pages deploy was stuck in a state where it would perpetually fail with:

Deployment request failed for 75152aed304ba9378fd527c226e9849ca39d5eda due to in progress deployment. Please cancel 4cb1cd27f0f1bceb7df2e4b7cb82785922dc978d first or wait for it to complete.

(Created a thread about that here: https://github.community/t/pages-deploy-wedged-incorrect-request-failed-due-to-in-progress-deployment/234793)

I eventually got the issue unwedged manually using the GitHub API to delete the relevant deployments. It seems what happened was a deploy job was cancelled midway through, but that cancellation failed: https://github.com/google/ink-stroke-modeler/runs/5457992551

Error: Error: Request failed with status code 404
{"message":"Not Found","documentation_url":"https://docs.github.com/rest"}
Sending telemetry for run id 1948901688
Deployment cancellation failed Error: Request failed with status code 401
[...]
url: 'https://api.github.com/repos/google/ink-stroke-modeler/pages/deployment/cancel/4cb1cd27f0f1bceb7df2e4b7cb82785922dc978d',
[...]

That URL seems to be coming from here:

const pagesCancelDeployEndpoint = `https://api.github.com/repos/${process.env.GITHUB_REPOSITORY}/pages/deployment/cancel/${process.env.GITHUB_SHA}`

But I'm confused because I don't see that API endpoint in the documentation: https://docs.github.com/en/rest/reference/pages

And I wasn't able to get it to work myself with curl, I had to use the deployments endpoint to search for and delete the relevant deployments instead: https://docs.github.com/en/rest/reference/deployments#delete-a-deployment

Was this problem due to a bug in cancelHandler?

@DmitryGrayscale
Copy link

got the same today, but with error 400 (same description though: Deployment request failed due to in progress deployment)

@hsm
Copy link

hsm commented Mar 17, 2022

Same issue here. I’ve created a minimal example here: https://github.com/hsm/pages/

Succeeds initially and then fails every time afterwards.

Steps to reproduce:

  1. Create a public repository with an index.html of your choice.
  2. Create a branch with the name gh-pages. This will automatically trigger deployment to Github Pages successfully.
  3. Make a change in the index.html file on the gh-pages branch. This time the automatic deployments fails for me with an error message like this one:
Failed to create deployment for 48e85257726336084dffa6b610228d4ca7b1adbc.
{"message":"Deployment request failed for 48e85257726336084dffa6b610228d4ca7b1adbc due to in progress deployment. Please cancel f7ee6f59e1b777003098bf2f1c6f000ff04c4ec2 first or wait for it to complete.","documentation_url":"https://docs.github.com/rest/reference/repos#create-a-github-pages-deployment"}
Error: Error: Request failed with status code 400
Error: Error: Request failed with status code 400

@leemlb06pmi
Copy link

I'm getting this same problem today - potentially a process change we don't know about? Do we have any solutions as of yet?

@manuelmeurer
Copy link

Same problem here:
https://github.com/manuelmeurer/manuelmeurer.github.io/runs/5586680873

@thomiz
Copy link

thomiz commented Mar 17, 2022

Same issue in my public repo

@hassaanbinaslam
Copy link

@HavenTong
Copy link

@Ryo-N7
Copy link

Ryo-N7 commented Mar 17, 2022

@cnftstats
Copy link

Same issue here:

@eliasnemr
Copy link

same issue

@CesarDavidMorenR
Copy link

same issue but error 400

@AmaniZandalari
Copy link

same issue, error 400

@ThesimpleGithub
Copy link

same issue, error 400....

@FabianBartl
Copy link

same issue, but error 500

@kera-cudmore
Copy link

I'm currently having the same issue too, error 400

@rocketstack-matt
Copy link

I had this issue for the last couple of hours, but seems to have been related to this outage - https://www.githubstatus.com/incidents/sksd097hm0y5 . . . a blank change to force a redeploy of the gh-pages branch has worked having failed multiple attempts in the prior couple of hours.

@leemlb06pmi
Copy link

I got one deploy out the door - but the subsequent change failed with the same 400 error

@TienZhao
Copy link

Same issue since a few hours ago. Fail to deploy now. 😭
https://github.com/TienZhao/covid-shanghai/actions/runs/1999467222

@leo2105
Copy link

leo2105 commented Mar 17, 2022

@kera-cudmore
Copy link

I got one deploy out the door - but the subsequent change failed with the same 400 error

Same here 😬

@mrbuche
Copy link

mrbuche commented Mar 17, 2022

I think GitHub is having some issues today, and it's probably nothing we are ALL doing wrong...

@HavenTong
Copy link

I got one deploy out the door - but the subsequent change failed with the same 400 error

Same here.
https://github.com/HavenTong/HavenTong.github.io/actions/runs/1999597626 deployed successfully.
https://github.com/HavenTong/HavenTong.github.io/actions/runs/1999624535 the subsequent failed.

@ahmtcn123
Copy link

+1 Here

@abubelinha
Copy link

Me too!
Thanks @sfreilich for opening this and linking back here.
@abubelinha

@counter-prop
Copy link

Same issue

@awagmi
Copy link

awagmi commented Mar 17, 2022

same here. i guess we just have to wait

@Clarit7
Copy link

Clarit7 commented Mar 17, 2022

Same issue. Like @mrbuche said, it should be GitHub problem.

@YiMysty
Copy link
Collaborator

YiMysty commented Mar 17, 2022

👋 we have identified the potential issue due to a recent deployment since yesterday. We gonna deploy a fix soon. Currently pages deployment is 1 hour per repo per pages deployment. Will notified again when the fix released. Deeply sorry for the incident.

@Ivens-Zhang
Copy link

I just supposed that something wrong I had done to cause it.

@harrydehix
Copy link

harrydehix commented Mar 28, 2022

Fixed for me now,

Just a matter of waiting?

I just tried rerunning the failed jobs again and again, and suddenly it worked. See here.

@Ivens-Zhang
Copy link

You are so lucky, I hope rerun failed job several times could work.

@TaylorLancaster12
Copy link

I'm also having this issue today on my public repo. Can't get my deploys to run, and all it is is CSS and HTML.

@izahn
Copy link

izahn commented Mar 28, 2022

@vasveena
Copy link

I get the same issue. I deleted the existing deployment. After that, the deployment is stuck in "Current status: updating_pages".

@Lmmmmmm-bb
Copy link

I got same issue in this repo. always Current status: updating_pages.

@fffabiang
Copy link

fffabiang commented Mar 28, 2022

I have the same issue. Stuck in Current status: updating_pages.

@rcanelav
Copy link

Same here :(

@GabrieLempert
Copy link

same here

@AlexandreAlvarez59
Copy link

Same here T.T

@aheze
Copy link

aheze commented Mar 28, 2022

Anyone find a workaround? Same thing here

@Jean-Monsieur
Copy link

Same here :(

@ZoranPandovski
Copy link

Make sure you delete all your previous failing builds and just re-run the latest. Now, it should work

@vasveena
Copy link

Re-deployed without any changes. Works for me now!

@TaylorLancaster12
Copy link

Working now for me as well; deleted my failed workflows and deployed a new change with no issues.

@yoannchaudet
Copy link
Collaborator

Hello 👋,

Just to confirm, it does correspond to our incident this morning https://www.githubstatus.com/incidents/z95g34pl8rdm.

It took us longer than expected to get everything under control again and we identified a few things we will need to improve shortly so it does not happen again. Sorry for the inconvenience.

@Ivens-Zhang
Copy link

Hello 👋,

Just to confirm, it does correspond to our incident this morning https://www.githubstatus.com/incidents/z95g34pl8rdm.

It took us longer than expected to get everything under control again and we identified a few things we will need to improve shortly so it does not happen again. Sorry for the inconvenience.

It has already worked, Thanks for your contribution!

@LearningGp
Copy link

I met similar problems yesterday, after making some attempts that didn't work, I chose to put it on hold. When retrying this morning, the deployment was successful. There does seem to be an unfinished deployment task, but there is no way to cancel.

IndrajeetPatil referenced this issue in Open-Systems-Pharmacology/OSPSuite-R Mar 29, 2022
Because:

- it is exceedingly rare that users would enter a nested lists accidentally
- #853 makes this check completely unnecessary
@orchetect
Copy link

This same issue is happening now on one of my repos. GitHub Status shows all green.

@yoannchaudet
Copy link
Collaborator

yoannchaudet commented Aug 24, 2022

@orchetect We made changes in the last few days (both on the backend and on the client side to limit this scenario from happening. In the worse case a deployment should not block another one for more than 10 minutes (previously 60 minutes), in most cases it should not have to come to that.

@orchetect
Copy link

@yoannchaudet It took a few hours but did seem to resolve

@tex0l
Copy link

tex0l commented Dec 6, 2022

I'm experiencing the same issue right now on my repo: https://github.com/tex0l/tex0l.github.io/

@ToonTalk
Copy link

ToonTalk commented Jan 20, 2023

I intermittently began experiencing this on 7 December have using the Github Desktop for several years without any problems. I saw a suggestion to replace the Github Pages with None at github.com and then restore it to master. But now the entire github.io site has disappeared https://ecraft2learn.github.io/ai/

Can someone please tell me what to enter in the command line to restore gh-pages to the current state of the master repository?

UPDATE: When I reduced my repository from about 5GB to less than 2GB the problem went away. I wish the error messages were more informative.

@jeremy-daley-kr
Copy link

@yoannchaudet I saw you closed this. I'm working on a PR to convert the use of axios out to the @actions/github package, but don't see an answer to @sfreilich question, as I'm wondering it also...

Where is this endpoint?

const pagesCancelDeployEndpoint = `${this.githubApiUrl}/repos/${this.repositoryNwo}/pages/deployment/cancel/${this.buildVersion}`

I don't see it documented anywhere.

@yoannchaudet
Copy link
Collaborator

This API is still marked as internal. This is customary for features while they are in Beta so we can make breaking changes without having to go through a regular depreciation process.

Will make a note for the team to decide if we publish it now or hold a bit longer.

@Peter-developer01
Copy link

Same problem. Any ideas?

@fmaussion
Copy link

@Peter-developer01 I also landed here today which I think hints at server problems at github. For me waiting and re-triggering a build after wait seems to have solved the issue (15 minutes maybe?)

@JamesMGreene JamesMGreene added the incident Service incident or outage label Dec 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
incident Service incident or outage
Projects
None yet
Development

No branches or pull requests