Bugs/Improvement proposals filed to GitHub Actions

[potiuk]

Description

Github Actions is great for us (much better than Travis used to be) but they change so fast that some of the features are not perfect and some of them are missing. This issue is an "umbrella issue" storing references to Github Suppor requests/ticket filed to GitHub Team and links to future changes planned by GitHub Actions that solve our pain points.

• Spelling mistakes in expression language are silently ignored and the values are replaced with an empty string. This has bitten us more than once already. @potiuk opened an issue https://support.github.com/ticket/personal/0/865380 to change it and fail on spelling mistakes.

• Merge button becomes "green" when pending checks have not yet started
  This issue is filed by @potiuk as ticket https://support.github.com/ticket/personal/0/857608. It's been acknowledged and files as a bug to GitHub Engineering

• No links between "workflow_run" and the originating workflow
  The issue is filed by @potiuk as ticket 'https://support.github.com/ticket/personal/0/859741. They acknowledged the request: "I have opened an internal customer feedback issue and passed your feature request along to the relevant teams" and it might get implemented. UPDATE: Not needed as we switched to pull_request_target worfklows.

• Jobs being often cancelled with 137 error. This seems to happen often .
  The issue is filed by @potiuk as https://support.github.com/ticket/personal/0/857606 . GitHub support advised us to enable verbose debugging for the project. The JIRA issue is created https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20945 to enable the debugging. Debugging enabled, they are looking into it. Continue looking after a week or so. The final answer: it's caused by exceeding number of jobs by whole ASF organization.

• No native cancelling of the workflows. @potiuk implemented custom "Cancel Workflow Runs" action that implements that feature https://github.com/marketplace/actions/cancel-workflow-runs and we are using it, but it has some limitations when the queue of jobs get really busy (the cancel step has to be executed in order to perform cancel but there might be not enough slotsl in the queue). @potiuk opened a ticket for that: https://support.github.com/ticket/personal/0/859753. Waiting for answer from GitHub support. UPDATE: Fixed by swtiching to newly added "concurrency" feature

Seems like this problem might be solved soon:

Hopefully we'll have improvements to share regarding this soon.

In the meantime, I'll go ahead and tentatively mark this ticket as solved, however please don't hesitate to reply if I can help with further questions about this request.

• Unknown Blob appear often when pushing to our Docker registry. @potiuk created https://support.github.com/ticket/personal/0/865404 to diagnose/solve it.
  Update from GH Support:

Our engineering team have identified that for the example provided in this ticket, there was a write failure to our storage provider:

failed to write entire blob: unexpected EOF; wrote: 244383744
Thank you for sharing your workaround. 
While we don't currently have an ETA on when a fix will be incorporated into the product on our side,
please keep an eye on our GitHub Changelog or the GitHub Blog which may reference this
fix should it become available at some point in the future.

Seems that this is a truly transient error on the GitHub side. The workaround #11411 introduced seems to completely solve the problem. UPDATE: Fixed as we switched to the new ghcr.io registry which does not have the problem.

[potiuk]

Another one - likely temporary - our queue of builds got stuck over the last 2-3 hours.

HIgh priority ticket opened: https://support.github.com/ticket/personal/0/860491

[ashb]

I wasn't happy with the answer we got from Github for the "137 exit code" ticket, so I've opened another ticket https://support.github.com/ticket/personal/0/880426 -- lets see if we get the same answer or not.

It just seems weird to me that they would cancel an in-progress job just because it hit a concurrency limit, when the rest of the time the behaviour we see on our jobs is that we just sit waiting in a queue.

[potiuk]

Cool. Just remember that this is single queue for ALL ASF projects. So unless you manually check all other ASF projects, we have no way of knowing what is the queue: I asked INFRA for that as well (at least to get better visibility and to know what causes the queue): https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20978 but they said they cannot help with it.

[potiuk]

So if you can also follow up with INFRA on that, it would be helpful