fix: fix get failured checks #28

nodece · 2022-04-07T16:57:23Z

Signed-off-by: Zixuan Liu nodeces@gmail.com

Motivation

I tried to rerun the failed test, but the failed test cannot be rerun. I checked the https://github.com/apache/pulsar/runs/5869256539?check_suite_focus=true#step:5:314, no tests were run.

Using https://docs.github.com/en/rest/reference/actions#list-workflow-runs-for-a-repository is not working, this API cannot find my actions, it seems to fetch all actions.

You can use this command to reproduce:

PR_USER=nodece PR_BRANCH=update_spring_library status=failure curl -s -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/apache/pulsar/actions/runs?actor=${PR_USER}&branch=${PR_BRANCH}&status=${status}&per_page=100"

If we use the old way to get the failed checks, it is exactly:

curl -s -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/apache/pulsar/commits/70266323cc7784c66c35044d64ef87e593559b7d/check-runs?per_page=1"

{
  "total_count": 57,
  "check_runs": [
    {
      "id": 5867399574,
      "name": "Unit-PROXY Tests",
      "node_id": "CR_kwDOA7PXtM8AAAABXblplg",
      "head_sha": "70266323cc7784c66c35044d64ef87e593559b7d",
      "external_id": "",
      "url": "https://api.github.com/repos/apache/pulsar/check-runs/5867399574",
      "html_url": "https://github.com/apache/pulsar/runs/5867399574",
      "details_url": "https://github.com/apache/pulsar/runs/5867399574",
      "status": "completed",
      "conclusion": "success",
      "started_at": "2022-04-07T11:22:16Z",
      "completed_at": "2022-04-07T11:22:16Z",
      "output": {
        "title": "Unit-PROXY Tests ✔️",
        "summary": "![Tests passed successfully](https://img.shields.io/badge/tests-92%20passed-success)\n|Report|Passed|Failed|Skipped|Time|\n|:---|---:|---:|---:|---:|\n|[TEST-org.apache.pulsar.proxy.extensions.ProxyExtensionsTest.xml](https://github.com/apache/pulsar/runs/5867399574#r0)|5✔️|||3s|\n|[TEST-org.apache.pulsar.proxy.extensions.ProxyExtensionUtilsTest.xml](https://github.com/apache/pulsar/runs/5867399574#r1)|3✔️|||471ms|\n|[TEST-org.apache.pulsar.proxy.extensions.ProxyExtensionWithClassLoaderTest.xml](https://github.com/apache/pulsar/runs/5867399574#r2)|2✔️|||1s|\n|[TEST-org.apache.pulsar.proxy.extensions.SimpleProxyExtensionSeparateThreadPoolTest.xml](https://github.com/apache/pulsar/runs/5867399574#r3)|1✔️|||27ms|\n|[TEST-org.apache.pulsar.proxy.extensions.SimpleProxyExtensionSharedThreadPoolTest.xml](https://github.com/apache/pulsar/runs/5867399574#r4)|1✔️|||24ms|\n|[TEST-org.apache.pulsar.proxy.server.AdminProxyHandlerTest.xml](https://github.com/apache/pulsar/runs/5867399574#r5)|3✔️|||1s|\n|[TEST-org.apache.pulsar.proxy.server.AuthedAdminProxyHandlerTest.xml](https://github.com/apache/pulsar/runs/5867399574#r6)|1✔️|||7s|\n|[TEST-org.apache.pulsar.proxy.server.BrokerProxyValidatorTest.xml](https://github.com/apache/pulsar/runs/5867399574#r7)|7✔️|||2s|\n|[TEST-org.apache.pulsar.proxy.server.FunctionWorkerRoutingTest.xml](https://github.com/apache/pulsar/runs/5867399574#r8)|1✔️|||5s|\n|[TEST-org.apache.pulsar.proxy.server.InvalidProxyConfigForAuthorizationTest.xml](https://github.com/apache/pulsar/runs/5867399574#r9)|1✔️|||102ms|\n|[TEST-org.apache.pulsar.proxy.server.ProxyAdditionalServletTest.xml](https://github.com/apache/pulsar/runs/5867399574#r10)|1✔️|||920ms|\n|[TEST-org.apache.pulsar.proxy.server.ProxyAuthenticatedProducerConsumerTest.xml](https://github.com/apache/pulsar/runs/5867399574#r11)|1✔️|||33s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyAuthenticationTest.xml](https://github.com/apache/pulsar/runs/5867399574#r12)|1✔️|||6s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyConfigurationTest.xml](https://github.com/apache/pulsar/runs/5867399574#r13)|2✔️|||372ms|\n|[TEST-org.apache.pulsar.proxy.server.ProxyConnectionTest.xml](https://github.com/apache/pulsar/runs/5867399574#r14)|1✔️|||9ms|\n|[TEST-org.apache.pulsar.proxy.server.ProxyConnectionThrottlingTest.xml](https://github.com/apache/pulsar/runs/5867399574#r15)|1✔️|||5s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyEnableHAProxyProtocolTest.xml](https://github.com/apache/pulsar/runs/5867399574#r16)|1✔️|||3s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyForwardAuthDataTest.xml](https://github.com/apache/pulsar/runs/5867399574#r17)|1✔️|||69s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyIsAHttpProxyTest.xml](https://github.com/apache/pulsar/runs/5867399574#r18)|10✔️|||3s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyParserTest.xml](https://github.com/apache/pulsar/runs/5867399574#r19)|5✔️|||10s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyPrometheusMetricsTest.xml](https://github.com/apache/pulsar/runs/5867399574#r20)|1✔️|||976ms|\n|[TEST-org.apache.pulsar.proxy.server.ProxyRolesEnforcementTest.xml](https://github.com/apache/pulsar/runs/5867399574#r21)|1✔️|||9s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyServiceStarterTest.xml](https://github.com/apache/pulsar/runs/5867399574#r22)|3✔️|||3s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyServiceTlsStarterTest.xml](https://github.com/apache/pulsar/runs/5867399574#r23)|2✔️|||3s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyStatsTest.xml](https://github.com/apache/pulsar/runs/5867399574#r24)|3✔️|||6s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyTest.xml](https://github.com/apache/pulsar/runs/5867399574#r25)|7✔️|||14s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyTlsTest.xml](https://github.com/apache/pulsar/runs/5867399574#r26)|2✔️|||5s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyWithAuthorizationNegTest.xml](https://github.com/apache/pulsar/runs/5867399574#r27)|1✔️|||6s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyWithAuthorizationTest.xml](https://github.com/apache/pulsar/runs/5867399574#r28)|13✔️|||86s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyWithJwtAuthorizationTest.xml](https://github.com/apache/pulsar/runs/5867399574#r29)|3✔️|||14s|\n|[TEST-org.apache.pulsar.proxy.server.ProxyWithoutServiceDiscoveryTest.xml](https://github.com/apache/pulsar/runs/5867399574#r30)|1✔️|||39s|\n|[TEST-org.apache.pulsar.proxy.server.SuperUserAuthedAdminProxyHandlerTest.xml](https://github.com/apache/pulsar/runs/5867399574#r31)|3✔️|||19s|\n|[TEST-org.apache.pulsar.proxy.server.UnauthedAdminProxyHandlerTest.xml](https://github.com/apache/pulsar/runs/5867399574#r32)|2✔️|||405ms|\n|[TEST-org.apache.pulsar.proxy.util.CmdTest.xml](https://github.com/apache/pulsar/runs/5867399574#r33)|1✔️|||170ms|",
        "text": null,
        "annotations_count": 0,
        "annotations_url": "https://api.github.com/repos/apache/pulsar/check-runs/5867399574/annotations"
      },
      "check_suite": {
        "id": 5968925454
      },
      "app": {
        "id": 15368,
        "slug": "github-actions",
        "node_id": "MDM6QXBwMTUzNjg=",
        "owner": {
          "login": "github",
          "id": 9919,
          "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=",
          "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/github",
          "html_url": "https://github.com/github",
          "followers_url": "https://api.github.com/users/github/followers",
          "following_url": "https://api.github.com/users/github/following{/other_user}",
          "gists_url": "https://api.github.com/users/github/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/github/subscriptions",
          "organizations_url": "https://api.github.com/users/github/orgs",
          "repos_url": "https://api.github.com/users/github/repos",
          "events_url": "https://api.github.com/users/github/events{/privacy}",
          "received_events_url": "https://api.github.com/users/github/received_events",
          "type": "Organization",
          "site_admin": false
        },
        "name": "GitHub Actions",
        "description": "Automate your workflow from idea to production",
        "external_url": "https://help.github.com/en/actions",
        "html_url": "https://github.com/apps/github-actions",
        "created_at": "2018-07-30T09:30:17Z",
        "updated_at": "2019-12-10T19:04:12Z",
        "permissions": {
          "actions": "write",
          "administration": "read",
          "checks": "write",
          "contents": "write",
          "deployments": "write",
          "discussions": "write",
          "issues": "write",
          "metadata": "read",
          "organization_packages": "write",
          "packages": "write",
          "pages": "write",
          "pull_requests": "write",
          "repository_hooks": "write",
          "repository_projects": "write",
          "security_events": "write",
          "statuses": "write",
          "vulnerability_alerts": "read"
        },
        "events": [
          "branch_protection_rule",
          "check_run",
          "check_suite",
          "create",
          "delete",
          "deployment",
          "deployment_status",
          "discussion",
          "discussion_comment",
          "fork",
          "gollum",
          "issues",
          "issue_comment",
          "label",
          "milestone",
          "page_build",
          "project",
          "project_card",
          "project_column",
          "public",
          "pull_request",
          "pull_request_review",
          "pull_request_review_comment",
          "push",
          "registry_package",
          "release",
          "repository",
          "repository_dispatch",
          "status",
          "watch",
          "workflow_dispatch",
          "workflow_run"
        ]
      },
      "pull_requests": [

      ]
    }
  ]
}

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

lhotari

There's a misunderstanding. The new CI requires using the new API for re-running just failed jobs and not the complete workflow. Please report what is broken first and we can think of a solution.

lhotari · 2022-04-07T18:38:20Z

I explained some of the changes in https://lists.apache.org/thread/n05xp17krb4f3cc0jjkh65ytqk1w6130 .
"The limitation is that you need to wait for all jobs to complete before failed jobs can be re-run."

You can also provide feedback on the mailing list.

lhotari · 2022-04-07T18:43:23Z

With the new CI there's another case where you see a red mark. It's when flaky tests fail, but later pass in a retry, the failed test result will get recorded to a test report. That doesn't block merging, but will show up so that the failures can be inspected. It's a bit confusing since GitHub Actions has a bug that the test reports get attached randomly to a workflow when there are multiple workflows executing. I'll write to the dev list explaining this since I have received a few questions about this.

It might appear that re-running doesn't work because of the new test reports. Those failures won't clear off, but they don't block merging either.

lhotari · 2022-04-07T18:47:30Z

In your case, it's the flaky tests report: https://github.com/apache/pulsar/pull/15065/checks?check_run_id=5867398812 .

nodece · 2022-04-08T03:05:46Z

@lhotari Thanks for your explanation!

There's a misunderstanding. The new CI requires using the new API for re-running just failed jobs and not the complete workflow. Please report what is broken first and we can think of a solution.

I have a question about the get_runs failure don't seem to get the correct data, this data includes the conclusion: success item. Do you notice this?

I explained some of the changes in https://lists.apache.org/thread/n05xp17krb4f3cc0jjkh65ytqk1w6130 .
"The limitation is that you need to wait for all jobs to complete before failed jobs can be re-run."

You can also provide feedback on the mailing list.

This is detailed.

With the new CI there's another case where you see a red mark. It's when flaky tests fail, but later pass in a retry, the failed test result will get recorded to a test report. That doesn't block merging, but will show up so that the failures can be inspected. It's a bit confusing since GitHub Actions has a bug that the test reports get attached randomly to a workflow when there are multiple workflows executing. I'll write to the dev list explaining this since I have received a few questions about this.

It might appear that re-running doesn't work because of the new test reports. Those failures won't clear off, but they don't block merging either.

This is confusing, I'm looking forward your solution.

lhotari · 2022-04-08T03:20:55Z

I have a question about the get_runs failure don't seem to get the correct data, this data includes the conclusion: success item. Do you notice this?

Yes. That happens when an actual build run hasn't failed. As explained in previous comments, the test reports added to the build are causing this.

The benefit of seeing the failed test reports is that we can start paying more attention to the flaky test problem and fixing them. Flaky tests might be indicating real production code issues.

This is confusing, I'm looking forward your solution.

There's really no other solution than documenting this. We could remove the rendered test reports, but I think that integrating test reports to GitHub Actions UI is useful.
GitHub Actions UI has a bug that test reports get attached to wrong workflows when there's more than one workflow that gets triggered by a build job. Perhaps some day GitHub fixes this issue and the test reports can be shown in the correct location, attached to each build job. That would help resolve the issue since the test reports wouldn't be a separate check and instead the test reports could be attached to the build job that runs the tests. That's currently not possible because of the GitHub Actions bug.

lhotari · 2022-04-14T09:36:36Z

There's more details in https://lists.apache.org/thread/60x7sqg2p4mlssj5jtow6zwq3jksf6w3 .

Currently there's an issue with GitHub Actions that the workflow run search doesn't return all results and that's why rerunning failed jobs doesn't work in some cases.

lhotari · 2022-04-21T08:58:46Z

I have fixed the rerunning in cases where the PR's head sha doesn't change (for example closing and reopening the PR to trigger a new build): #33

fix: fix get failed checks

c8be027

Signed-off-by: Zixuan Liu <nodeces@gmail.com>

nodece mentioned this pull request Apr 7, 2022

Use the new GitHub API to re-run failed jobs #27

Merged

lhotari requested changes Apr 7, 2022

View reviewed changes

lhotari closed this Apr 7, 2022

lhotari mentioned this pull request Apr 8, 2022

[fix][broker] Fix potential to add duplicated consumer apache/pulsar#15051

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fix get failured checks #28

fix: fix get failured checks #28

nodece commented Apr 7, 2022 •

edited

Loading

lhotari left a comment

lhotari commented Apr 7, 2022 •

edited

Loading

lhotari commented Apr 7, 2022

lhotari commented Apr 7, 2022

nodece commented Apr 8, 2022

lhotari commented Apr 8, 2022

lhotari commented Apr 14, 2022

lhotari commented Apr 21, 2022

fix: fix get failured checks #28

fix: fix get failured checks #28

Conversation

nodece commented Apr 7, 2022 • edited Loading

Motivation

lhotari left a comment

Choose a reason for hiding this comment

lhotari commented Apr 7, 2022 • edited Loading

lhotari commented Apr 7, 2022

lhotari commented Apr 7, 2022

nodece commented Apr 8, 2022

lhotari commented Apr 8, 2022

lhotari commented Apr 14, 2022

lhotari commented Apr 21, 2022

nodece commented Apr 7, 2022 •

edited

Loading

lhotari commented Apr 7, 2022 •

edited

Loading