Skip to content

[BUG] Leaderboard CI validation fails with improper ORG_REPO url extraction logic #295

@kami619

Description

@kami619

Bug Description

The leaderboard validation workflow fails for any submission where the assessed repository's .repository.url field uses the SSH format (git@github.com:org/repo.git) instead of HTTPS (https://github.com/org/repo.git). The ORG_REPO extraction uses sed that only strips https://github.com/ prefixes, leaving SSH URLs unparsed and causing downstream API calls to fail with invalid repo paths.
## To Reproduce

  1. Clone a repository that has an SSH remote (e.g. git@github.com:feast-dev/feast.git)
  2. Run agentready assess against it
  3. Run agentready submit to create a leaderboard PR
  4. Observe the "Verify submitter has access" step fails with Not Found (HTTP 404)

Expected Behavior

The workflow should correctly parse both SSH (git@github.com:org/repo.git) and HTTPS (https://github.com/org/repo.git) URLs from the assessment JSON, extract org/repo, and successfully validate repository access.

Actual Behavior

The sed command passes the raw SSH URL through unchanged, resulting in API calls to invalid paths like /repos/git@github.com:feast-dev/feast/collaborators/username, which return 404. The workflow then incorrectly reports the submitter has no access to the repository.

Additionally, the "Post validation results" step fails with a 403 (Resource not accessible by integration) on fork PRs because the pull_request trigger provides a read-only GITHUB_TOKEN that cannot post comments.

Environment

  • OS: Ubuntu 24.04 (GitHub Actions runner)
  • Version: 2.27.4
  • Python Version: 3.12

Additional Context

Affected PRs:

Both are cross-repository fork PRs. The failures are not related to the submitters' actual access to the target repositories. Previous submissions likely used HTTPS remotes, so this code path was never exercised until now.

Possible Solution

Fix (blocker): Add SSH URL handling to the sed pipeline in both the "Verify repository exists" and "Verify submitter has access" steps:

ORG_REPO=$(echo "$REPO_URL" | sed 's|git@github.com:||' | sed 's|https://github.com/||' | sed 's|.git$||')

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingreleased

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions