Scorecard limitations

[kittyandrew]

When looking through scorecard data, I've noticed some issues related to scorecard checks that lead to poor resulting data. Some of these problems are related to scorecard checks not being opinionated, which can be be argued against, but often it clearly leads to strange results.

Note: results here are converted to risk and are not pure scorecard results.

---

Case study github.com/kittyandrew/klocc-frontend

It seems that you can easily game/troll scorecard results by including fake files.

Example 1: License

if you talk about any security vulnerability we have, we will sue you!!

will result in low risk score (9/10 in original scorecard result):

{
    "name": "License",
    "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/7ed886f1bd917d19cb9d6ce6c10e80e81fa31c39/docs/checks.md#license",
        "short": "Determines if the project has defined a license."
    },
    "score": 1,
    "reason": "license file detected",
    "details": [
        "Info: License file found in expected location: LICENSE:1",
        "Warn: Any licence detected not an FSF or OSI recognized license: LICENSE:1"
    ]
}

Example 2: Security-Policy

if you talk about any security vulnerability we have, we will sue you!!
if you talk about any security vulnerability we have, we will sue you!!
if you talk about any security vulnerability we have, we will sue you!!

causes scorecard to incorrectly identify some properties of the SECURITY.md file:

{
    "name": "Security-Policy",
    "documentation": {
        "url": "https://github.com/ossf/scorecard/blob/7ed886f1bd917d19cb9d6ce6c10e80e81fa31c39/docs/checks.md#security-policy",
        "short": "Determines if the project has published a security policy."
    },
    "score": 6,
    "reason": "security policy file detected",
    "details": [
        "Info: security policy file detected: SECURITY.md:1",
        "Warn: no linked content found: On GitHub:\nEnable private vulnerability disclosure in your repository settings https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository\nAdd a section in your SECURITY.md indicating you have enabled private reporting, and tell them to follow the steps in https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability to report vulnerabilities.\nOn GitLab:\nProvide a point of contact in your SECURITY.md.\nExamples: https://github.com/ossf/scorecard/blob/main/SECURITY.md, https://github.com/slsa-framework/slsa-github-generator/blob/main/SECURITY.md, https://github.com/sigstore/.github/blob/main/SECURITY.md. (Low effort)",
        "Info: Found disclosure, vulnerability, and/or timelines in security policy: SECURITY.md:1",
        "Info: Found text in security policy: SECURITY.md:1"
    ]
}

---

Case study github.com/ansible/ansible

Example 1: Security-Policy

Sometimes security policy (and potentially other key files) may not be detected at all, while they exist and are configured properly. Reported issue: ossf/scorecard#2489 (comment)

{
    "name": "Security-Policy",
    "documentation": {
        "shortDescription": "Determines if the project has published a security policy.",
        "url": "https://github.com/ossf/scorecard/blob/e780e089f512f12cd1fc3d090a2424f186ac1a78/docs/checks.md#security-policy"
    },
    "score": 10.0,
    "reason": "security policy file not detected",
    "details": [
        "Warn: no security policy file detected",
        "Warn: no security file to analyze",
        "Warn: no security file to analyze",
        "Warn: no security file to analyze"
    ]
}

---

Case study github.com/pytorch/pytorch

Example 1: Signed-Releases

You can see here that pytorch receives 0 in scorecard (highest risk here) for not signing releases, while in reality releases provided by pytorch mostly contain source code and are signed, where signature is on the commit that makes it to release and is recognized by github: https://github.com/pytorch/pytorch/releases

{
    "name": "Signed-Releases",
    "documentation": {
        "shortDescription": "Determines if the project cryptographically signs release artifacts.",
        "url": "https://github.com/ossf/scorecard/blob/153e06d99fed8ff950ac1c5b881e374020b520fb/docs/checks.md#signed-releases"
    },
    "score": 10.0,
    "reason": "Project has not signed or included provenance with any releases.",
    "details": [
        "Warn: release artifact v2.2.2 not signed: https://api.github.com/repos/pytorch/pytorch/releases/148742991",
        "Warn: release artifact v2.2.1 not signed: https://api.github.com/repos/pytorch/pytorch/releases/143208965",
        "Warn: release artifact v2.2.0 not signed: https://api.github.com/repos/pytorch/pytorch/releases/139160052",
        "Warn: release artifact v2.1.2 not signed: https://api.github.com/repos/pytorch/pytorch/releases/133972717",
        "Warn: release artifact v2.1.1 not signed: https://api.github.com/repos/pytorch/pytorch/releases/129423939",
        "Warn: release artifact v2.2.2 does not have provenance: https://api.github.com/repos/pytorch/pytorch/releases/148742991",
        "Warn: release artifact v2.2.1 does not have provenance: https://api.github.com/repos/pytorch/pytorch/releases/143208965",
        "Warn: release artifact v2.2.0 does not have provenance: https://api.github.com/repos/pytorch/pytorch/releases/139160052",
        "Warn: release artifact v2.1.2 does not have provenance: https://api.github.com/repos/pytorch/pytorch/releases/133972717",
        "Warn: release artifact v2.1.1 does not have provenance: https://api.github.com/repos/pytorch/pytorch/releases/129423939"
    ]
}

Example 2: Pinned-Dependencies

Project seems to receive the highest risk score for a few unpinned workflow test dependencies. This is more an opinion, but seems to inaccurately represent state of pinned dependencies in the repository, although it might be difficult to verify if actions performed in those workflows produce any artifacts or change repository code.

{
    "name": "Pinned-Dependencies",
    "documentation": {
        "shortDescription": "Determines if the project has declared and pinned the dependencies of its build process.",
        "url": "https://github.com/ossf/scorecard/blob/153e06d99fed8ff950ac1c5b881e374020b520fb/docs/checks.md#pinned-dependencies"
    },
    "score": 10.0,
    "reason": "dependency not pinned by hash detected -- score normalized to 10.0",
    "details": [
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:40",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:62",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:68",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:75",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:80",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-build-test.yml:144",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-full-build-test.yml:62",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-full-build-test.yml:68",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-full-build-test.yml:75",
        "Warn: third-party GitHubAction not pinned by hash: .github/workflows/_android-full-build-test.yml:80"
    ]
}