Feature: Create a scorecard API

[justaugustus]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

• Feature: use go-git  #1709

Tracking requests based on consumers:

• scorecard-action: Use Scorecard's Go APIs scorecard-action#107
  • ✨ cmd: Refactor to make importable #1696
  • ...
• allstar: Update Scorecard Interface allstar#21
  • ...
  • ...
• witness:
  • Public JSON Type: Feature: Public JSON Type #1673
  • ...

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

From @justaugustus in #1680 (comment):

I'm concerned about directing users here while we don't officially yet support Scorecard as a library. Apart from the LGTM.

I think we should start supporting it. There are projects using it.

This presentation is worth a look: https://dave.cheney.net/practical-go/presentations/gophercon-israel.html#apidesign

Of note:

If it’s exported, its part of your public API.

That said, we have a responsibility to design APIs that are hard to misuse.

* Scorecard-actions (this is WIP)

ossf/scorecard-action#107 (cc: @rohankh532)

* Witness

#1673 / #1245 / #1682 (cc: @colek42)

* Allstar

#1645 / ossf/allstar#114

#1645 just merged, which makes some decent steps in the right direction and I'll open a separate tracking issue for this if one doesn't already exist.

We should document that we don't support Semver but scorecard/v4 similar to Go GitHub Libary.

Disagree here and I've expressed the opinion in a few places, including the most recent (2022-02-24) biweekly meeting.

We've adopted SemVer-compliant versioning, which means regardless if we've enforce SemVer or not, consumers will have at least a baseline expectation that we are compliant. Whether that is true or not is a different story.

I think we should instead declare that the next major release version (v5.0.0) is the first release in which we will be SemVer-compliant and direct our efforts into designing the module in a way that we can ensure that.

I think we'll all agree that scorecard is a high-value project in a space with lots of eyes on it today, so we have a responsibility to those who consume it to make it as easy and clear to do so as possible.

cc: @ossf/scorecard-maintainers

[azeemshaikh38]

I think we'll all agree that scorecard is a high-value project in a space with lots of eyes on it today, so we have a responsibility to those who consume it to make it as easy and clear to do so as possible.

+1.

I think we should instead declare that the next major release version (#1490) is the first release in which we will be SemVer-compliant and direct our efforts into designing the module in a way that we can ensure that.

I would absolutely love to have this accomplished in v5. This might require a significant amount of thought and effort though. @justaugustus do you think we can accomplish this by v5 timeline (end of Q2 - early Q3)? If so, a big +1 on this and let us know how we can help contribute here.

[justaugustus]

This might require a significant amount of thought and effort though. @justaugustus do you think we can accomplish this by v5 timeline (end of Q2 - early Q3)?

For whatever reason, I often prefer to attack a problem by brute-forcing usage, instead of immediately trying to write a design doc (especially if the code is "newer"/has less guarantees/restrictions and doing the work will outstrip the value).

Our closest consumers will help refine the API.

With that in mind, I've already done the following:

• landed a set of refactors to make a few things more consumable: ✨ Miscellaneous refactors to ease downstream consumption #1645
• opened a PR in allstar to attempt to consume it: [WIP] policies: Add check for scorecard policies allstar#114

I'm also taking a look at the scorecard-action: ossf/scorecard-action#107
One of the repeated requests on that issue was "remove the env vars" and some of that is already possible with the new options package: https://github.com/ossf/scorecard/blob/d71866ca16b41cc482ddc656c5f3884d6c2e00ef/options/options.go

If so, a big +1 on this and let us know how we can help contribute here.

Absolutely! Will keep you posted :)

[colek42]

In witness we can simplify the UX around attesting scorecards with a public API. We are looing forward to this feature.

[justaugustus]

In witness we can simplify the UX around attesting scorecards with a public API. We are looking forward to this feature.

That's awesome, @colek42!
Could you say more about what specifically would be helpful to have exposed?

[colek42]

We would want to embed the scorecard into an attestor so the user would not have to maintain it is a separate tool.

[justaugustus]

We would want to embed the scorecard into an attestor so the user would not have to maintain it is a separate tool.

@colek42 -- Given https://github.com/testifysec/witness is cobra, this is maybe useful --> #1696
Example of what I'm working on in scorecard-action to wrap/remix the scorecard cobra.Command: https://github.com/justaugustus/scorecard-action/blob/5a922336081787b82a6cbf4018780ce4a1a15fa8/entrypoint/entrypoint.go#L29-L112 / ossf/scorecard-action#122

Am I correct in saying that what we want long-term is really the logic of rootCmd

scorecard/cmd/root.go

Lines 74 to 163 in 241b0f4

	// rootCmd runs scorecard checks given a set of arguments.
	func rootCmd(o *options.Options) {
	// Set `repo` from package managers.
	pkgResp, err := fetchGitRepositoryFromPackageManagers(o.NPM, o.PyPI, o.RubyGems)
	if err != nil {
	log.Panic(err)
	}
	if pkgResp.exists {
	o.Repo = pkgResp.associatedRepo
	}
	pol, err := policy.ParseFromFile(o.PolicyFile)
	if err != nil {
	log.Panicf("readPolicy: %v", err)
	}
	ctx := context.Background()
	logger := sclog.NewLogger(sclog.ParseLevel(o.LogLevel))
	repoURI, repoClient, ossFuzzRepoClient, ciiClient, vulnsClient, err := checker.GetClients(
	ctx, o.Repo, o.Local, logger)
	if err != nil {
	log.Panic(err)
	}
	defer repoClient.Close()
	if ossFuzzRepoClient != nil {
	defer ossFuzzRepoClient.Close()
	}
	// Read docs.
	checkDocs, err := docs.Read()
	if err != nil {
	log.Panicf("cannot read yaml file: %v", err)
	}
	var requiredRequestTypes []checker.RequestType
	if o.Local != "" {
	requiredRequestTypes = append(requiredRequestTypes, checker.FileBased)
	}
	if !strings.EqualFold(o.Commit, clients.HeadSHA) {
	requiredRequestTypes = append(requiredRequestTypes, checker.CommitBased)
	}
	enabledChecks, err := policy.GetEnabled(pol, o.ChecksToRun, requiredRequestTypes)
	if err != nil {
	log.Panic(err)
	}
	if o.Format == options.FormatDefault {
	for checkName := range enabledChecks {
	fmt.Fprintf(os.Stderr, "Starting [%s]\n", checkName)
	}
	}
	repoResult, err := pkg.RunScorecards(
	ctx,
	repoURI,
	o.Commit,
	o.Format == options.FormatRaw,
	enabledChecks,
	repoClient,
	ossFuzzRepoClient,
	ciiClient,
	vulnsClient,
	)
	if err != nil {
	log.Panic(err)
	}
	repoResult.Metadata = append(repoResult.Metadata, o.Metadata...)
	// Sort them by name
	sort.Slice(repoResult.Checks, func(i, j int) bool {
	return repoResult.Checks[i].Name < repoResult.Checks[j].Name
	})
	if o.Format == options.FormatDefault {
	for checkName := range enabledChecks {
	fmt.Fprintf(os.Stderr, "Finished [%s]\n", checkName)
	}
	fmt.Println("\nRESULTS\n-------")
	}
	resultsErr := pkg.FormatResults(
	o,
	&repoResult,
	checkDocs,
	pol,
	)
	if resultsErr != nil {
	log.Panicf("Failed to output results: %v", err)
	}
	}

Pushed into pkg.RunScorecards

scorecard/pkg/scorecard.go

Lines 77 to 123 in 241b0f4

	// RunScorecards runs enabled Scorecard checks on a Repo.
	func RunScorecards(ctx context.Context,
	repo clients.Repo,
	commitSHA string,
	raw bool,
	checksToRun checker.CheckNameToFnMap,
	repoClient clients.RepoClient,
	ossFuzzRepoClient clients.RepoClient,
	ciiClient clients.CIIBestPracticesClient,
	vulnsClient clients.VulnerabilitiesClient) (ScorecardResult, error) {
	if err := repoClient.InitRepo(repo, commitSHA); err != nil {
	// No need to call sce.WithMessage() since InitRepo will do that for us.
	//nolint:wrapcheck
	return ScorecardResult{}, err
	}
	defer repoClient.Close()
	commitSHA, err := getRepoCommitHash(repoClient)
	if err != nil {
	return ScorecardResult{}, err
	}
	ret := ScorecardResult{
	Repo: RepoInfo{
	Name: repo.URI(),
	CommitSHA: commitSHA,
	},
	Scorecard: ScorecardInfo{
	Version: GetSemanticVersion(),
	CommitSHA: GetCommit(),
	},
	Date: time.Now(),
	}
	resultsCh := make(chan checker.CheckResult)
	if raw {
	go runEnabledChecks(ctx, repo, &ret.RawResults, checksToRun, repoClient, ossFuzzRepoClient,
	ciiClient, vulnsClient, resultsCh)
	} else {
	go runEnabledChecks(ctx, repo, nil, checksToRun, repoClient, ossFuzzRepoClient,
	ciiClient, vulnsClient, resultsCh)
	}
	for result := range resultsCh {
	ret.Checks = append(ret.Checks, result)
	}
	return ret, nil
	}

???

(Because that's what I was considering working on next :))

[github-actions]

This issue is stale because it has been open for 60 days with no activity.