Feature: Error on projects without a permissible license

[jpmcb]

Type of feature

🍕 Feature

Current behavior

Currently, on the alpha branch, any and all projects will be indexed. This includes project without permissible licenses (i.e., projects with hard copy left licenses). There is no way to filter based on the project's license.

Suggested solution

We should find a way to dynamically check licenses of projects that are being queried / ingested:

1. Get the zip for the project in question
2. Parse the files looking for likely license files (LICENSE, license.txt, MIT-LICENSE, etc.) - ideally, we'd use a well known crate for this. Maybe we can use what cargo deny uses internally to check for licenses
3. Only allow projects that have permissible licenses to be indexed into the vector db.
4. If the license is not permissible, reject the request and return early with an error.

Additional context

See comments in #5 (comment)

Code of Conduct

• I agree to follow this project's Code of Conduct

Contributing Docs

• I agree to follow this project's Contribution Docs

[Anush008]

If a repository happens to have no licence specified, which means that it doesn't allow for modification and redistribution, do we proceed to index it?

[Anush008]

We can add a check using the https://api.github.com/repos/open-sauced/repo-query/license endpoint before downloading the repo zip.

[Anush008]

The list of license keys is available here.
https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository#searching-github-by-license-type

@bdougie, I've derived a some license keys from the above that I believe are permissive.
apache, mit, wtfpl, zlib, bsd, unlicensed. Will these be fine?

[jpmcb]

If a repository happens to have no licence specified, which means that it doesn't allow for modification and redistribution, do we proceed to index it?

Right. For now, we should err on the side of caution and not proceed if there isn't a license.

[jpmcb]

I've derived a some license keys from the above that I believe are permissive.
apache, mit, wtfpl, zlib, bsd, unlicensed. Will these be fine?

Those all look good to me: most software licenses have specific clauses for distribution and as long as we aren't modifying the actual source when getting it from GitHub, there shouldn't be any problem.

I'd say go ahead with those as a starting point and we can explore from there!

[Anush008]

Right. I'm working on it.