Add API endpoint to request indexing for PURLs

[pombredanne]

The current code is only a command line tool. We should have either:

1. an explicit API endpoint to request mining and indexing of specific PURLs and or URLs
2. an implicit process triggered when there is a matching request

[JonoYang]

@pombredanne

For 2, I am wondering about the case where a person is filtering for a package using the PackageDB API and purl values, and the package does not exist in the PackageDB. Would we update the API in PackageDB to add that package to the on-demand queue? Or would this be a new API endpoint in minecode/matchcode where someone can look up packages by purl and then purldb would fetch the package information for packages it doesn't have based on the requests it gets?

[pombredanne]

@pombredanne

For 2, I am wondering about the case where a person is filtering for a package using the PackageDB API and purl values, and the package does not exist in the PackageDB. Would we update the API in PackageDB to add that package to the on-demand queue?

Or would this be a new API endpoint in minecode/matchcode where someone can look up packages by purl and then purldb would fetch the package information for packages it doesn't have based on the requests it gets?

@JonoYang the case where we trigger on demand retrieval should be limited to the case where we ask for a single whole and fully qualified PURL and this does not exist. It could be the existing endpoint or a new endpoint (likely better to have a new endpoint): your call. It feels like a new endpoint would be best where we go and try to fetch the package metadata synchronously.

And if a purl is otherwise requested in a batch or other endpoint and does not exist in the DB, then we would return nothing in this case, BUT we should kick some background job to try fetch these asynchronously in the background.

This way we have these cases:

1. an explicit request to get on-demand results now would return these results if they exist anywhere
2. a matching or other PURL lookup request would return whatever is in the DB and would attempt to later find them if they exist anywhere.

[armijnhemel]

Possibly related, but what also would be cool if this would be a web interface where someone could point to a URL of a package and indicate that it is interesting to be harvested (perhaps with a little bit of extra data for triage to avoid spam/malware being indexed?)

There are many projects which aren't on GitHub or other well known forges (but less well known forges, or completely independent hosting) or which are only released as tarballs on a website.

I would not expect a user to be able to create a valid PURL so making it a low barrier entry thing would be nice. I am not sure if I would make it completely automated (spam, malware, duplications, etc.).