Reorganize API endpoints

[JonoYang]

We have a lot of API endpoints exposed that are not used and/or unnecessary. These endpoints should be removed:

approximate_directory_content_index
approximate_directory_structure_index
exact_file_index
exact_package_archive_index
cditems
on_demand_queue

The matching related endpoints, approximate_directory_content_index, approximate_directory_structure_index, exact_file_index, and exact_package_archive_index, are being removed because the current D2D pipeline only uses sha1 lookups on the packages and resources endpoints. We also plan to revamp matching such that it is all done on the purldb server.

We should reorganize the API endpoints into three categories:

1. lookup (query, get)
2. match (post)
3. "mining" (live retrieval of things)

We also need to rethink the actions on the packages endpoint. The get_or_fetch_package endpoint should be merged with the get_package endpoint. get_package should also be updated to create a package from a download url, in addition to purls.

[mjherzog]

Yes please

[pombredanne]

@keshav ping ... can you add your research details here?

[keshav-space]

@keshav ping ... can you add your research details here?

@pombredanne

PurlDB - Streamline the API

This exercise aims to make the consumption of PurlDB simpler and more obvious.

Summary Page 

Minimal summary page similar to https://mvnrepository.com/ ?

• Total # of packages and resources indexed
• Package breakdown by Ecosystem
• Package breakdown by License?

API documentation - /api/docs

Minimal API documentaion using ReDoc or Swagger.

Public API - /api

1. /api/package - GET

   1. Request

      • Filters: (Purl/s, Checksum/s, Purl fields, download URL) 

      • Enhance using PackageSet: (default = False)

   2. Response

      • List of Packages

2. /api/collect - (Endpoint for registering On Demand tasks)

   1. /api/collect - GET

      • Request:

        • A PURL

      • Response:

        • Package metadata

      Return the package metadata, if not present index and return the metadata.

   2. /api/collect/index_package - POST

      • Request:

        • List containing PURLs/ Versionless PURLs/ Versionless PURLs along with Vers Range

        • Depth of Indexing ( Version/ Metadata/ Scan)

        • Flag: reIndex (default=False)

      • Response:

        • List of PURLs successfully queued for indexing

        • List of PURLs can’t be queued (Not supported or incorrect PURL/s)

3. /api/match - (Matching endpoint based on Resource/Directory matching) - POST

   1. Request:

      • ScanCode scan

      • Webhook

   2. Response:

      • Matching-id if matching is successfully started else error message

   3. Call the webhook with enhanced scan result

   4. /api/match/<matching-id> - Shows the status and result of match instance

The `/api/match` endpoint takes an existing scancode scan and runs a specialized pipeline that creates a project using the input scan in companion SCIO. It enhances the project with a package/resource/approximate directory match. This SCIO instance will have direct access to the PurlDB database.

4. /api/validate -  GET

   1. Request:

      • A PURL

      • Flag: check existence (default=False)

   2. Response:

      • valid:  True/False

      • exists: Null/True/False

Take a PURL and validate if the syntax is valid PURL, additionally if the existence field is true check whether the particular package exists or not. 

Optionally we would also wnat to index these packages since people checking for a particular package are likely interested in it and may at some point want to get the package detail for the same.

See #217

Service API - /api/service

1. /api/service/status/ondemand - GET

   1. Status on # of tasks processed and # of tasks left for processing.

2. /api/service/status/batch - GET

   1. Status on batch process across different ecosystem

      • /api/service/status/batch/<ecosystem>