Add support for on demand addon_pipeline in /api/collect endpoints

[keshav-space]

• /api/collect endpoint now supports optional list of addon_pipelines to run on the package.
  Example GET request:

    /api/collect/?purl=pkg:npm/foo@1.2.3&addon_pipelines=collect_symbols&addon_pipelines=inspect_elf_binaries

• In bulk indexing /api/collect/index_packages/ endpoint each package now also supports optional list of addon_pipelines to run on the package.
  Example POST request:

    {
        "packages": [
            {
                "purl": "pkg:npm/foobar",
                "vers": null,
                "source_purl": None,
                "addon_pipelines": ['inspect_elf_binaries', 'collect_symbols']
            }
        ]
        "reindex": true,
        "reindex_set": false,
    }

• fixes PurlDB: make symbols and strings indexing a separate, on demand option #376

Note

If no addon_pipeline is provided, then only the default pipelines will be run on the package.
https://github.com/nexB/purldb/blob/32ea6e810c3c371e24318440f079a7453f1c5d4a/minecode/model_utils.py#L30-L33

[JonoYang]

@keshav-space I just have a question about when the symbol collection pipelines would be run

[JonoYang]

Can you give me an example of when these pipelines would be run when indexing a package. I see how the default pipelines would be run, but where would we add the symbol collecting pipelines when we index a package?

[keshav-space]

@JonoYang
Suppose we receive this request: /api/collect/?purl=pkg:npm/foo@1.2.3&addon_pipelines=collect_symbols.

First, the CollectPackageSerializer will validate whether collect_symbols is a valid pipeline or not using SUPPORTED_ADDON_PIPELINES. Then, will add this package to the scan queue using add_package_to_scan_queue, with the pipeline argument being DEFAULT_PIPELINES + ( 'collect_symbols' ).

[JonoYang]

@keshav-space I see, do we currently use this anywhere? Otherwise, I think the code looks good unless there's something you want to change.

[keshav-space]

@JonoYang No more change from my side, it's ready for merge.