Introduce notion of Package set

[JonoYang]

We have the case where we have different instances of the same package. For example, for a maven Package, we can have the source, test, or doc JAR for that package. These JARs are part of the same package but would be considered different packages in the PackageDB, as they have different download URLs. These JARS would have similar purl values (same type, namespace, name, and version) but different subpath and qualifiers. We want to be able to relate this group of Packages together such that we can combine all the findings from the different varieties of a Package and return the data to the user.

After some discussion with @pombredanne, we will add two new fields to the Package model:

• package_set (working name)
  • This field contains a uuid or unique identifier for a group of packages. This ID would be unique to a particular purl (type, namespace, name, version) combination. The ID is generated when a Package with a purl that is not yet in the PackageDB is created. When another package with the same type, namespace, name, and version is created, the ID would also be used here.
• package_content
  • This field contains a tag that describes what is in this package, either source, binary, doc, tests, etc.

There would be an API endpoint (or equivalent), where we query for a package, then we would combine the metadata based on some set precedence (curated package data, source data data, then binary, etc) and return the combined package data to the user.

@pombredanne @DennisClark

I'd like to hear your thoughts on grouping like packages via ID

[DennisClark]

@JonoYang Package Set is a great idea. I think it would help to get some concrete examples of the proposed new field package_content since I am having trouble visualizing the detail values in a such a field.

[JonoYang]

@DennisClark

There are multiple JARs of log4j-core v2.0 at https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.0/ ,

We would create an entry in the PackageDB for each JAR. The tag in package_content the category the files within it fall under. log4j-core-2.0-javadoc.jar would have a package_content value of doc, log4j-core-2.0-sources.jar would be source, log4j-core-2.0-tests.jar would be test, and log4j-core-2.0.jar would be binary.

[DennisClark]

@JonoYang thanks -- my original problem was thinking that package_content referred to the whole set, but instead we are planning to choose a specific label for each jar -- that makes a lot of sense!

[DennisClark]

@JonoYang and of course, I also wondered if the the package_set concept might be applied to other technologies other than Java, where a group of packages might be considered part of a set.

[pombredanne]

@JonoYang and of course, I also wondered if the the package_set concept might be applied to other technologies other than Java, where a group of packages might be considered part of a set.

@DennisClark I think so. For instance, these could be sets:

• all the different wheels of an lxml release and the source tarballs: https://pypi.org/project/lxml/#files
• all the binary packages of Python 3.7 in Debian as well as the "orig" source and the "debian" patches https://packages.debian.org/buster/libpython3.7
• a source rpm and all its binaries
• a npm tarball and its GitHub source repo

[pombredanne]

@JonoYang you wrote:

There would be an API endpoint (or equivalent), where we query for a package, then we would combine the metadata based on some set precedence (curated package data, source data data, then binary, etc) and return the combined package data to the user.

I guess some examples of how we could combine metadata could be either generic or package type specific or even package name or PURL-specific.

• a generic way license may flow through may be: Git repo -> source archive -> binary build
• a type-specific way may be: maven JAR -> maven binary JAR

[AyanSinhaMahapatra]

Btw, adding another example for this: https://repo1.maven.org/maven2/commons-daemon/commons-daemon/1.3.3/
We have a lot of variations of java source jars, native jars and other type of packages here