Adding minor versions: Versioning refactor proposal

[lumjjb]

Versioning refactor proposal

This design document covers a proposal for refactoring SPDX versioned structs with the goal of making the process of adding a new semantic minor version of the SPDX spec easier (e.g. with the coming introduction of 2.3).

Linked Issues:

• Reduce utility duplication by implementing document conversions #167
• Refactoring based on versions #165

Goals

1. Making the SPDX structs it more consumable by other libraries by organizing them in a golang-idiomatic way
2. Deduplication of code within tools-golang spanning multiple versions
3. Provide the ability for functions to more easily work across versions of SPDX minor versions, preserving the necessary information for compatibility
4. Future additions of a convenience/helper layer of which SBOM construction and mannipulation functions can be built upon and used across document versions

Considerations and assumptions

• The spec data type structs will not change (except for bug fixes)
• New spec minor versions are backward compatible
• Users will update the package whenever they wish to upgrade to a new spec
• Semantic PATCH versions will not affect the library

Design proposal

Part 1: Re-organize structs

Addresses: Goal 1

The way in which version structs are usually defined in goal, taking suite from projects such as kubernetes, is through the use of sub-packages. For example, with a format of type Document, with versions v1 and v2, the way to define them within the go project would be:

v1.Document (from package v1)
v2.Document (from package v2)

In the SPDX case this would be:

v2_1.Document
v2_2.Document

With SPDX 3 on the way, the folders can be organized as:

spdx/spec/v2/v2_1
spdx/spec/v2/v2_2

Part 2: Creating a common working struct

Addresses: Goals 2, 3, 4

The idea here revolves around performing most operations around SPDX structures within a single versioned struct. Similar to what other projects like containerd and other OCI libraries that need to handle operations across multiple versions of images (and specs), having a core canonical structure that expresses all the information required is going to make things easier to work with.

In the case of how SPDX semantic versioning is done, this comes fairly naturally. All new SPDX minor version are additive and backward compatible. Therefore, the latest version of the SPDX struct can be used as the common struct to perform operations on - with the possibility of adding additional metadata when required to "convert" back to the original format. This is a form of generics that works well because of how SPDX versions are defined.

Each SPDX minor version will have a conversion function where it can then be converted to the core struct. For example, there will be:

spdx/spec/v2 (core struct)
spdx/spec/v2/v2_1
spdx/spec/v2/v2_2

Each struct within the minor versions would have a conversion function:

// Converting to core document
func (d *v2_1.Document) ToCore() *v2.Document
func (d *v2_2.Document) ToCore() *v2.Document


// Converting from core document
func (d *v2.Document) ToV2_1() *v2_1.Document
func (d *v2.Document) ToV2_2() *v2_2.Document

The struct may look something like:

package v2

type Document struct {
    // Embedded latest minor version
    v2_3.Document

    // Additional metadata for compatibility
    metadata CoreMetadata
}

type CoreMetadata struct {
    ...
}

Each package can then implement functions just on the Core struct (for example, licensediff):

// New
func MakePairs(p1 v2.Package, p2 v2.Package) (map[string]LicensePair, error)

// Old
func MakePairs2_1(p1 *spdx.Package2_1, p2 *spdx.Package2_1) (map[string]LicensePair, error) {
func MakePairs2_2(p1 *spdx.Package2_2, p2 *spdx.Package2_2) (map[string]LicensePair, error) {

In addition to the code de-duplication, another bonus here is that we can operate on documents of different versions as well, while preserving version information within the metadata.

Alternatives

Struct codegen

This involves creating code that will generate the appropriate structs and helper functions for the SPDX objects. Given that a new release of SPDX spec doesn't happen that often, this probably may be overkill.

Golang generics

At the moment, generics are limited in their implementation and will generics based on struct fields are not implemented. This means for most of our uses for generics for would not be possible today. However, golang release notes do indicate that this may be supported in the future, but with no definite promises/roadmap. It is notable that code that is produced as part of this design proposal will be needed for use in generics as well, and so is a good stepping stone towards it.

[lumjjb]

@swinslow for visibility - we had a chat about this last week.

[swinslow]

I believe Part 1 / Goal 1 from your list should now be complete, with #146 merged?

[lumjjb]

Yup! Going to move on to part 2!!

…

On Wed, Aug 3, 2022, 2:54 PM Steve Winslow ***@***.***> wrote: I believe Part 1 / Goal 1 from your list should now be complete, with #146 <#146> merged? — Reply to this email directly, view it on GitHub <#152 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXLDBUQCA2DE6QFFHZCT53VXK55HANCNFSM54AJMETQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kzantow]

Regarding Part 2: common working struct:

I would suggest this is always the latest SPDX version, and each version has a function to convert from the prior version (much of this could be handled through reflection). Of course this may be lossy when going between major versions, but I suspect in the long term it will result in fewer problems than trying to convert every version to a common working model that changes over time.

The nice thing about this is it would at least allow converting an older SPDX version to a newer one, so it could cover half of the conversion case.

[lumjjb]

Adding reference to issue which is related to this

[lumjjb]

Done from the awesome work from @kzantow !! 🎉