Remove MetadataType from core package object and normalize JSON metadataType values

[wagoodman]

This PR:

• removes the pkg.Package.MetadataType from the core package model, keeping it as a concern for the syftjson format.
• adds a convention around how to name JSON metadataType values, documented in the DEVELOPING.md.
• adds a 1:many lookup for json metadataType names to support legacy conversions of JSON payloads into a package metadata struct
• adds an adapter layer between the package metadata structs (an implementation detail) and the data shapes that are officially supported in the syft json output. This allows us to rename package metadata structs without needing to break the JSON schema (since today that would change a type definition, and if we were being consistent, would change the name of the MetadataType name). This adapter layer breaks the need for MetadataTypes to be strongly associated with the struct names.
• adds a new SYFT_FORMAT_JSON_LEGACY=<bool> (defaulting to false) to the syft application config. This allows folks to be able to fallback to the old JSON metadata type names (and other soon-to-be-breaking changes) to get to a pre-1.0 state of the JSON output.
• Migrates the SYFT_TEMPLATE configuration to SYFT_FORMAT_TEMPLATE to be consistent with future format related configurations.
• Renames the existing pkg.*Metadata structs to be consistent with the metadata type names (they do not always match exactly).

Doing this necessarily breaks the JSON schema, so it has been rev'd to v12 in this PR.

The downstream grype PR has been drafted: anchore/grype#1423

For a semantic diff of the v11.0.1 vs v12 JSON schema see #1983 (comment) .

Fixes #1844
Fixes #1735

[github-actions]

Benchmark Test Results

Benchmark results from the latest changes vs base branch

goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz%0A                                                              │ ./.tmp/benchmark-db2153c.txt │%0A                                                              │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       12.08m ±  1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        650.3µ ±  4%25%0AImagePackageCatalogers/binary-cataloger-2                                       204.6µ ±  0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       545.4µ ±  2%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   20.82µ ±  0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             92.67µ ± 14%25%0AImagePackageCatalogers/java-cataloger-2                                         13.26m ±  2%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         90.32µ ±  1%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           342.0µ ±  1%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    247.2µ ±  2%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       718.0µ ±  2%25%0AImagePackageCatalogers/portage-cataloger-2                                      411.0µ ±  2%25%0AImagePackageCatalogers/python-package-cataloger-2                               3.167m ±  1%25%0AImagePackageCatalogers/r-package-cataloger-2                                    172.9µ ±  1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       469.3µ ±  2%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 827.3µ ±  2%25%0AImagePackageCatalogers/sbom-cataloger-2                                         115.8µ ±  0%25%0Ageomean                                                                         449.4µ%0A%0A                                                              │ ./.tmp/benchmark-db2153c.txt │%0A                                                              │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       5.142Mi ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        202.0Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                       30.57Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       167.3Ki ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   3.697Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             9.906Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                         2.817Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         8.594Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           92.81Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    47.63Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       181.4Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                      118.7Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                              1021.8Ki ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                    51.75Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       179.7Ki ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 140.9Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                         14.21Ki ± 0%25%0Ageomean                                                                         99.34Ki%0A%0A                                                              │ ./.tmp/benchmark-db2153c.txt │%0A                                                              │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                        88.12k ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                         4.136k ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                         848.0 ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                        3.064k ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                     132.0 ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                               281.0 ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                          40.51k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                           228.0 ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                            1.315k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                      860.0 ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                        3.998k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                       2.245k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                                16.34k ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                      902.0 ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                        3.954k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                  2.393k ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                           394.0 ± 0%25%0Ageomean                                                                          2.038k

[wagoodman]

Semantic diff for reviewers between the v11 and v12 json schemas BEFORE the struct renames

Code

The python code that generated this list

import json
import difflib

original_schema = "schema/json/schema-11.0.1.json"
new_schema = "schema/json/schema-12.0.0.json"

# {old-type-name: new-type-name}
type_def_mapping = {
 "AlpmMetadata": "arch-alpm-db-record",
 "ApkMetadata": "alpine-apk-db-record",
 "BinaryMetadata": "binary-signature",
 "CocoapodsMetadata": "cocoa-podfile-lock",
 "ConanLockMetadata": "c-conan-lock",
 "ConanMetadata": "c-conan",
 "DartPubMetadata": "dart-pubspec-lock",
 "DotnetPortableExecutableMetadata": "dotnet-portable-executable",
 "DotnetDepsMetadata": "dotnet-deps",
 "DpkgMetadata": "debian-dpkg-db-record",
 "GemMetadata": "ruby-gemspec",
 "GolangBinMetadata": "go-module-binary-buildinfo",
 "GolangModMetadata": "go-module",
 "HackageMetadata": "haskell-hackage-stack",
 "JavaMetadata": "java-archive",
 "KbPackageMetadata": "microsoft-kb-patch",
 "LinuxKernelMetadata": "linux-kernel-archive",
 "LinuxKernelModuleMetadata": "linux-kernel-module",
 "MixLockMetadata": "elixir-mix-lock",
 "NixStoreMetadata": "nix-store",
 "NpmPackageJSONMetadata": "javascript-npm-package",
 "NpmPackageLockJSONMetadata": "javascript-npm-package-lock",
 "PhpComposerJSONMetadata": "php-composer-lock",
 "PortageMetadata": "gentoo-portage-db-record",
 "PythonPackageMetadata": "python-package",
 "PythonPipfileLockMetadata": "python-pipfile-lock",
 "PythonRequirementsMetadata": "python-pip-requirements",
 "RebarLockMetadata": "erlang-rebar-lock",
 "RDescriptionFileMetadata": "r-description",
 "RpmdbFileRecord": "rpm-file-record",
 "RpmMetadata": "redhat-rpm-db-record",
 "RpmdbMetadata": "redhat-rpm-db-record",
 "RpmDBMetadata": "redhat-rpm-db-record",
 "RpmArchiveMetadata": "redhat-rpm-archive",
 "SwiftPackageManagerMetadata": "swift-package-manager-lock",
 "CargoPackageMetadata": "rust-cargo-lock"
}


def main():
 original_type_definitions = extract_type_definitions(original_schema)
 new_type_definitions = extract_type_definitions(new_schema)

 new_names_diffed = set()
 names_with_same_content = set()

 for definition_name, old_def in original_type_definitions.items():
     new_name = get_new_name(definition_name)
     if not new_name:
         new_name = definition_name
     new_def = new_type_definitions.get(new_name, "")
     if not new_def:
         print("Missing definition in new schema: {}".format(definition_name))
         continue

     new_names_diffed.add(new_name)

     # diff the definitions
     diff = difflib.unified_diff(old_def.splitlines(), new_def.splitlines(), fromfile=original_schema, tofile=new_schema)
     diff = "\n".join(diff)
     if diff:
         print("Diff for {}".format(definition_name))
         print(diff)
         print()
     else:
         names_with_same_content.add(definition_name)

 # for all new names not processed, print a warning

 for definition_name, new_def in new_type_definitions.items():
     if definition_name not in new_names_diffed:
         print("Missing equivalent definition in original schema: {}".format(definition_name))

 print(f"Definitions with same content: {len(names_with_same_content)}")
 for name in sorted(list(names_with_same_content)):
     print("  -", name)


def extract_type_definitions(schema_file_path) -> dict[str, str]:
 with open(schema_file_path, "r") as schema_file:
     schema = json.load(schema_file)

 definitions = schema.get("$defs", {})
 type_definitions = {}

 for definition_name, definition in definitions.items():
     # if definition_name in type_def_mapping:
     #     new_name = to_camel_case(type_def_mapping[definition_name])
     #     # print("Renaming {} to {}".format(definition_name, new_name))
     #     definition_name = new_name
     # # else:
     # #     print("No mapping for {}".format(definition_name))
     type_definitions[definition_name] = json.dumps(definition, indent=2, sort_keys=True)

 return type_definitions


def get_new_name(name: str) -> str | None:
 if name in type_def_mapping:
     return to_camel_case(type_def_mapping[name])


def to_camel_case(s: str) -> str:
 s = s.replace("-", "_")
 return ''.join(x.capitalize() or '_' for x in s.split('_'))


if __name__ == "__main__":
 main()

Diff for HackageMetadata
--- schema/json/schema-11.0.1.json

+++ schema/json/schema-12.0.0.json

@@ -1,21 +1,8 @@

 {
   "properties": {
-    "name": {
-      "type": "string"
-    },
     "pkgHash": {
-      "type": "string"
-    },
-    "snapshotURL": {
-      "type": "string"
-    },
-    "version": {
       "type": "string"
     }
   },
-  "required": [
-    "name",
-    "version"
-  ],
   "type": "object"
 }

Diff for Package
--- schema/json/schema-11.0.1.json

+++ schema/json/schema-12.0.0.json

@@ -30,100 +30,109 @@

           "type": "null"
         },
         {
-          "$ref": "#/$defs/AlpmMetadata"
+          "$ref": "#/$defs/AlpineApkDbRecord"
         },
         {
-          "$ref": "#/$defs/ApkMetadata"
+          "$ref": "#/$defs/ArchAlpmDbRecord"
         },
         {
-          "$ref": "#/$defs/BinaryMetadata"
+          "$ref": "#/$defs/BinarySignature"
         },
         {
-          "$ref": "#/$defs/CargoPackageMetadata"
+          "$ref": "#/$defs/CConan"
         },
         {
-          "$ref": "#/$defs/CocoapodsMetadata"
+          "$ref": "#/$defs/CConanLock"
         },
         {
-          "$ref": "#/$defs/ConanLockMetadata"
+          "$ref": "#/$defs/CocoaPodfileLock"
         },
         {
-          "$ref": "#/$defs/ConanMetadata"
+          "$ref": "#/$defs/DartPubspecLock"
         },
         {
-          "$ref": "#/$defs/DartPubMetadata"
+          "$ref": "#/$defs/DebianDpkgDbRecord"
         },
         {
-          "$ref": "#/$defs/DotnetDepsMetadata"
+          "$ref": "#/$defs/DotnetDeps"
         },
         {
-          "$ref": "#/$defs/DotnetPortableExecutableMetadata"
+          "$ref": "#/$defs/DotnetPortableExecutable"
         },
         {
-          "$ref": "#/$defs/DpkgMetadata"
+          "$ref": "#/$defs/ElixirMixLock"
         },
         {
-          "$ref": "#/$defs/GemMetadata"
+          "$ref": "#/$defs/ErlangRebarLock"
         },
         {
-          "$ref": "#/$defs/GolangBinMetadata"
+          "$ref": "#/$defs/GentooPortageDbRecord"
         },
         {
-          "$ref": "#/$defs/GolangModMetadata"
+          "$ref": "#/$defs/GoModule"
         },
         {
-          "$ref": "#/$defs/HackageMetadata"
+          "$ref": "#/$defs/GoModuleBinaryBuildinfo"
         },
         {
-          "$ref": "#/$defs/JavaMetadata"
+          "$ref": "#/$defs/HaskellHackageStack"
         },
         {
-          "$ref": "#/$defs/KbPackageMetadata"
+          "$ref": "#/$defs/HaskellHackageStackLock"
         },
         {
-          "$ref": "#/$defs/LinuxKernelMetadata"
+          "$ref": "#/$defs/JavaArchive"
         },
         {
-          "$ref": "#/$defs/LinuxKernelModuleMetadata"
+          "$ref": "#/$defs/JavascriptNpmPackage"
         },
         {
-          "$ref": "#/$defs/MixLockMetadata"
+          "$ref": "#/$defs/JavascriptNpmPackageLock"
         },
         {
-          "$ref": "#/$defs/NixStoreMetadata"
+          "$ref": "#/$defs/LinuxKernelArchive"
         },
         {
-          "$ref": "#/$defs/NpmPackageJSONMetadata"
+          "$ref": "#/$defs/LinuxKernelModule"
         },
         {
-          "$ref": "#/$defs/NpmPackageLockJSONMetadata"
+          "$ref": "#/$defs/MicrosoftKbPatch"
         },
         {
-          "$ref": "#/$defs/PhpComposerJSONMetadata"
+          "$ref": "#/$defs/NixStore"
         },
         {
-          "$ref": "#/$defs/PortageMetadata"
+          "$ref": "#/$defs/PhpComposerInstalled"
         },
         {
-          "$ref": "#/$defs/PythonPackageMetadata"
+          "$ref": "#/$defs/PhpComposerLock"
         },
         {
-          "$ref": "#/$defs/PythonPipfileLockMetadata"
+          "$ref": "#/$defs/PythonPackage"
         },
         {
-          "$ref": "#/$defs/PythonRequirementsMetadata"
+          "$ref": "#/$defs/PythonPipRequirements"
         },
         {
-          "$ref": "#/$defs/RDescriptionFileMetadata"
+          "$ref": "#/$defs/PythonPipfileLock"
         },
         {
-          "$ref": "#/$defs/RebarLockMetadata"
+          "$ref": "#/$defs/RDescription"
         },
         {
-          "$ref": "#/$defs/RpmMetadata"
+          "$ref": "#/$defs/RedhatRpmArchive"
         },
         {
-          "$ref": "#/$defs/SwiftPackageManagerMetadata"
+          "$ref": "#/$defs/RedhatRpmDbRecord"
+        },
+        {
+          "$ref": "#/$defs/RubyGemspec"
+        },
+        {
+          "$ref": "#/$defs/RustCargoLock"
+        },
+        {
+          "$ref": "#/$defs/SwiftPackageManagerLock"
         }
       ]
     },

Diff for RpmMetadata
--- schema/json/schema-11.0.1.json

+++ schema/json/schema-12.0.0.json

@@ -15,7 +15,7 @@

     },
     "files": {
       "items": {
-        "$ref": "#/$defs/RpmdbFileRecord"
+        "$ref": "#/$defs/RpmFileRecord"
       },
       "type": "array"
     },

Missing equivalent definition in original schema: HaskellHackageStackLock
Missing equivalent definition in original schema: PhpComposerInstalled
Missing equivalent definition in original schema: RedhatRpmArchive
Definitions with same content: 64
  - AlpmFileRecord
  - AlpmMetadata
  - ApkFileRecord
  - ApkMetadata
  - BinaryMetadata
  - CargoPackageMetadata
  - ClassifierMatch
  - CocoapodsMetadata
  - ConanLockMetadata
  - ConanMetadata
  - Coordinates
  - DartPubMetadata
  - Descriptor
  - Digest
  - Document
  - DotnetDepsMetadata
  - DotnetPortableExecutableMetadata
  - DpkgFileRecord
  - DpkgMetadata
  - File
  - FileLicense
  - FileLicenseEvidence
  - FileMetadataEntry
  - GemMetadata
  - GolangBinMetadata
  - GolangModMetadata
  - IDLikes
  - JavaManifest
  - JavaMetadata
  - KbPackageMetadata
  - License
  - LinuxKernelMetadata
  - LinuxKernelModuleMetadata
  - LinuxKernelModuleParameter
  - LinuxRelease
  - Location
  - MixLockMetadata
  - NixStoreMetadata
  - NpmPackageJSONMetadata
  - NpmPackageLockJSONMetadata
  - PhpComposerAuthors
  - PhpComposerExternalReference
  - PhpComposerJSONMetadata
  - PomParent
  - PomProject
  - PomProperties
  - PortageFileRecord
  - PortageMetadata
  - PythonDirectURLOriginInfo
  - PythonFileDigest
  - PythonFileRecord
  - PythonPackageMetadata
  - PythonPipfileLockMetadata
  - PythonRequirementsMetadata
  - RDescriptionFileMetadata
  - RebarLockMetadata
  - Relationship
  - RpmdbFileRecord
  - Schema
  - SearchResult
  - Secrets
  - Source
  - SwiftPackageManagerMetadata
  - licenses

[kzantow]

I don't see any blocking issues, but left a suggestion about defining the type-to-name mappings for JSON.

Removing approval so this doesn't accidentally get merged until we're ready for it

[spiffcs]

@wagoodman I read through everything here and no notes or comments the change makes sense IMO - feel free to merge when you think syft is ready for the major schema bump

there have been enough changes to warrant a review on the new change set

[kzantow]

🎉