Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guess unpinned versions in python requirements.txt #1966

Merged
merged 9 commits into from
Jul 27, 2023

Conversation

wagoodman
Copy link
Contributor

@wagoodman wagoodman commented Jul 27, 2023

Expands on #1597 by exposing a new python.guess-unpinned-requirements configuration, allowing loose requirements to be filled in with the lowest expressible version instead of dropping the package altogether. Note: since this is synthesizing version information this is an opt-in feature.

This replaces the v10 schema changes in #1967 since they have not been released yet (a condition specifically allowed in the schema readme). The small change is to allow for some elements to be optional in the requirements metadata (everything except for name and version).

manifestori and others added 5 commits February 21, 2023 14:06
Signed-off-by: manifestori <ori@manifestcyber.com>
Signed-off-by: manifestori <ori@manifestcyber.com>
…txt-parsing

feat: make python requirements.txt parser more inclusive
…ntstxt-parsing

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
…tion

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
@wagoodman wagoodman added the breaking-change Change is not backwards compatible label Jul 27, 2023
@github-actions
Copy link

github-actions bot commented Jul 27, 2023

Benchmark Test Results

Benchmark results from the latest changes vs base branch
goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       12.61m ±  1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        751.0µ ±  2%25%0AImagePackageCatalogers/binary-cataloger-2                                       210.2µ ± 11%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       623.3µ ±  1%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   23.36µ ±  1%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             98.89µ ±  1%25%0AImagePackageCatalogers/java-cataloger-2                                         13.99m ±  1%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         98.50µ ±  3%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           405.8µ ±  1%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    290.8µ ±  2%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       834.4µ ±  2%25%0AImagePackageCatalogers/portage-cataloger-2                                      504.6µ ±  1%25%0AImagePackageCatalogers/python-package-cataloger-2                               3.558m ±  1%25%0AImagePackageCatalogers/r-package-cataloger-2                                    219.8µ ±  1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       580.1µ ±  3%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 968.4µ ±  1%25%0AImagePackageCatalogers/sbom-cataloger-2                                         122.8µ ±  1%25%0Ageomean                                                                         509.7µ%0A%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                       5.123Mi ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                        204.9Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                       30.18Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                       169.0Ki ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                   3.695Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                             9.906Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                         2.824Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                         8.594Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                           94.22Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                    49.14Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                       186.7Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                      119.9Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                               1.003Mi ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                    53.30Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                       180.9Ki ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                 144.1Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                         14.20Ki ± 0%25%0Ageomean                                                                         100.3Ki%0A%0A                                                              │ ./.tmp/benchmark-e497ead.txt │%0A                                                              │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                        87.75k ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                         4.182k ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                         830.0 ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                        3.002k ± 0%25%0AImagePackageCatalogers/dotnet-portable-executable-cataloger-2                     132.0 ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                               281.0 ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                          39.88k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                           228.0 ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                            1.342k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                      895.0 ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                        4.080k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                       2.268k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                                16.44k ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                      929.0 ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                        3.989k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                                  2.447k ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                           394.0 ± 0%25%0Ageomean                                                                          2.051k

@wagoodman wagoodman removed the breaking-change Change is not backwards compatible label Jul 27, 2023
@wagoodman wagoodman force-pushed the feat-python-requirementstxt-parsing branch from d1231b4 to ad3926b Compare July 27, 2023 14:03
@wagoodman
Copy link
Contributor Author

Note: depends on #1967 before reviewing / merging

…ntstxt-parsing

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman
Copy link
Contributor Author

# $ diff -C 2 ./schema/json/schema-9.0.2.json ./schema/json/schema-10.0.0.json
*** ./schema/json/schema-9.0.2.json0.2.jThu Jul 27 08:55:01 2023-10.0.0.json
--- ./schema/json/schema-10.0.0.json    Thu Jul 27 11:30:14 2023
***************
*** 1,5 ****
  {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
!   "$id": "anchore.io/schema/syft/json/9.0.2/document",
    "$ref": "#/$defs/Document",
    "$defs": {
--- 1,5 ----
  {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
!   "$id": "anchore.io/schema/syft/json/10.0.0/document",
    "$ref": "#/$defs/Document",
    "$defs": {
***************
*** 1621,1630 ****
          },
          "markers": {
!           "patternProperties": {
!             ".*": {
!               "type": "string"
!             }
!           },
!           "type": "object"
          }
        },
--- 1621,1625 ----
          },
          "markers": {
!           "type": "string"
          }
        },
***************
*** 1632,1639 ****
        "required": [
          "name",
!         "extras",
!         "versionConstraint",
!         "url",
!         "markers"
        ]
      },
--- 1627,1631 ----
        "required": [
          "name",
!         "versionConstraint"
        ]
      },

@wagoodman wagoodman added breaking-change Change is not backwards compatible and removed breaking-change Change is not backwards compatible labels Jul 27, 2023
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
@wagoodman wagoodman marked this pull request as ready for review July 27, 2023 15:38
@wagoodman wagoodman added the changelog-ignore Don't include this issue in the release changelog label Jul 27, 2023
@wagoodman wagoodman requested a review from a team July 27, 2023 15:40
@wagoodman wagoodman self-assigned this Jul 27, 2023
@wagoodman wagoodman removed the changelog-ignore Don't include this issue in the release changelog label Jul 27, 2023
@wagoodman wagoodman merged commit 063e9da into main Jul 27, 2023
9 checks passed
@wagoodman wagoodman deleted the feat-python-requirementstxt-parsing branch July 27, 2023 18:27
@kzantow kzantow added the enhancement New feature or request label Jul 31, 2023
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
* feat: python requirements.txt parsing inclusive

Signed-off-by: manifestori <ori@manifestcyber.com>

* refactor: parseVersion

Signed-off-by: manifestori <ori@manifestcyber.com>

* add python config for optional requirements version constraint resolution

Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* fix tests

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* allow for python requirements metadata to be optional

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

* restore cyclonedx dependency

Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>

---------

Signed-off-by: manifestori <ori@manifestcyber.com>
Signed-off-by: Alex Goodman <wagoodman@users.noreply.github.com>
Signed-off-by: Alex Goodman <alex.goodman@anchore.com>
Co-authored-by: manifestori <ori@manifestcyber.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

4 participants