Add new Path filter function to index paths based on the filter #151

gnmahanth · 2023-01-11T09:33:04Z

Fix: anchore/syft#1446

Signed-off-by: gnmahanth <mahanth@deepfence.io>

wagoodman

I like the improvement to the visitor 👍 I have a couple of comments:

My preference on the changes at the top-level API is to lean into the existing functional options (type Option) approach and allow for a filter to be optional. In that sense, passing a default function that always returns true is alright, but a slightly more preferable approach would be to pass nil by default and not call the filter if it doesn't exist down in the visitor.
A test is needed to show the effect of the path filter on the visitor.

Shout out if you want to chat more about this!

* feat: update golang to 1.19 Go 1.18 will become EOL with the upcoming 1.20 release Signed-off-by: Bradley Jones <bradley.jones@anchore.com> * chore: update golangci lint Move to a newer version of golangci lint compatibile with go 1.19 and remove now deprecated linters. Signed-off-by: Bradley Jones <bradley.jones@anchore.com> * chore: update deprecated ioutil functions Signed-off-by: Bradley Jones <bradley.jones@anchore.com> * chore: add crypto/internal/boring to bouncer ignore list Signed-off-by: Bradley Jones <bradley.jones@anchore.com> * fix: nolintlint formatting updated Signed-off-by: Bradley Jones <bradley.jones@anchore.com> --------- Signed-off-by: Bradley Jones <bradley.jones@anchore.com>

* add additional catalog indexes for performance Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * [wip] link resolution Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add leaf link resolution on tree responses (defer ancestor link resolution) Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add filetree search context Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add tests for new search context object Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove unused tar header fields from file.Metadata struct Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * use singular file type definitions Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add logging for filetree searches Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add limited support for glob classes and alternatives Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add failing test to show that index shortcircuits correct behavior Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add link resolution via filetree search context Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * allow index symlink resolution to function through cycles Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add tests for filetree.Index Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add search by parent basename and fix requirements filtering Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * sort search results Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * change file.Type to int + fix layer 0 squashed search context Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * more cleanup Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * switch to generic set implementation Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update linter Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * replace generic set implemetation with plain set (unstable in go1.19) Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * introduce filtree builter and foster usage of reader interfaces Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename content helper functions Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update docs with background Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix get_xid for cross compilation Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * upgrade CI validations workflow Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix snapshot builds Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add tests for file.Index.GetByFileType Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * rename file.Type and file.Resolution Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * ensure that glob results match search facade Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * replace stringset implementation + move resolution tests Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add note about podman dependency for testing Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * address PR comments Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove extra whitespace Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * constrain OS build support Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * update/remove TODO comments Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

)

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>

* test: add failing test for cycle case Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * test: test updates sym links Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * change the filetree recursive pathset to represent open calls Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add another cycle test Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * change filetree attempting path set to counters Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove comment Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Christopher Phillips <christopher.phillips@anchore.com>

* test: add failing test for cycle case Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * test: test updates sym links Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * change the filetree recursive pathset to represent open calls Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add another cycle test Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * change filetree attempting path set to counters Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * remove comment Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * feat: decrement stack depth Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * test: remove old wip test name Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * chore: style updates Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * feat: move maxLinkDepth decrement to inside ancestor loop Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * feat: move maxLinkDepth decrement into resolveNodeLinks loop Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * feat: move detection to top and write minimal test case Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> * test: update linkResolution test to use internal value Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> --------- Signed-off-by: Christopher Phillips <christopher.phillips@anchore.com> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Alex Goodman <alex.goodman@anchore.com>

Signed-off-by: Kushal Beniwal <kbeniwal2305@gmail.com>

…n. (anchore#164) * Add Authenticator as an explicit option. Signed-off-by: Ville Aikas <vaikas@chainguard.dev> * wire in keychain also. Signed-off-by: Ville Aikas <vaikas@chainguard.dev> --------- Signed-off-by: Ville Aikas <vaikas@chainguard.dev>

* fix tmpDirGenerator chain of responsibility associated with anchore#132 Signed-off-by: Joseph Barnett <jb@slim.ai> Signed-off-by: jb@slim.ai <jb@slim.ai> * reduce log message to debug Signed-off-by: Joseph Barnett <jb@slim.ai> Signed-off-by: jb@slim.ai <jb@slim.ai> * restore global cleanup function Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Joseph Barnett <jb@slim.ai> Signed-off-by: jb@slim.ai <jb@slim.ai> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Alex Goodman <alex.goodman@anchore.com>

…re#152) * set the default platform for select sources based on host arch Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * decompose into smaller function and add tests for setting default platform Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 20.10.12+incompatible to 20.10.24+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v20.10.12...v20.10.24) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bind file times to metadata Signed-off-by: Adrian Wobito <adrian.wobito@gmail.com> * expose link strategy to walkConditions Signed-off-by: Adrian Wobito <adrian.wobito@gmail.com> * update: test times Signed-off-by: Adrian Wobito <adrian.wobito@gmail.com> * update: link options api Signed-off-by: Adrian Wobito <adrian.wobito@gmail.com> * stub mod time for dynamic tar test fixtures Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * set mtime on tar fixture explicitly Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * always interpret tar header timestamps as UTC Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * interpret all file metadata timestamps as UTC Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Adrian Wobito <adrian.wobito@gmail.com> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Alex Goodman <alex.goodman@anchore.com>

Signed-off-by: Amar Babu <amarbabu@outlook.in>

* fix linting Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add format make target Signed-off-by: Alex Goodman <alex.goodman@anchore.com> * add developing reference to format target Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Alex Goodman <alex.goodman@anchore.com>

* refactor: embed fs.FileInfo within file.Metadata Embed the original `fs.FileInfo` directly within the `file.Metadata struct`, replacing the previous `Size`, `IsDir`, `Mode`, and `ModTime` fields. Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com> * add ManualInfo helper struct Signed-off-by: Alex Goodman <alex.goodman@anchore.com> --------- Signed-off-by: Adam Hughes <9903835+tri-adam@users.noreply.github.com> Signed-off-by: Alex Goodman <alex.goodman@anchore.com> Co-authored-by: Alex Goodman <alex.goodman@anchore.com>

Otherwise, "make integration" will fail on M1 Macs. Note that this is a workaround which assumes that, if "make integration" is run on an arm64 host, that host is able to run containers from amd64 images. This change enables development on stereoscope on M1 Macs, but may not enable it on other arm64 machines. Signed-off-by: Will Murphy <will.murphy@anchore.com>

Bumps [github.com/docker/distribution](https://github.com/docker/distribution) from 2.8.1+incompatible to 2.8.2+incompatible. - [Release notes](https://github.com/docker/distribution/releases) - [Commits](distribution/distribution@v2.8.1...v2.8.2) --- updated-dependencies: - dependency-name: github.com/docker/distribution dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This reverts commit d7551b7. Defaulting to platform for all providers resulted in syft having unnecessary when pulling an image that had a particular digest, if that digest didn't match the architecture of the host running the pull. Revert commit d7551b7, which introduced that error, but then add platform defaulting logic back for the OCI Registry Provider, since defaulting there has been specifically requested. Add integ tests to cover the new behavior. Also, update integ tests to use the manifest ID for assertsion, since the RepoDigests array can be empty. Signed-off-by: Will Murphy <will.murphy@anchore.com>

wagoodman · 2023-07-24T13:03:45Z

@gnmahanth would you like help incorporating the above comments?

gnmahanth added 3 commits January 11, 2023 09:06

skip paths not in catalogers glob patterns

ca742f2

Signed-off-by: gnmahanth <mahanth@deepfence.io>

fix missing java and binary files from sbom

30a1c6d

Signed-off-by: gnmahanth <mahanth@deepfence.io>

modify code to take path filter func as parmater instead of all paths

ee0c971

Signed-off-by: gnmahanth <mahanth@deepfence.io>

gnmahanth mentioned this pull request Jan 11, 2023

Optimise resolver anchore/syft#1447

Closed

fix lint issue

08e5b9c

Signed-off-by: gnmahanth <mahanth@deepfence.io>

wagoodman reviewed Jan 13, 2023

View reviewed changes

bradleyjones and others added 21 commits June 15, 2023 17:06

Bump github.com/containerd/containerd from 1.6.12 to 1.6.18 (anchore#156

38cc5e5

)

fix: bump golang.org/x/net to v0.7.0 (anchore#157)

bd07bb5

Signed-off-by: Weston Steimel <weston.steimel@anchore.com>

fix: thread safety for progress (anchore#162)

dbb5dc0

chore: fix typo (anchore#163)

9425f2a

Signed-off-by: Kushal Beniwal <kbeniwal2305@gmail.com>

Bump github.com/containerd/containerd from 1.6.18 to 1.7.0 (anchore#168)

162a14e

Signed-off-by: Amar Babu <amarbabu@outlook.in>

Merge branch 'anchore:main' into main

dde2147

Merge branch 'main' into path-filter

b9ab08f

wagoodman self-assigned this Jul 25, 2023

gnmahanth force-pushed the path-filter branch from abf8444 to b9ab08f Compare October 26, 2023 12:17

wagoodman removed their assignment Feb 26, 2024

wagoodman self-assigned this May 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new Path filter function to index paths based on the filter #151

Add new Path filter function to index paths based on the filter #151

gnmahanth commented Jan 11, 2023

wagoodman left a comment

wagoodman commented Jul 24, 2023

Add new Path filter function to index paths based on the filter #151

Are you sure you want to change the base?

Add new Path filter function to index paths based on the filter #151

Conversation

gnmahanth commented Jan 11, 2023

wagoodman left a comment

Choose a reason for hiding this comment

wagoodman commented Jul 24, 2023