Allow cancelling tasks that generate SBOM with syft.CreateSBOM()

[danielpacak]

What would you like to be added:

It seems that syft.CreateSBOM(context.Context, source.Source, *CreateSBOMConfig) accepts the Go context as the first argument, but when I pass a cancellable context it's ignored and cataloguers' tasks continue running nevertheless.

ctx := context.Background()
scanCtx, cancelScan := context.WithCancelCause(ctx)
defer cancelScan(nil)

go func() {
	// periodically check for memory pressure against soft memory limit ...

	// if memory is above the watermark
	cancelScan(memory.ErrMemoryPressure)
}()

syftJson, err := syft.CreateSBOM(scanCtx, ...)

Why is this needed:

I'm facing some high memory usage issues with Syft running in a container as a Kubernetes Pod. To bound somehow the upper memory limit before I get OOMKill errors I just want to cancel SBOM generation in idiomatic Go way, i.e. by cancelling the context passed to Syft library method.

Additional context:

• It seems that the cancellation request is respected by functions that pull down sources from remote registry, but once cataloguer tasks start I cannot stop Syft from running anymore.
• https://go.dev/doc/database/cancel-operations

[kzantow]

Hi @danielpacak, this sounds like something that we would be happy to get added, generally speaking. The challenge is getting all the functions to respond to the cancel request at appropriate times. Most catalogers search for a set of files and then parse the file(s) one at a time, so a reasonably easy spot to add a hook to stop processing is during a file cataloging loop. However, there are a lot of catalogers and this may require a lot of different updates to really halt processing when asked to. There is a generic cataloger that handles selecting files and calling a parse function in a loop for a large number of catalogers, so updating this one cataloger would probably get over 60% cataloger coverage. But adding cancel behavior to catalogers is only a part of the problem -- there are other processes such as filesystem indexing, online enrichment, or other processes that could result in a lot of memory used, too.

I would also just note that I'm looking to parallelize a number of the catalogers similarly, which could be a good time to think about adding some cancel behavior.

Regardless of adding cancel behavior, I'd love to get more information about what problems you are running in to, it seems like rather than cancelling due to OOM, we should be able to run without getting OOM in the first place.

Do you have any information about particular catalogers or other file processing that is particularly memory intensive?

It could be useful to run in debug mode. In Syft CLI, you can use the environment variable SYFT_DEV_PROFILE=mem to get a PPROF memory profile generated, but it seems like you're using Syft as a library, so you would probably have to do something similar programmatically or by some other means. Would you be able to share a profile during a typical OOM situation so we could understand if there are improvements we could make to help prevent the memory issues in the first place?

[danielpacak]

Thank you @kzantow for a detailed reply and confirmation that the context cancellation is not handled internally by Syft.

Regarding memory consumption something I see immediately with Pyroscope and our daemon, which is running Syft as a library, is quite significant usage of memory by the license checker. It also seems that a new instance of license checker is created each time the syft.CreateSBOM() method is invoked. In other words, the license checker instance is not reused.

I will spend a bit more time analyzing these profiles, but do you have some immediate feedback on the license checker maybe? Just to make sure it's worth further investigation.

[kzantow]

The license scanner is created each time CreateSBOM is called, you're right. It is set in the context, however, so with a little bit of tweaking you could create an instance and provide it via context to create a single instance and reuse it. We would need to expose a method to do this, it looks like everything is internal at the moment.

[danielpacak]

The license scanner is created each time CreateSBOM is called, you're right. It is set in the context, however, so with a little bit of tweaking you could create an instance and provide it via context to create a single instance and reuse it. We would need to expose a method to do this, it looks like everything is internal at the moment.

This was exactly my understanding of the current code and thank you for the confirmation again. I also wanted to instantiate the license scanner once and add it to the context, but then I run into the internal Go package. I need to debug it a bit more, but now it makes sense that we create a new license scanner for each and every image that we scan. I'll draft a minimal pull request that allows reusing the single instance of the license scanner and see how the memory profile changes. It will probably reveal other parts of the Syft library that consume significant amounts of memory.

[danielpacak]

I think it's important observation to share before I proceed with changing code. I actually realized that we were using Syft v1.16.0 compiled with Go 1.23. But when I switched to the latest, i.e. Syft v1.20.0 compiled with Go 1.24 I saw great, consistently reproducible, enhancement in memory management for the very same workload that I've been using for my tests.

These memory usage data was collected from our proprietary Syft daemon running as a Kubernetes pod with the following resource limits:

resources.requests.cpu: 500m
resources.requests.memory: 500Mi
resources.limits.cpu: 1000m
resources.limits.memory: 1500Mi
GOMEMLIMIT: 1200Mi

Notice also the GOMEMLIMIT environment variable set. It makes Go GC more aggressive, but less likely that the pod will get OOMKilled. Also, looking at the changes log at https://tip.golang.org/doc/go1.24#runtime - it seems reasonable that Go 1.24 has some enhancement in allocating memory for small objects as well as new implementation of the builtin map.

Anyway, reusing license scanner instance and making sure we can cancel scan jobs is something I'd like to proceed with.