Concurrent gyrpe runs result in SQLITE_BUSY error

[scothis]

What happened:

When running multiple concurrent grype runs, it works for a bit and then I see a spew of SQLite errors that the database is locked.

[0000]  WARN matcher failed for pkg=Pkg(type=deb, name=mount, version=2.33.1-0.1ubuntu3, upstreams=1): failed to match by source indirection: failed to find vulnerabilities for dpkg upstream source package: matcher failed to fetch distro="ubuntu 19.04" pkg="util-linux": provider failed to fetch namespace='ubuntu:distro:ubuntu:19.04' pkg='util-linux': database is locked (5) (SQLITE_BUSY)

Emphasis on the end of the error:

database is locked (5) (SQLITE_BUSY)

Running grype sequentially works as expected.

What you expected to happen:

I expect to be able to run multiple grype commands concurrently. If there is contention on a shared resource, the process should wait some reasonable period of time to gain access.

I don't know how SQLite locks work under the hood, but I (admittedly naively) expect a RWLock to allow many concurrent reads while restricting writes. I'm assuming the db is not written to during a scan except when updating the db at the start of a scan (which I have disabled).

How to reproduce it (as minimally and precisely as possible):

#!/bin/bash

export GRYPE_DB_AUTO_UPDATE=false

# fetch sboms first to maximize contention within grype

syft -q -ojson registry:ubuntu:artful > artful.json &
syft -q -ojson registry:ubuntu:bionic > bionic.json &
syft -q -ojson registry:ubuntu:cosmic > cosmic.json &
syft -q -ojson registry:ubuntu:disco > disco.json &
syft -q -ojson registry:ubuntu:eoan > eoan.json &
syft -q -ojson registry:ubuntu:focal > focal.json &
syft -q -ojson registry:ubuntu:groovy > groovy.json &

wait

# show output from first grype run, suppress output from other grype runs

grype sbom:artful.json &
grype -q sbom:bionic.json > /dev/null &
grype -q sbom:cosmic.json > /dev/null &
grype -q sbom:disco.json > /dev/null &
grype -q sbom:eoan.json > /dev/null &
grype -q sbom:focal.json > /dev/null &
grype -q sbom:groovy.json > /dev/null &

wait

Anything else we need to know?:

Turning on verbose output makes the error less likely, but does not eliminate the possibility.

Grype 0.42 was the last version that seemed to work without this issue. This is presumably a regression caused by #155 or #825

Environment:

• Output of grype version:

Application:          grype
Version:              0.44.0
Syft Version:         v0.52.0
BuildDate:            2022-07-25T16:28:13Z
GitCommit:            a6ec8f11be5fdc116dc7bbfebdc03bfc13cd2005
GitDescription:       v0.44.0
Platform:             darwin/arm64
GoVersion:            go1.18.4
Compiler:             gc
Supported DB Schema:  4

• OS (e.g: cat /etc/os-release or similar): Mac OS 12.5 (21G72)

[cpendery]

@scothis I think this just got fixed earlier today in #854

[scothis]

#854 works. Thank you