Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot run multiple instances in parallel #650

Closed
ca-scribner opened this issue Sep 24, 2020 · 3 comments · Fixed by #660
Closed

Cannot run multiple instances in parallel #650

ca-scribner opened this issue Sep 24, 2020 · 3 comments · Fixed by #660
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation.

Comments

@ca-scribner
Copy link

Description

I would like to run trivy on multiple images in parallel (eg: two scripts are running separately and both will call trivy on different images)

What did you expect to happen?

Trivy processes would be isolated and proceed as if only one was running

What happened instead?

The second trivy process to start appears to wait until the first trivy process has completed before starting

Output of run with -debug:

Scanning python:3.7.1 and python:3.7.2 (with images already docker pull'd)

In series:

trivy image -c ; time trivy -d image python:3.7.1:

2020-09-24T11:58:04.118-0400	INFO	Removing image caches...
2020-09-24T11:58:04.132-0400	DEBUG	Severities: CRITICAL
2020-09-24T11:58:04.163-0400	DEBUG	cache dir:  /home/scribs/.cache/trivy
2020-09-24T11:58:04.164-0400	DEBUG	DB update was skipped because DB is the latest
2020-09-24T11:58:04.164-0400	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2020-09-24 12:16:58.904183153 +0000 UTC, NextUpdate: 2020-09-25 00:16:58.904182253 +0000 UTC
2020-09-24T11:58:04.168-0400	DEBUG	Vulnerability type:  [os library]
2020-09-24T11:58:18.328-0400	DEBUG	Artifact ID: sha256:1e80caffd59ed9edbd05c6ad47427caadf297f399ec069ec014a5aadb2f261a0
2020-09-24T11:58:18.328-0400	DEBUG	Blob IDs: [sha256:90d1009ce6fe3102fee728742a3bd73eea2b39c88cdda99977a3fb130dbc17ac sha256:c23711a84ad43e5062e70a4055d7df344087649bec7eac06a40a8b1af4cef86e sha256:8f7ee6d76fd9da1ecfcfcd893357e78d7dcd4ca877295b7565c71728411f9126 sha256:f75e64f96dbc610a00722afa3826745952db8d1416f6405e7e87c20868b9f542 sha256:e02b32b1ff99a237da5c0cc4bb46dd672447b66c1b524975ae1f59c11abfd79c sha256:9e89ea4aeda3f53ff7052678d85fd9cb486808aa83b594fd3ca7211e6119eda9 sha256:3294fae586267362646c1a2d57f2dbdb741cc7fd81f432c1af0662cdcf3dae2f sha256:9e701d0e0edd1f076bd90d4b1805c2268bacb7a5fcc2ec840ccc6c4a7eb2b699 sha256:c660d5dc256a20f14655d9a7610b8122fa91b451ab67dfa90d98bc46b42e9173]
2020-09-24T11:58:18.337-0400	INFO	Detecting Debian vulnerabilities...
2020-09-24T11:58:18.337-0400	DEBUG	debian: os version: 9
2020-09-24T11:58:18.337-0400	DEBUG	debian: the number of packages: 425

python:3.7.1 (debian 9.6)
=========================
Total: 0 (CRITICAL: 0)

0.90user 1.21system 0:14.60elapsed 14%CPU (0avgtext+0avgdata 43320maxresident)k
0inputs+1854928outputs (0major+6870minor)pagefaults 0swaps

(~14s runtime)

trivy image -c ; time trivy -d image python:3.7.2:

2020-09-24T11:56:58.404-0400	INFO	Removing image caches...
2020-09-24T11:56:58.424-0400	DEBUG	Severities: CRITICAL
2020-09-24T11:56:58.456-0400	DEBUG	cache dir:  /home/scribs/.cache/trivy
2020-09-24T11:56:58.456-0400	DEBUG	DB update was skipped because DB is the latest
2020-09-24T11:56:58.457-0400	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2020-09-24 12:16:58.904183153 +0000 UTC, NextUpdate: 2020-09-25 00:16:58.904182253 +0000 UTC
2020-09-24T11:56:58.460-0400	DEBUG	Vulnerability type:  [os library]
2020-09-24T11:57:11.310-0400	DEBUG	Artifact ID: sha256:2053ca75899e830873fcb55b8eba20f9c9f6bc35b3985c99d66f1f393163f7c0
2020-09-24T11:57:11.310-0400	DEBUG	Blob IDs: [sha256:fbb641a8b94349e89886f65d79928e4673530e2a2b4d33c2c95e0426713f78e4 sha256:604829a174eb966a2102e2e68c7669e1fe56721e8d7ea27f9a286aa33be8be20 sha256:12cb127eee44270330891b1b610ce34e81f53a91a22e3a7f53f0632391d99892 sha256:b17cc31e431beb2f39988dff23d04f85ba4b446fc0a13f304774852fa3c87d85 sha256:0fe19df8b8f8eb5f545f50e11f958bf37d27ab7da5c260a9bd2bc0ff6fb760a4 sha256:9f72e27f9aa489ec42f79fd3adeedc7c9dab6a1de6ccec6227ffc7fac50f466a sha256:b4074dfd8a235adc7a5d7a7b7d4a055d59d25eb86e3d83d06bfe999bc196d784 sha256:c76a8d0f7f2444b1cd3c7c240c47cae5c4f77ff885df056642b6345bab9f502a sha256:b24438cea2cc5b597ca782324d04b958a59cde6473daf31f20099305e2264b05]
2020-09-24T11:57:11.321-0400	INFO	Detecting Debian vulnerabilities...
2020-09-24T11:57:11.322-0400	DEBUG	debian: os version: 9
2020-09-24T11:57:11.322-0400	DEBUG	debian: the number of packages: 428

python:3.7.2 (debian 9.8)
=========================
Total: 0 (CRITICAL: 0)

0.93user 1.12system 0:13.27elapsed 15%CPU (0avgtext+0avgdata 39976maxresident)k
0inputs+1867632outputs (0major+6257minor)pagefaults 0swaps

(~14s runtime)

If run at the same time, first one still completes in ~14s but second shows:

2020-09-24T11:59:08.397-0400	DEBUG	Severities: CRITICAL
(hangs here for the ~14s it takes the first command to run)
2020-09-24T11:59:20.012-0400	DEBUG	cache dir:  /home/scribs/.cache/trivy
2020-09-24T11:59:20.012-0400	DEBUG	DB update was skipped because DB is the latest
2020-09-24T11:59:20.012-0400	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2020-09-24 12:16:58.904183153 +0000 UTC, NextUpdate: 2020-09-25 00:16:58.904182253 +0000 UTC
2020-09-24T11:59:20.014-0400	DEBUG	Vulnerability type:  [os library]
2020-09-24T11:59:33.591-0400	DEBUG	Artifact ID: sha256:1e80caffd59ed9edbd05c6ad47427caadf297f399ec069ec014a5aadb2f261a0
2020-09-24T11:59:33.591-0400	DEBUG	Blob IDs: [sha256:90d1009ce6fe3102fee728742a3bd73eea2b39c88cdda99977a3fb130dbc17ac sha256:c23711a84ad43e5062e70a4055d7df344087649bec7eac06a40a8b1af4cef86e sha256:8f7ee6d76fd9da1ecfcfcd893357e78d7dcd4ca877295b7565c71728411f9126 sha256:f75e64f96dbc610a00722afa3826745952db8d1416f6405e7e87c20868b9f542 sha256:e02b32b1ff99a237da5c0cc4bb46dd672447b66c1b524975ae1f59c11abfd79c sha256:9e89ea4aeda3f53ff7052678d85fd9cb486808aa83b594fd3ca7211e6119eda9 sha256:3294fae586267362646c1a2d57f2dbdb741cc7fd81f432c1af0662cdcf3dae2f sha256:9e701d0e0edd1f076bd90d4b1805c2268bacb7a5fcc2ec840ccc6c4a7eb2b699 sha256:c660d5dc256a20f14655d9a7610b8122fa91b451ab67dfa90d98bc46b42e9173]
2020-09-24T11:59:33.600-0400	INFO	Detecting Debian vulnerabilities...
2020-09-24T11:59:33.600-0400	DEBUG	debian: os version: 9
2020-09-24T11:59:33.600-0400	DEBUG	debian: the number of packages: 425

python:3.7.1 (debian 9.6)
=========================
Total: 0 (CRITICAL: 0)

0.92user 1.18system 0:25.59elapsed 8%CPU (0avgtext+0avgdata 43296maxresident)k
328inputs+1855200outputs (2major+6900minor)pagefaults 0swaps

(completed in ~26s)

Output of trivy -v:

Version: 0.11.0
Vulnerability DB:
  Type: Light
  Version: 1
  UpdatedAt: 2020-09-24 12:16:58.904183153 +0000 UTC
  NextUpdate: 2020-09-25 00:16:58.904182253 +0000 UTC

Additional details (base image name, container registry info...):

running with trivy --cache-dir someUniquDirPerProcess ... allows the processes to run in isolation as expected. Seems like one locks the cache or db and the other waits? If this is the expected behavior, it could be documented more prominently as running multiple processes at once does not seem unlikely.

Thanks!
@ca-scribner ca-scribner added the kind/bug Categorizes issue or PR as related to a bug. label Sep 24, 2020
@knqyf263 knqyf263 added kind/documentation Categorizes issue or PR as related to documentation. and removed kind/bug Categorizes issue or PR as related to a bug. labels Oct 1, 2020
@knqyf263
Copy link
Collaborator

knqyf263 commented Oct 1, 2020

@ca-scribner Thank you for pointing it out. This limitation is from boltdb.

Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it. To prevent an indefinite wait you can pass a timeout option to the Open() function:

https://github.com/boltdb/bolt#opening-a-database

As you said, it should be mentioned in README.

@knqyf263 knqyf263 added the good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. label Oct 1, 2020
@avats-dev
Copy link

avats-dev commented Oct 3, 2020 •
edited
Loading

@ca-scribner @knqyf263 can you guide a bit how to resolve this? I would like to work on this. Thanks

@karancode karancode mentioned this issue Oct 3, 2020
Merged
@ca-scribner
Copy link
Author

Sorry I don't really know boltdb or the inners of trivy, but the ideal behaviour would be that one could run two instances of trivy at the same time. Maybe trivy doesn't need to keep a lock on the database for the whole scan and can instead grab the db at the start, do some things, release the lock, scan, then grab the db again? That might help with concurrent uses.

I haven't tried it, but I'm guessing the client/server setup for trivy is meant to handle this sort of thing? It felt like overkill for my use case (I had a few windows open at once on my local machine) but for large scale maybe that solves this issue?

@NitroCao NitroCao mentioned this issue May 5, 2022
Closed
@NitroCao NitroCao mentioned this issue Nov 11, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. kind/documentation Categorizes issue or PR as related to documentation.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update readme for parallel run issue karancode/trivy
3 participants
@knqyf263 @avats-dev @ca-scribner