Add EPSS Ingestor command to ingest-worker #1070
Conversation
Detailed Summary:
- This commit adds a new epss command to the ingest-worker.
- The epss command includes an "ingest" subcommand which allows the user to update EPSS scores.
- The epss command is implemented using the clifx library and includes a NewCommand function to create the cli.Command struct.
- The epss command has a dependency on an EPSSIngester interface which is provided through the fx dependency injection library.
- The epss command logs informative messages when starting and completing the ingestion process.
(Thank ChatGPT for the git commit message lol)
The SQL I wrote is the fastest that I could figure out. The first implementation took >10 minutes and this one takes about 30 seconds now.
It turns out that query for the IDs ahead of time and then running an "UPDATE" for that is the fastest way. (worth noting)
|
|
||
| // Get all CVEs and GHSAs from the database | ||
| cvePairs, err := tx.QueryContext(ctx, ` | ||
| SELECT v1.source_id, v2.source_id FROM vulnerability.equivalent e |
There was a problem hiding this comment.
im glad this works for now, but there must be a real library we can use for this in the future. passing the raw sql strings is starting to look clunky, although god knows ive done it a lot myself
There was a problem hiding this comment.
In node there are quite a few query builder libraries. There are also libraries which will read in .sql files and template stuff into them. That has the advantage of code separation, and it will allow the IDE to work some autocomplete magic on the SQL. We might consider that in the future
There was a problem hiding this comment.
The SQL autocomplete works great for me FWIW
There was a problem hiding this comment.
please run go fmt on your changes, they don't match the language style specifically use tabs for indentation and there is no limit on line length.
lunatrace/bsl/ingest-worker/cmd/ingestworker/vulnerability/vulnerability.go
Show resolved
Hide resolved
Update generated files Co-authored-by: breadchris <breadchris@users.noreply.github.com>
|
|
Detailed Summary:
(Thank ChatGPT for the git commit message lol)
The SQL I wrote is the fastest that I could figure out. The first implementation took >10 minutes and this one takes about 30 seconds now.
It turns out that query for the IDs ahead of time and then running an "UPDATE" for that is the fastest way. (worth noting)