csaf-walker versions

[helio-frota]

Comparing two versions of csaf-walker when executing the command:

cargo run --bin xtask --release generate-dump

Important

This part was commented out to focus only on CsafImporter link.
This part was modified to retrieve results from 2025 only link.

Version 0.10.2

• Release Date: November 27, 2024
• Execution Time: 26 minutes

Log Output:

2025-02-18T13:26:42.371797Z  INFO run_once:run_once_csaf: return=Ok(RunOutput { report: Report { start_date: 2025-02-18 13:00:38.099277186 +00:00:00, end_date: 2025-02-18 13:26:42.371795905 +00:00:00, number_of_items: 1851, messages: {} }, continuation: None }) context=Context { name: "run" } importer=CsafImporter { common: CommonImporter { disabled: false, period: 0ns, description: Some("Red Hat VEX documents from 2025"), labels: Labels({}) }, source: "redhat.com", v3_signatures: true, only_patterns: ["^cve-2025-"], fetch_retries: None, ignore_missing: false } last_success=None
2025-02-18T13:26:42.371820Z  INFO Done ingesting
2025-02-18T13:26:42.472775Z  INFO Dumped to: dump.sql

Version 0.11.0

• Release Date: February 18, 2025
• Execution Time: 27 minutes

• Start

2025-02-18T20:06:55.052060Z  INFO Ingesting: Red Hat VEX documents from 2025
2025-02-18T20:06:55.628496Z  INFO run_once:run_once_csaf: Loaded 1 public key
2025-02-18T20:06:55.628586Z  INFO run_once:run_once_csaf: processing 2 distribution URLs
2025-02-18T20:06:55.628604Z  INFO run_once:run_once_csaf: Walking directory URL: Directory(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("security.access.redhat.com")), port: None, path: "/data/csaf/v2/advisories/", query: None, fragment: None })
2025-02-18T20:06:56.163788Z  INFO run_once:run_once_csaf: Found 'changes.csv', processing data
2025-02-18T20:06:56.168937Z  INFO run_once:run_once_csaf: Detected 18563 entries
2025-02-18T20:06:56.191977Z  INFO run_once:run_once_csaf: Starting: run: Import CSAF: redhat.com - total: 18563

• End

2025-02-18T20:33:26.260043Z  INFO run_once:run_once_csaf: run: Import CSAF: redhat.com: 83.33% (ETA: 5m 17s) current=110709 total=132849 percent=83.33446243479439
2025-02-18T20:33:29.174834Z  INFO run_once:run_once_csaf: run: Import CSAF: redhat.com: 100% (complete) percent=100.0
2025-02-18T20:33:29.174882Z  INFO Done ingesting
2025-02-18T20:33:29.283349Z  INFO Dumped to: dump.sql

This part is not clear for me:

• Previous version shows number_of_items: 1851, the new version shows Detected 18563 entries
  • They are same thing with different names ?
• Other part is current=110709 total=132849 has relation with previous number_of_items ?

[ctron]

That may be the filtered out number of items. total vs processed?

[helio-frota]

Previous version shows number_of_items: 1851, the new version shows Detected 18563 entries

This happened again and looks like something being updated on the remote server 👍 ( or Was my rate limited? )

I was testing some things and started and stopped importers several times today

before

after some runs over time...

[helio-frota]

back to normal after cancelling and starting again 👍