feat: exploit intelligence integration

[Strum355]

Summary

Feature branch merge for Exploit Intelligence integration.

Tasks implemented:

• TC-4675: feat(entity): add exploit intelligence job tracking entities and migration #2435
• TC-4677: feat: add exploit-intelligence job polling, configuration & API endpoints #2464

Implements TC-4441

Related trustify-ui PR: guacsec/trustify-ui#1125

[sourcery-ai]

Sorry @Strum355, your pull request is larger than the review limit of 150000 diff characters

[codecov]

Codecov Report

❌ Patch coverage is 82.80936% with 257 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.75%. Comparing base (93e2f8e) to head (f715b7f).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
modules/exploit-intelligence/src/service/mod.rs	74.60%	43 Missing and 5 partials ⚠️
modules/exploit-intelligence/src/runner/worker.rs	76.74%	30 Missing and 10 partials ⚠️
modules/exploit-intelligence/src/runner/polling.rs	82.67%	15 Missing and 20 partials ⚠️
...odules/exploit-intelligence/src/runner/analysis.rs	80.45%	18 Missing and 16 partials ⚠️
modules/exploit-intelligence/src/runner/mod.rs	50.00%	30 Missing ⚠️
modules/exploit-intelligence/src/error.rs	39.28%	17 Missing ⚠️
modules/exploit-intelligence/src/endpoints/mod.rs	82.27%	4 Missing and 10 partials ⚠️
server/src/profile/api.rs	93.10%	9 Missing and 3 partials ⚠️
entity/src/exploit_intelligence_job.rs	0.00%	6 Missing ⚠️
entity/src/exploit_intelligence_job_component.rs	0.00%	6 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2534      +/-   ##
==========================================
+ Coverage   72.13%   72.75%   +0.61%     
==========================================
  Files         466      483      +17     
  Lines       28634    30166    +1532     
  Branches    28634    30166    +1532     
==========================================
+ Hits        20654    21946    +1292     
- Misses       6810     6974     +164     
- Partials     1170     1246      +76     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ctron]

I think that, unless impossible, this should go into it's own module. It doesn't seem to be "fundemental".

Also, how can the UI evaluate if this feature is configured or not?

[ruromero]

Review: Exploit Intelligence Integration

Solid feature with well-designed job queue, comprehensive tests, and good resilience patterns. The convention and code quality items below should be addressed before merge.

See inline comments for details.

[Strum355]

Also, how can the UI evaluate if this feature is configured or not?

Thanks for pointing this out, seems my changes disappeared into a git stash. Im taking the approach of adding a flag to /.well-known/trustify, any concerns with that? Seems reasonably straightforward and clean

{"version":"0.5.0-rc.1","readOnly":false,"exploitIntelligence":false,"build":{...}}

I think that, unless impossible, this should go into it's own module. It doesn't seem to be "fundemental".

Ill have a look into this 👍

[ctron]

The command line arguments of EI mix with others and are inconsistently named:

| Exploit Intelligence:
|       --exploit-intelligence-url <EXPLOIT_INTELLIGENCE_URL>
|           Base URL of the Exploit Intelligence client service
|           
|           [env: EXPLOIT_INTELLIGENCE_URL=]
| 
|       --exploit-intelligence-ui-url <EXPLOIT_INTELLIGENCE_UI_URL>
|           Base URL of the Exploit Intelligence web UI for deep-linking to reports
|           
|           [env: EXPLOIT_INTELLIGENCE_UI_URL=]
| 
|       --exploit-intelligence-poll-interval <EXPLOIT_INTELLIGENCE_POLL_INTERVAL>
|           Polling interval for EI analysis completion (humantime, e.g. "30s")
|           
|           [env: EXPLOIT_INTELLIGENCE_POLL_INTERVAL=]
|           [default: 30s]
| 
|       --exploit-intelligence-max-poll-duration <EXPLOIT_INTELLIGENCE_MAX_POLL_DURATION>
|           Maximum duration before EI polling is considered timed out (humantime, e.g. "30m")
|           
|           [env: EXPLOIT_INTELLIGENCE_MAX_POLL_DURATION=]
|           [default: 30m]
| 
|       --exploit-intelligence-upload-max-retries <EXPLOIT_INTELLIGENCE_UPLOAD_MAX_RETRIES>
|           Maximum number of upload retry attempts for EI analysis
|           
|           [env: EXPLOIT_INTELLIGENCE_UPLOAD_MAX_RETRIES=]
|           [default: 3]
| 
|       --exploit-intelligence-upload-retry-delay <EXPLOIT_INTELLIGENCE_UPLOAD_RETRY_DELAY>
|           Initial delay between upload retries (humantime, exponential backoff)
|           
|           [env: EXPLOIT_INTELLIGENCE_UPLOAD_RETRY_DELAY=]
|           [default: 1s]
| 
|       --exploit-intelligence-max-consecutive-poll-failures <EXPLOIT_INTELLIGENCE_MAX_CONSECUTIVE_POLL_FAILURES>
|           Maximum consecutive poll failures before giving up
|           
|           [env: EXPLOIT_INTELLIGENCE_MAX_CONSECUTIVE_POLL_FAILURES=]
|           [default: 5]
| 
|       --worker-poll-interval <WORKER_POLL_INTERVAL>
|           How often the background worker checks for new EI jobs
|           
|           [env: EXPLOIT_INTELLIGENCE_WORKER_POLL_INTERVAL=]
|           [default: 5s]
| 
|       --exploit-intelligence-auth-token <EXPLOIT_INTELLIGENCE_AUTH_TOKEN>
|           Authentication token for the Exploit Intelligence service (static token, for backward compatibility)
|           
|           [env: EXPLOIT_INTELLIGENCE_AUTH_TOKEN=]
| 
| Exploit Intelligence OIDC:
|       --ei-oidc-client-id <ei_oidc_client_id>
|           [env: EXPLOIT_INTELLIGENCE_OIDC_CLIENT_ID=]
| 
|       --ei-oidc-client-secret <ei_oidc_client_secret>
|           [env: EXPLOIT_INTELLIGENCE_OIDC_CLIENT_SECRET=]
| 
|       --ei-oidc-issuer-url <ei_oidc_issuer_url>
|           [env: EXPLOIT_INTELLIGENCE_OIDC_ISSUER_URL=]
| 
|       --ei-oidc-refresh-before <ei_oidc_refresh_before>
|           [env: EXPLOIT_INTELLIGENCE_OIDC_REFRESH_BEFORE=]
|           [default: 30s]
| 
|       --ei-oidc-tls-insecure
|           [env: EXPLOIT_INTELLIGENCE_OIDC_TLS_INSECURE=]
| 
|       --max-cache-size <max-cache-size>
|           Maximum size of the graph cache.
|           
|           [env: TRUSTD_MAX_CACHE_SIZE=8GiB]
|           [default: "200 MiB"]
| 
|       --concurrency <CONCURRENCY>
|           The number of concurrent tasks for analysis (must be > 0).
|           
|           [env: TRUSTIFY_ANALYSIS_CONCURRENCY=]
|           [default: 10]
| 
|       --sbom-published-cutoff-days <SBOM_PUBLISHED_CUTOFF_DAYS>
|           Only consider SBOMs published within this many days. Disabled by default.
|           
|           [env: TRUSTIFY_LATEST_SBOM_PUBLISHED_CUTOFF_DAYS=]

[ctron]

When setting the EI URL, but not the EI UI URL, what's expected to happen?

[ctron]

Are --exploit-intelligence-auth-token <EXPLOIT_INTELLIGENCE_AUTH_TOKEN> and the OIDC configuration mutually exclusive? Then we need to enforce this using clap.

[ctron]

Looks like a great improvement. Less complex. Lot of minor things to clean up. Error handling, types, imports and common stuff.

Also I noticed that the arguments are still mixed:

Exploit Intelligence OIDC:
      --ei-oidc-client-id <ei_oidc_client_id>
          [env: EXPLOIT_INTELLIGENCE_OIDC_CLIENT_ID=]

      --ei-oidc-client-secret <ei_oidc_client_secret>
          [env: EXPLOIT_INTELLIGENCE_OIDC_CLIENT_SECRET=]

      --ei-oidc-issuer-url <ei_oidc_issuer_url>
          [env: EXPLOIT_INTELLIGENCE_OIDC_ISSUER_URL=]

      --ei-oidc-refresh-before <ei_oidc_refresh_before>
          [env: EXPLOIT_INTELLIGENCE_OIDC_REFRESH_BEFORE=]
          [default: 30s]

      --ei-oidc-tls-insecure
          [env: EXPLOIT_INTELLIGENCE_OIDC_TLS_INSECURE=]

      --max-cache-size <max-cache-size>
          Maximum size of the graph cache.
          
          [env: TRUSTD_MAX_CACHE_SIZE=]
          [default: "200 MiB"]

      --concurrency <CONCURRENCY>
          The number of concurrent tasks for analysis (must be > 0).
          
          [env: TRUSTIFY_ANALYSIS_CONCURRENCY=]
          [default: 10]

      --sbom-published-cutoff-days <SBOM_PUBLISHED_CUTOFF_DAYS>
          Only consider SBOMs published within this many days. Disabled by default.
          
          [env: TRUSTIFY_LATEST_SBOM_PUBLISHED_CUTOFF_DAYS=]

EI is claiming other settings.