Skip to content

Max Scan Concurrency#1654

Merged
netomi merged 5 commits intoeclipse-openvsx:masterfrom
yeeth-security:yeeth/max_scan_concurrency
Mar 4, 2026
Merged

Max Scan Concurrency#1654
netomi merged 5 commits intoeclipse-openvsx:masterfrom
yeeth-security:yeeth/max_scan_concurrency

Conversation

@janbro
Copy link
Contributor

@janbro janbro commented Mar 3, 2026

Adds cross-pod concurrency limiting for external scanners using a periodic dispatcher backed by the shared database. A recurring JobRunr job claims QUEUED jobs in FIFO order up to a configurable max-concurrency per scanner, preventing overload on external scanning services. Startup and watchdog recovery handle concurrency-limited scanners without interfering with the dispatcher.

Two additional configuration properties for configured scanners:

ovsx:
  scanning:
    configured:
      my-scanner:
        max-concurrency: 4           # max simultaneous jobs across all pods
        max-queue-wait-minutes: 120   # fail queued jobs after this long
  • max-concurrency - Maximum simultaneous jobs (PROCESSING + SUBMITTED) for this scanner across all pods. A periodic dispatcher promotes QUEUED jobs in FIFO order up to this limit. Set to -1 (default) for unlimited.
  • max-queue-wait-minutes - Maximum time a job may sit in QUEUED before being marked FAILED. Only relevant when max-concurrency is positive. Prevents unbounded queue growth if the external service is down. Defaults to 120 minutes.

janbro added 3 commits March 1, 2026 18:35
…her to take into account processing and submitted jobs when rate limiting jobs to external services
Copy link
Contributor

@netomi netomi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creation of the indices in the ScannerJob entity might require a db migration, need to test that first

@netomi
Copy link
Contributor

netomi commented Mar 4, 2026

can you allow maintainers to push to the branch or resolve the conflicts yourself?

@janbro
Copy link
Contributor Author

janbro commented Mar 4, 2026

Just resolved the conflict. For some reason not seeing the option to allow maintainers to edit

@janbro
Copy link
Contributor Author

janbro commented Mar 4, 2026

resolving the ci error

@netomi netomi merged commit bd6245b into eclipse-openvsx:master Mar 4, 2026
4 checks passed
@netomi
Copy link
Contributor

netomi commented Mar 5, 2026

the additional index needs a db migration, will add it accordingly, there are also other things that are out of sync

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants