feat: UEX API rate limiting and advisory lock service#210
Merged
Conversation
- Add UexApiClient: shared Axios wrapper for ETL step implementations (#190–#199) with configurable inter-request delay (UEX_REQUEST_DELAY_MS, default 500ms), reactive backoff when x-ratelimit-remaining ≤ 5 (doubles delayMs), 429 retry with Retry-After header and exponential backoff up to 3 attempts, 5xx → UEXServerException. Legacy per-endpoint clients (UEXCategoriesClient etc.) are intentionally not migrated. - Add AdvisoryLockService: reusable withLock(key, fn) using QueryRunner to pin both pg_try_advisory_lock and pg_advisory_unlock to the same physical connection — pool-based DataSource.query() cannot guarantee same-session execution and risks stranding locks on recycled connections. - Wire AdvisoryLockService into CatalogEtlService; remove inline lock logic and DataSource injection from that service. - Register UexApiClient as a factory provider in UexSyncModule; export it for injection by upcoming ETL step implementations. - Document UEX_REQUEST_DELAY_MS in .env.example. - 18 unit tests: UexApiClient (7), AdvisoryLockService (3), CatalogEtlService (8)
…ient
UEX returns HTTP 200 with body { status: 'error', message: 'requests_limit_reached' }
as a soft throttle signal. UexApiClient was only handling HTTP 429, so ETL
steps would receive null data and continue silently instead of backing off.
Now throws RateLimitException on that response shape, matching the behaviour
of the legacy per-endpoint clients. Added a unit test for this case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
UexApiClient— shared Axios wrapper for all ETL step implementations (ETL: sync factions, jurisdictions, and companies into station_* tables #190–ETL: sync commodities with self-referencing parent hierarchy #199). Applies a configurable inter-request delay (UEX_REQUEST_DELAY_MS, default 500ms) before every call. After each response, inspectsx-ratelimit-remaining; if ≤ 5, doublesdelayMsfor the next request. On HTTP 429, readsRetry-Afterheader and retries with exponential backoff up to 3 times before throwingRateLimitException. 5xx responses throwUEXServerException. Registered as a factory provider inUexSyncModuleand exported for injection by upcoming ETL steps.AdvisoryLockService— reusablewithLock(key, fn)using aQueryRunnerto pin bothpg_try_advisory_lockandpg_advisory_unlockto the same physical PostgreSQL connection. Pool-basedDataSource.query()cannot guarantee same-session execution and risks leaving locks stranded on recycled connections. Lock is released unconditionally infinally; throwsConflictExceptionon contention.CatalogEtlService— replaced inline advisory lock logic andDataSourceinjection withAdvisoryLockService.withLock()..env.example— documentsUEX_REQUEST_DELAY_MS=500.Test plan
pnpm typecheckpassesnpx jest "uex-api.client|advisory-lock|catalog-etl" --no-coverage— 18 tests passPOST /admin/catalog-etl/runsucceeds; a second concurrent call returns HTTP 409Notes
The legacy per-endpoint clients (
UEXCategoriesClient,UEXItemsClient, etc.) are intentionally not migrated toUexApiClient— they predate this issue. New ETL step implementations (#190–#199) will injectUexApiClientdirectly.Closes #189