feat(utils/sqs): add MessageDeduplicationId support for FIFO sends#1566
Merged
akshaymagapu merged 1 commit intomainfrom Apr 29, 2026
Merged
feat(utils/sqs): add MessageDeduplicationId support for FIFO sends#1566akshaymagapu merged 1 commit intomainfrom
akshaymagapu merged 1 commit intomainfrom
Conversation
FIFO queues with content-based deduplication disabled require producers to supply an explicit MessageDeduplicationId on every publish. The existing sendMessage helper accepted MessageGroupId but had no way to pass MessageDeduplicationId through, so callers either had to bypass the helper or rely on content-based dedup at the queue level. This adds an optional fourth parameter, mirroring the existing messageGroupId pattern: ignored on standard queues, applied only on FIFO queues, and only when the caller passes a non-empty value. Backward compatible — existing 2/3-arg call sites are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This PR will trigger a minor release when merged. |
4 tasks
ekremney
approved these changes
Apr 29, 2026
solaris007
pushed a commit
that referenced
this pull request
Apr 29, 2026
## [@adobe/spacecat-shared-utils-v1.113.0](https://github.com/adobe/spacecat-shared/compare/@adobe/spacecat-shared-utils-v1.112.5...@adobe/spacecat-shared-utils-v1.113.0) (2026-04-29) ### Features * **utils/sqs:** add MessageDeduplicationId support for FIFO sends ([#1566](#1566)) ([b3b9511](b3b9511))
Member
|
🎉 This PR is included in version @adobe/spacecat-shared-utils-v1.113.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
akshaymagapu
added a commit
to adobe/spacecat-audit-worker
that referenced
this pull request
Apr 29, 2026
#2438) ## Summary Pairs with [spacecat-infrastructure#480](adobe/spacecat-infrastructure#480), which converts the analytics ingestion path (SNS topic + SQS queue + DLQ) to FIFO. Once that infra PR applies, the audit-worker's `dispatchAnalyticsEvent` will hard-fail on every attempt because the FIFO queue rejects publishes that lack `MessageGroupId` and `MessageDeduplicationId` (the new queue has `content_based_deduplication = false`). This PR adds the missing attributes. ## Changes **`src/support/sqs.js`** — extend the local `SQS.sendMessage` signature with an optional `msgDedupId` parameter and forward it as `MessageDeduplicationId`. Backward compatible: only set when the caller supplies a non-empty value, so all existing 2/3-arg call sites are untouched. **`src/cdn-logs-report/agentic-daily-export.js`** — `dispatchAnalyticsEvent`: - `MessageGroupId = ` `` `${pipeline_id}:${site_id}` `` per the infra PR's convention. Concurrent imports for the same scope serialize through the FIFO group; different sites/pipelines run in parallel. - `MessageDeduplicationId = batchId` (uuid generated per export run). Retries inside SQS's 5-minute dedup window collapse to a single delivery. - Return shape now exposes `messageGroupId` and `messageDeduplicationId` for observability. ## Why this group key `(pipeline_id, site_id)` matches the infra PR convention so the audit-worker's `agentic_traffic` import group is independent of the projector's post-success `agentic_traffic_daily_refresh` and `agentic_traffic_weekly_refresh` groups (different DB tables, no contention). Putting them in the same group would needlessly serialize unrelated work. ## Test plan - [x] `npx mocha --spec test/support/sqs.test.js --spec test/audits/cdn-logs-report/agentic-daily-export.test.js --exit` — 28 passing. - [x] New tests cover: dedup-id applied on FIFO publish, dedup-id omitted when not provided, log line includes DedupID, end-to-end `dispatchAnalyticsEvent` returns the new fields and calls `sendMessage` with all five args. - [x] Lint clean. - [ ] Coordinate apply-time with infra PR #480. ## Related - Pairs with [spacecat-infrastructure#480](adobe/spacecat-infrastructure#480) — must land + deploy in lockstep. - Pairs with the projector PR (separate) which adds FIFO attributes on the post-success refresh enqueues + DLQ writes. - Optional follow-up: [spacecat-shared#1566](adobe/spacecat-shared#1566) adds the same `messageDeduplicationId` parameter to the shared utils helper, useful for any future FIFO consumer using the shared sender. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
messageDeduplicationIdtoSQS.sendMessage.MessageDeduplicationIdon theSendMessageCommandonly when the caller passes a non-empty value.Why
FIFO queues with
content_based_deduplication = falserequire producers to supply an explicitMessageDeduplicationIdon every publish — without one, SQS rejects the call. Today the helper has no way to pass it through, so callers either bypass the helper or rely on content-based dedup at the queue level.This change unblocks producers (
spacecat-audit-worker,mysticat-projector-service) that need to publish to the analytics ingestion FIFO queue introduced in spacecat-infrastructure#480, where the topic and queue both have explicit dedup disabled per the design (producer sets explicit MessageDeduplicationId).Backward compatible — existing 2/3-arg call sites are unchanged.
Test plan
npx mocha --spec test/sqs.test.js --exit— all 1035 utils tests pass.Coordination
Pairs with consumer PRs that use the new parameter on the analytics FIFO publish path. This PR can land independently — it is purely additive.
🤖 Generated with Claude Code