Skip to content

Commit 1a7f88f

Browse files
authored
feat(aws): support provisioned mode for sqs, kafka, and msk events (#13835)
1 parent 5168612 commit 1a7f88f

14 files changed

Lines changed: 725 additions & 0 deletions

File tree

docs/sf/providers/aws/events/kafka.md

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,3 +195,32 @@ functions:
195195
maximumBatchingWindow: 30
196196
startingPosition: LATEST
197197
```
198+
199+
## Provisioned mode
200+
201+
[Provisioned mode](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#invocation-eventsourcemapping-provisioned-mode) gives the event source mapping a dedicated pool of event pollers with minimum and maximum bounds, for consistent low-latency processing. Configure it with `provisionedPollers`:
202+
203+
```yml
204+
functions:
205+
compute:
206+
handler: handler.compute
207+
events:
208+
- kafka:
209+
accessConfigurations:
210+
saslScram512Auth: arn:aws:secretsmanager:us-east-1:01234567890:secret:MyBrokerSecretName
211+
bootstrapServers:
212+
- abc3.xyz.com:9092
213+
topic: MyTopic
214+
provisionedPollers:
215+
min: 1 # optional, 1-200, AWS default 1
216+
max: 100 # optional, 1-2000, AWS default 200
217+
group: shared-capacity # optional poller group name
218+
```
219+
220+
Provisioned mode incurs additional AWS charges per event poller — the `min` value is an always-on cost. See [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/).
221+
222+
`group` (letters, digits, hyphens and underscores, up to 128 characters) groups up to 100 event source mappings within the event source's VPC to share Event Poller Unit capacity, reducing provisioned-mode cost; the aggregate `max` across a group cannot exceed 2,000.
223+
224+
Deleting the `group` line removes the mapping from its poller group on the next deploy; its own `min` and `max` are unaffected.
225+
226+
**Disabling provisioned mode:** removing `provisionedPollers` from your configuration does **not** disable it — provisioned mode stays active on the deployed event source mapping, so the pollers (and their cost) remain. The same applies to `rollback` to an older deployment. To actually disable it, deploy once with `provisionedPollers: false`. `false` clears provisioned mode on an existing mapping; it cannot be used on a mapping that is being created for the first time — for new mappings, simply omit the key.

docs/sf/providers/aws/events/msk.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,3 +147,29 @@ functions:
147147
## IAM Permissions
148148

149149
The Serverless Framework will automatically configure the most minimal set of IAM permissions for you. However you can still add additional permissions if you need to. Read the official [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html) for more information about IAM Permissions for MSK events.
150+
151+
## Provisioned mode
152+
153+
[Provisioned mode](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#invocation-eventsourcemapping-provisioned-mode) gives the event source mapping a dedicated pool of event pollers with minimum and maximum bounds, for consistent low-latency processing. Configure it with `provisionedPollers`:
154+
155+
```yml
156+
functions:
157+
compute:
158+
handler: handler.compute
159+
events:
160+
- msk:
161+
arn: arn:aws:kafka:region:XXXXXX:cluster/MyCluster/xxxx-xxxxx-xxxx
162+
topic: mytopic
163+
provisionedPollers:
164+
min: 1 # optional, 1-200, AWS default 1
165+
max: 100 # optional, 1-2000, AWS default 200
166+
group: shared-capacity # optional poller group name
167+
```
168+
169+
Provisioned mode incurs additional AWS charges per event poller — the `min` value is an always-on cost. See [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/).
170+
171+
`group` (letters, digits, hyphens and underscores, up to 128 characters) groups up to 100 event source mappings within the event source's VPC to share Event Poller Unit capacity, reducing provisioned-mode cost; the aggregate `max` across a group cannot exceed 2,000.
172+
173+
Deleting the `group` line removes the mapping from its poller group on the next deploy; its own `min` and `max` are unaffected.
174+
175+
**Disabling provisioned mode:** removing `provisionedPollers` from your configuration does **not** disable it — provisioned mode stays active on the deployed event source mapping, so the pollers (and their cost) remain. The same applies to `rollback` to an older deployment. To actually disable it, deploy once with `provisionedPollers: false`. `false` clears provisioned mode on an existing mapping; it cannot be used on a mapping that is being created for the first time — for new mappings, simply omit the key.

docs/sf/providers/aws/events/sqs.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,43 @@ functions:
116116
maximumConcurrency: 250
117117
```
118118

119+
## Provisioned mode
120+
121+
[Provisioned mode](https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-scaling.html) gives the event source mapping a dedicated pool of event pollers with minimum and maximum bounds, for faster scaling and higher throughput than the default on-demand polling. Configure it with `provisionedPollers`:
122+
123+
```yml
124+
functions:
125+
compute:
126+
handler: handler.compute
127+
events:
128+
- sqs:
129+
arn: arn:aws:sqs:region:XXXXXX:myQueue
130+
provisionedPollers:
131+
min: 2 # optional, 2-200, AWS default 2
132+
max: 500 # optional, 2-10000, AWS default 200
133+
```
134+
135+
Provisioned mode incurs additional AWS charges per event poller — the `min` value is an always-on cost. See [AWS Lambda pricing](https://aws.amazon.com/lambda/pricing/).
136+
137+
`provisionedPollers` and `maximumConcurrency` are mutually exclusive scaling modes — setting both fails validation at package time. In provisioned mode, control concurrency through `max` (each SQS event poller drives up to 10 concurrent invocations).
138+
139+
**Disabling provisioned mode:** removing `provisionedPollers` from your configuration does **not** disable it — provisioned mode stays active on the deployed event source mapping, so the pollers (and their cost) remain. The same applies to `rollback` to an older deployment. To actually disable it, deploy once with:
140+
141+
```yml
142+
provisionedPollers: false
143+
```
144+
145+
`false` clears provisioned mode on an existing mapping. It cannot be used on a mapping that is being created for the first time — for new mappings, simply omit the key.
146+
147+
**Switching back to `maximumConcurrency`:** set both keys for one deploy — `false` clears provisioned mode while the concurrency limit applies — then optionally remove the `false` line:
148+
149+
```yml
150+
- sqs:
151+
arn: arn:aws:sqs:region:XXXXXX:myQueue
152+
maximumConcurrency: 250
153+
provisionedPollers: false
154+
```
155+
119156
## IAM Permissions
120157

121158
The Serverless Framework will automatically configure the most minimal set of IAM permissions for you. However you can still add additional permissions if you need to. Read the official [AWS documentation](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html) for more information about IAM Permissions for SQS events.

docs/sf/providers/aws/guide/serverless.yml.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1231,6 +1231,14 @@ functions:
12311231
functionResponseType: ReportBatchItemFailures
12321232
filterPatterns:
12331233
- a: [1, 2]
1234+
# Optional, must be in 2-1000 range. Mutually exclusive with provisionedPollers
1235+
maximumConcurrency: 250
1236+
# Optional, provisioned mode: dedicated event pollers (additional cost). Set to false to disable on an existing mapping
1237+
provisionedPollers:
1238+
# Optional, must be in 2-200 range
1239+
min: 2
1240+
# Optional, must be in 2-10000 range
1241+
max: 500
12341242
```
12351243
12361244
### Streams
@@ -1286,6 +1294,14 @@ functions:
12861294
filterPatterns:
12871295
- value:
12881296
a: [1, 2]
1297+
# Optional, provisioned mode: dedicated event pollers (additional cost). Set to false to disable on an existing mapping
1298+
provisionedPollers:
1299+
# Optional, must be in 1-200 range
1300+
min: 1
1301+
# Optional, must be in 1-2000 range
1302+
max: 100
1303+
# Optional, poller group name (letters, digits, hyphens, underscores) shared by up to 100 mappings in the same VPC
1304+
group: shared-capacity
12891305
```
12901306
12911307
### ActiveMQ
@@ -1352,6 +1368,14 @@ functions:
13521368
# Optional, specifies event pattern content filtering
13531369
filterPatterns:
13541370
- eventName: INSERT
1371+
# Optional, provisioned mode: dedicated event pollers (additional cost). Set to false to disable on an existing mapping
1372+
provisionedPollers:
1373+
# Optional, must be in 1-200 range
1374+
min: 1
1375+
# Optional, must be in 1-2000 range
1376+
max: 100
1377+
# Optional, poller group name (letters, digits, hyphens, underscores) shared by up to 100 mappings in the same VPC
1378+
group: shared-capacity
13551379
```
13561380
13571381
### RabbitMQ

packages/serverless/lib/plugins/aws/package/compile/events/kafka.js

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,43 @@ events:
128128
pattern: '[a-zA-Z0-9-/*:_+=.@-]*',
129129
},
130130
filterPatterns: { $ref: '#/definitions/filterPatterns' },
131+
provisionedPollers: {
132+
description: `Provisioned mode for the event source mapping — dedicated event pollers with min/max bounds and optional poller group. Set to false to disable provisioned mode on an existing mapping.
133+
@see https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#invocation-eventsourcemapping-provisioned-mode
134+
@example
135+
provisionedPollers:
136+
min: 1
137+
max: 100`,
138+
anyOf: [
139+
{ const: false },
140+
{
141+
type: 'object',
142+
properties: {
143+
min: {
144+
description: `Minimum number of provisioned event pollers (1-200). AWS default: 1.`,
145+
type: 'integer',
146+
minimum: 1,
147+
maximum: 200,
148+
},
149+
max: {
150+
description: `Maximum number of provisioned event pollers (1-2000). AWS default: 200.`,
151+
type: 'integer',
152+
minimum: 1,
153+
maximum: 2000,
154+
},
155+
group: {
156+
description: `Provisioned poller group name — groups up to 100 event source mappings in the event source's VPC to share Event Poller Unit capacity. Letters, digits, hyphens and underscores only.`,
157+
type: 'string',
158+
minLength: 1,
159+
maxLength: 128,
160+
pattern: '^[a-zA-Z0-9_-]+$',
161+
},
162+
},
163+
minProperties: 1,
164+
additionalProperties: false,
165+
},
166+
],
167+
},
131168
},
132169
additionalProperties: false,
133170
required: ['accessConfigurations', 'bootstrapServers', 'topic'],
@@ -352,6 +389,33 @@ events:
352389
}
353390
}
354391

392+
const provisionedPollers = event.kafka.provisionedPollers
393+
if (provisionedPollers && typeof provisionedPollers === 'object') {
394+
if (
395+
provisionedPollers.min != null &&
396+
provisionedPollers.max != null &&
397+
provisionedPollers.min > provisionedPollers.max
398+
) {
399+
throw new ServerlessError(
400+
`The "kafka" event of function "${functionName}" has "provisionedPollers.min" (${provisionedPollers.min}) greater than "max" (${provisionedPollers.max}).`,
401+
'EVENT_PROVISIONED_POLLERS_INVALID',
402+
)
403+
}
404+
kafkaResource.Properties.ProvisionedPollerConfig = {
405+
...(provisionedPollers.min != null && {
406+
MinimumPollers: provisionedPollers.min,
407+
}),
408+
...(provisionedPollers.max != null && {
409+
MaximumPollers: provisionedPollers.max,
410+
}),
411+
...(provisionedPollers.group && {
412+
PollerGroupName: provisionedPollers.group,
413+
}),
414+
}
415+
} else if (provisionedPollers === false) {
416+
kafkaResource.Properties.ProvisionedPollerConfig = {}
417+
}
418+
355419
cfTemplate.Resources[kafkaEventLogicalId] = kafkaResource
356420
}
357421

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
/**
2+
* Pure builder for the `provisionedPollers` block of the
3+
* sfcore.analysis.generated.v1 analytics event. Consumed by the sf-core
4+
* framework runner's getAnalysisEventDetails().
5+
*
6+
* Contract (mirrors sandboxes/mcp analytics builders):
7+
* - Fixed keys only — never a user-authored string (the poller group NAME
8+
* is never reported, only its presence count).
9+
* - Explicit-only + omit-empty: a key appears only when a user set it.
10+
* - HARD REQUIREMENT: total function — never throws; malformed input
11+
* degrades to {} so analytics can never break a user command.
12+
*/
13+
14+
const isObj = (v) => v !== null && typeof v === 'object' && !Array.isArray(v)
15+
16+
const sortedUniqueNumbers = (values) =>
17+
[...new Set(values.filter((n) => typeof n === 'number'))].sort(
18+
(a, b) => a - b,
19+
)
20+
21+
const SOURCES = ['sqs', 'kafka', 'msk']
22+
// `group` (PollerGroupName) exists only on kafka/msk; the sqs schema rejects it,
23+
// so a group on sqs is invalid config and is never reported.
24+
const SOURCES_WITH_GROUP = new Set(['kafka', 'msk'])
25+
26+
export const buildProvisionedPollersAnalytics = (config) => {
27+
try {
28+
const functions = isObj(config?.functions)
29+
? Object.values(config.functions)
30+
: []
31+
const perSource = {}
32+
for (const source of SOURCES) {
33+
const values = []
34+
for (const fn of functions) {
35+
const events = Array.isArray(fn?.events) ? fn.events : []
36+
for (const event of events) {
37+
if (!isObj(event?.[source])) continue
38+
const value = event[source].provisionedPollers
39+
if (value !== undefined) values.push(value)
40+
}
41+
}
42+
const objects = values.filter(isObj)
43+
const block = {}
44+
if (objects.length > 0) block.configured = objects.length
45+
const disabled = values.filter((v) => v === false).length
46+
if (disabled > 0) block.disabled = disabled
47+
const min = sortedUniqueNumbers(objects.map((o) => o.min))
48+
if (min.length > 0) block.min = min
49+
const max = sortedUniqueNumbers(objects.map((o) => o.max))
50+
if (max.length > 0) block.max = max
51+
if (SOURCES_WITH_GROUP.has(source)) {
52+
const groups = objects.filter(
53+
(o) => typeof o.group === 'string' && o.group.length > 0,
54+
).length
55+
if (groups > 0) block.groups = groups
56+
}
57+
if (Object.keys(block).length > 0) perSource[source] = block
58+
}
59+
return Object.keys(perSource).length > 0
60+
? { provisionedPollers: perSource }
61+
: {}
62+
} catch {
63+
return {}
64+
}
65+
}

packages/serverless/lib/plugins/aws/package/compile/events/msk/index.js

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,43 @@ msk:
7676
@see https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html`,
7777
$ref: '#/definitions/filterPatterns',
7878
},
79+
provisionedPollers: {
80+
description: `Provisioned mode for the event source mapping — dedicated event pollers with min/max bounds and optional poller group. Set to false to disable provisioned mode on an existing mapping.
81+
@see https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventsourcemapping.html#invocation-eventsourcemapping-provisioned-mode
82+
@example
83+
provisionedPollers:
84+
min: 1
85+
max: 100`,
86+
anyOf: [
87+
{ const: false },
88+
{
89+
type: 'object',
90+
properties: {
91+
min: {
92+
description: `Minimum number of provisioned event pollers (1-200). AWS default: 1.`,
93+
type: 'integer',
94+
minimum: 1,
95+
maximum: 200,
96+
},
97+
max: {
98+
description: `Maximum number of provisioned event pollers (1-2000). AWS default: 200.`,
99+
type: 'integer',
100+
minimum: 1,
101+
maximum: 2000,
102+
},
103+
group: {
104+
description: `Provisioned poller group name — groups up to 100 event source mappings in the event source's VPC to share Event Poller Unit capacity. Letters, digits, hyphens and underscores only.`,
105+
type: 'string',
106+
minLength: 1,
107+
maxLength: 128,
108+
pattern: '^[a-zA-Z0-9_-]+$',
109+
},
110+
},
111+
minProperties: 1,
112+
additionalProperties: false,
113+
},
114+
],
115+
},
79116
},
80117
additionalProperties: false,
81118
required: ['arn', 'topic'],
@@ -200,6 +237,33 @@ msk:
200237
}
201238
}
202239

240+
const provisionedPollers = event.msk.provisionedPollers
241+
if (provisionedPollers && typeof provisionedPollers === 'object') {
242+
if (
243+
provisionedPollers.min != null &&
244+
provisionedPollers.max != null &&
245+
provisionedPollers.min > provisionedPollers.max
246+
) {
247+
throw new ServerlessError(
248+
`The "msk" event of function "${functionName}" has "provisionedPollers.min" (${provisionedPollers.min}) greater than "max" (${provisionedPollers.max}).`,
249+
'EVENT_PROVISIONED_POLLERS_INVALID',
250+
)
251+
}
252+
mskResource.Properties.ProvisionedPollerConfig = {
253+
...(provisionedPollers.min != null && {
254+
MinimumPollers: provisionedPollers.min,
255+
}),
256+
...(provisionedPollers.max != null && {
257+
MaximumPollers: provisionedPollers.max,
258+
}),
259+
...(provisionedPollers.group && {
260+
PollerGroupName: provisionedPollers.group,
261+
}),
262+
}
263+
} else if (provisionedPollers === false) {
264+
mskResource.Properties.ProvisionedPollerConfig = {}
265+
}
266+
203267
mskStatement.Resource.push(eventSourceArn)
204268

205269
cfTemplate.Resources[mskEventLogicalId] = mskResource

0 commit comments

Comments
 (0)