docs: Clarify input transformation support for EventBridge Pipes #287
Conversation
HarshCasper
requested review from
gregfurman,
joe4dev,
quetzalliwrites and
tiurin
as code owners
| - Support for wildcards (`*`) in JSONPath in input transformation is not implemented. | ||
| - Lack of concurrency support (i.e., ParallelizationFactor), resulting in slower processing in high-throughput scenarios. | ||
| - Lack of lifecycle management for pipe states (i.e., missing tests for state transitions). | ||
| - Lack of re-sharding support when polling from Kinesis and DynamoDB streams. |
There was a problem hiding this comment.
Perhaps we can rather phrase this as "limited support", since we do have re-sharding functionality... it's just rough around the edges 😅
| - Lack of concurrency support (i.e., ParallelizationFactor), resulting in slower processing in high-throughput scenarios. | ||
| - Lack of lifecycle management for pipe states (i.e., missing tests for state transitions). | ||
| - Lack of re-sharding support when polling from Kinesis and DynamoDB streams. | ||
| - Batch handling behavior may have parity issues (e.g., batch flushing rules by size, length, time, etc. are not implemented). |
There was a problem hiding this comment.
We handle batching and have quite a few parity tests AFAIK 🤔 Unsure if it's worth mentioning this here.
See the ESM docs https://docs.localstack.cloud/aws/services/lambda/#behaviour-coverage (the pipes implementation is based off of the ESM one).
There was a problem hiding this comment.
@gregfurman this line used to be in the docs before. This PR's change only clarifies input transformation. If batch behavior is not described correctly I'd suggest to raise a separate docs ticket about it and clarify batching behavior there, without blocking this PR.
| - Lack of concurrency support (i.e., ParallelizationFactor), resulting in slower processing in high-throughput scenarios. | ||
| - Lack of lifecycle management for pipe states (i.e., missing tests for state transitions). | ||
| - Lack of re-sharding support when polling from Kinesis and DynamoDB streams. | ||
| - Batch handling behavior may have parity issues (e.g., batch flushing rules by size, length, time, etc. are not implemented). |
There was a problem hiding this comment.
@gregfurman this line used to be in the docs before. This PR's change only clarifies input transformation. If batch behavior is not described correctly I'd suggest to raise a separate docs ticket about it and clarify batching behavior there, without blocking this PR.
No description provided.