SDK External Payload Storage #765
Description
Is your feature request related to a problem? Please describe.
Some clients, workflows, and activities may produce payloads that are too large such that they cause workflow failures and require workflows to be rearchitected to brake down the data into smaller sizes. Sometimes customers have to write abstractions to offload large data into storage systems, pass around references, and manually retrieve that data.
Describe the solution you'd like
The SDK should provide an abstraction to allow customers to opt into offloading payloads to external storage systems when they are large or for other customer-defined determinations.
- Customers can use pre-built storage abstractions (e.g. drivers) for well-known storage systems (e.g. Amazon S3) or define their own that integrates with the external payload storage abstraction.
- External payload storage should remain transparent to workflows and activities. External payload storage may use additional contextual information to better inform how payloads should be externally stored, if at all.
- Offloaded payloads will be represented with claim information in workflow history so that they can be retrieved at a later time or by other systems using the same storage abstractions.
Per-SDK Tickets
- SDK External Payload Storage sdk-go#2208
- Java -
- TypeScript -
- SDK External Payload Storage sdk-python#1342
- .NET -
- Ruby -
- Rust -
- PHP -
- Temporal CLI - N/A