feat: add filesystem payload offloader and retries - #681
Conversation
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Error
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| Thread.currentThread().interrupt(); | ||
| var interrupted = new RetryablePayloadOffloadException( | ||
| String.format("Interrupted while waiting to retry payload %s after attempt %d", action, attempt), | ||
| e); | ||
| interrupted.addSuppressed(failure); | ||
| throw interrupted; |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_5vmint2jktj7b53xftxnhr7dib
[P1] An interrupted backoff is converted to RetryablePayloadOffloadException. DurableExecutor treats this subtype as RETRYING, so cancellation can trigger a fresh Lambda invocation and repeat storage work. The filesystem classifier similarly marks actual I/O interruptions retryable. Restore the interrupt flag but throw a non-retryable PayloadOffloadException, and classify ClosedByInterruptException or an interrupt-signaled InterruptedIOException as permanent interruption.
| var inlinePayload = OffloadedPayload.inline( | ||
| serializedPayload, context.durableExecutionArn(), context.entityId(), payloadDigest) | ||
| .bindProducer(context, payloadDigest); | ||
| if (fitsCheckpoint(inlinePayload)) { |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_6olmknqlyp3zhgezq3cu4y2oso
[P1] OVERFLOW sizing materializes the complete escaped envelope and then another UTF-8 byte array through PayloadCodec.envelopeSizeBytes. For the multi-megabyte payloads this offloader is intended to handle, these extra full-size copies can exhaust Lambda memory before the code chooses filesystem storage. Short-circuit when a bounded UTF-8 count already exceeds the limit, and perform exact envelope encoding only for near-boundary payloads.
| var expectedPrefix = payloadOwnerPrefix(payload.ownerDurableExecutionArn(), payload.ownerEntityId()) | ||
| + "-" | ||
| + payload.payloadDigest() | ||
| + "-"; |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_2cuyghirtpc6zjugxs55p2ygo2
[P2] Loading recomputes the expected filename using the current pathEncoding. An in-flight execution that wrote a URI-form reference will therefore fail replay after configuration changes to HASH, despite the persisted reference, owner, and digest remaining valid. Make encoding a write-only choice by accepting both versioned filename formats during load, or persist the encoding with the reference, and add a cross-configuration replay test.
| import software.amazon.lambda.durable.offload.PayloadStorageMode; | ||
| import software.amazon.lambda.durable.offload.SerDesPayloadKind; | ||
|
|
||
| class FileSystemPayloadOffloaderTest { |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_nlr5gumo4w2iw45unbkquvbkep
[P2] The new public offloader is tested only through direct method calls. That does not exercise DurableConfig, PayloadCodec, executor routing, checkpoint persistence, or replay, and project rules require integration coverage for public API changes. Add a LocalDurableTestRunner test that configures this offloader and verifies checkpoint/suspend/replay loading, including an OVERFLOW payload and retryable storage failure.
Codex AI reviewFound four actionable issues affecting interruption handling, large-payload reliability, replay compatibility, and required integration coverage. Reviewed commit |
1 participant
Stack
Scope
FileSystemPayloadOffloaderin the core SDKCREATE_NEWSecureDirectoryStream, no-follow access, direct-child confinement, ownership binding, and SHA-256 verificationALWAYS/OVERFLOW, URI/hash path encodings, envelope-size limits, and structured/custom previewsRetryPayloadOffloaderfor explicitly retryable storage failuresIntentionally excluded:
Validation
git diff --checkmvn spotless:checkRelated to #463.