AWS S3 destination type

[leggetter]

Several developers have asked about sending events from Outpost to Amazon S3 - for example, storing each event (or batch of events) as Parquet files in an S3 bucket for later processing.

Current Thinking

We don’t currently plan to add S3 as a destination in the near term. Outpost is focused on routing events to event buses (like Kafka, Google Pub/Sub, or EventBridge), rather than acting as a storage or data lake pipeline.

Here’s why we’re leaning that way:

• S3 is a better fit for native AWS tools. Services like Kinesis and EventBridge already support direct delivery to S3, and often handle the batching, compression, and schema concerns more effectively.

• Outpost is not optimized for storage workflows. Supporting formats like Parquet would require Outpost to handle batching, compression, and schema management, which shifts it away from its core purpose as an event delivery gateway.

• Complexity tradeoff. Adding S3 support would introduce new responsibilities (e.g. storage format logic, delivery guarantees, large file handling) that might be better solved downstream.

That said, we understand that some systems emit both real-time events and large state snapshots, and S3 is a common and affordable destination for those snapshots.

We’re leaving this issue open to track interest and feedback.

If sending data to S3 from Outpost would solve a real problem for you, please add a 👍 or leave a comment describing your use case — especially if it involves batching, file formats (like Parquet or JSONL), or volume requirements.

[utibeabasi6]

Hey
I would like to work on this if it isn't currently being worked on

[leggetter]

Hey I would like to work on this if it isn't currently being worked on

Hi @utibeabasi6 - thanks for expressing an interest in working on this. For the moment, we don't think that an S3 destination is a good idea for Outpost. See the updated description of this issue. However, we remain open to feedback and are willing to change our minds.

Were you interested in contributing because you had an underlying use case you needed to solve?

[eberle1080]

I work at Ampersand. We have a need for this feature. We have customers who want records placed in their S3 bucket. I'd prefer not to say specifically who on public channels but if you require specifics before allowing this to proceed, I can share details on our private slack channel.

I've implemented this functionality in a fork of your repository. The code is there. If you're open to the request, I could easily make it in to a PR against your repo. Just let me know. This would be helpful for us.

[alexbouchardd]

@eberle1080 To clarify, have you implemented it without batching, or did you also add batching functionalities? Do you save the file as JSON or another format?

[eberle1080]

It's without. Is batching a requirement?

[eberle1080]

And yes I'm serializing to JSON. But I allow various configurations, like bucket, region, storage class. And I use the S3 metadata to store metadata from the event.

[alexbouchardd]

Understood. My understanding from your use case is that you needed batching in paquet files. In the case where the files are uploaded as JSON without batching I can definitely see a stronger argument for supporting it in Outpost

[eberle1080]

Really there's 2 use-cases here:

• The specific customer (wants parquet, batching)
• Generic Ampersand capability to deliver to S3

It might be nice if we could have both, but I suspect the parquet + batching requirement is fairly niche, and won't be common. But just having the capability to deliver to S3 as something we can advertise as one of our core competencies would really help us attract new customers.

[alexbouchardd]

Thanks for the clarification, understood! As mentioned previously, we planned to "support" it through AWS EventBridge / Hookdeck, which supports S3 Upload natively. Is there a reason why that is "insufficient"?

[eberle1080]

We're self-hosting the outpost stack, and it would be less code for us if that portion could natively deliver S3, since we already have code that does Kinesis delivery via Outpost.