Store large documents as single objects#254
Merged
Merged
Conversation
elffjs
commented
Mar 17, 2026
| output: | ||
| switch: | ||
| cases: | ||
| # Large CloudEvents → individual document storage |
Member
Author
There was a problem hiding this comment.
Need to remove these slop comments.
Revisit this: this should not be possible
elffjs
changed the title
elffjs
added a commit
to DIMO-Network/cloudevent
that referenced
this pull request
Apr 27, 2026
We wanted to store the data from large documents (per some configurable threshold) in separate S3 objects so that they can be downloaded easily through presigned S3 links and not balloon the size of Parquet files. Since DIMO-Network/dis#254 we've been storing these large events as standalone JSON objects, including the headers. But clients do not enjoy unwrapping the data inside the event and parsing the base64. To accommodate this, we introduce a new ClickHouse column `data_index_key`. Unlike the existing `index_key`, this column is also written to the Parquet files (morally, we want to be able to reconstruct a working system from S3 alone). In Go we represent this in a new `StoredEvent` type, rather than adding to `CloudEventHeader`, for two reasons: 1. Security. With this choice it's easier to ensure that client input is never placed into this field. If we failed at this, a client might be able to gain access to files that he should not be able to see. 2. Separation. Many users of the cloudevent package don't care about ClickHouse storage at all. The struct currently doesn't bother them with an `IndexKey` field, and it would be nice to maintain this separation. Old clients that assume the old table structures should still work, but they should upgrade soon.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If a CloudEvent's size is larger than a configurable threshold—1 MiB by default—write it to S3 as an individual object instead of as a row in a Parquet batch. This happens in a new stream,
output-document.yaml.This new stream has a very small custom Go processor that pairs each large event with a ClickHouse row message, much as in the
rawparquetpackage. We could do all of this in Bloblang if we wanted to, but doing it this way lets us use common helpers and keep the behavior consistent. The object keys look likeThe S3 and ClickHouse portions are extremely similar to those of the Parquet stream. We might consider using resources or chaining with another stream.