Skip to content

Add data_index_key#53

Merged
elffjs merged 2 commits into
mainfrom
add_data_index_key
Apr 27, 2026
Merged

Add data_index_key#53
elffjs merged 2 commits into
mainfrom
add_data_index_key

Conversation

@elffjs

@elffjs elffjs commented Apr 26, 2026

Copy link
Copy Markdown
Member

We wanted to store the data from large documents (per some configurable threshold) in separate S3 objects so that they can be downloaded easily through presigned S3 links and not balloon the size of Parquet files. Since DIMO-Network/dis#254 we've been storing these large events as standalone JSON objects, including the headers. But clients do not enjoy unwrapping the data inside the event and parsing the base64.

To accommodate this, we introduce a new ClickHouse column data_index_key. Unlike the existing index_key, this column is also written to the Parquet files (morally, we want to be able to reconstruct a working system from S3 alone). In Go we represent this in a new StoredEvent type, rather than adding to CloudEventHeader, for two reasons:

  1. Security. With this choice it's easier to ensure that client input is never placed into this field. If we failed at this, a client might be able to gain access to files that he should not be able to see.
  2. Separation. Many users of the cloudevent package don't care about ClickHouse storage at all. The struct currently doesn't bother them with an IndexKey field, and it would be nice to maintain this separation.

Old clients that assume the old table structures should still work, but they should upgrade soon.

@elffjs elffjs requested a review from zer0stars April 26, 2026 19:49
@elffjs elffjs merged commit 1d6fac1 into main Apr 27, 2026
4 checks passed
@elffjs elffjs deleted the add_data_index_key branch April 27, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant