Violation of Benchmark Rules Regarding JSON Flattening

[KVeschgini]

The benchmark rules explicitly state: “It is not allowed to use query results caching or flatten JSON into multiple non-JSON columns at insertion time.” However, the pipeline.yaml in the Greptime benchmark appears to define transformation rules that result in a table schema where critical fields from the commit JSON object are flattened into separate non-JSON columns:

CREATE TABLE IF NOT EXISTS "bluesky" (
  "did" STRING NULL,
  "kind" STRING NULL INVERTED INDEX,
  "commit_collection" STRING NULL INVERTED INDEX,
  "commit_operation" STRING NULL INVERTED INDEX,
  "commit" JSON NULL,
  "time_us" TIMESTAMP(6) NOT NULL,
  TIME INDEX ("time_us"),
  PRIMARY KEY ("kind", "commit_collection", "commit_operation")
)
ENGINE=mito
WITH(
  append_mode = 'true'
)

This schema directly violates the stated rule, as it includes several fields as separate columns instead of keeping them within the commit JSON structure.

[rschu1ze]

@KVeschgini Good find, I totally missed that in the original GreptimeDB submission. I marked the existing results as not retaining structure for now (here).

@shuiyisong We can leave everything as is, or, (preferred option), introduce a Greptime configuration which doesn't use flattening. For the latter, I would need some help from your side, i.e. basically provide a pipeline.yaml file which keeps all JSON documents in a single JSON type. I would be happy to re-run the benchmarks on the same machine which I am using to reproduce all measurements.

[shuiyisong]

Hi @rschu1ze , sorry for the inconvenience.
TL;DR, we'd like to leave it for now.

IIRC, there are other databases that 'extract the index field', so I thought it was OK.
In my personal thoughts, all databases creating indexes upon JSON paths are doing some kind of 'extracting'. It's just the level of visibility to the users. Also, some databases have materialized views as a built-in mechanism to speed up queries. Since the pipeline engine is a built-in mechanism of GreptimeDB, I thought we could use the pipeline along with GreptimeDB's index system to achieve a similar result.
The GreptimeDB has native support for the JSON datatype, and the whole structure is persisted and can be retrieved later. So it didn't strike me as 'flatten' first.

But anyway, after re-examining the blog post, I see the requirement for only containing the JSON commit object in the table. We're not particularly optimized for search and filter on JSON objects directly, and we plan to refactor the JSON subsystem in the future. So let's leave it for now.

Thanks again for your time, and sorry for the inconvenience.

[rschu1ze]

No worries. I agree ... let's keep the results for now and when you have a more optimized JSON implementation, feel free to make a new submission.

[KVeschgini]

Without the rule, the benchmark would essentially be equivalent to the 1 billion row benchmark. But in my opinion the point here is to test whether databases can handle real-world JSON as-is — including edge cases where a field is missing entirely vs. being explicitly set to null, or where types vary across rows. Flattening the structure defeats this purpose. It’s also about evaluating whether databases can natively work with JSON rather than requiring a wide relational schema.

These benchmarks are used by me to select the best database for specific use cases. If results don’t follow the rules, they become misleading and reduce the benchmark’s value for practical decision-making.

[rschu1ze]

@KVeschgini Just to double-check, are you proposing above^^ to remove Greptime entirely? Right now, the results are behind the Flattening = Yes button at https://jsonbench.com/. Asking because pipeline.yaml not only stores JSON fields as separate columns (that is the original sense of "flattening" and allowed by the rules), it also creates indexes on the extracted fields. That's arguably outside the original intention of the benchmark.