Skip to content

Record which systems retain the document structure and which don't #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rschu1ze merged 2 commits into from
Mar 11, 2025
Merged

Record which systems retain the document structure and which don't #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9df4ec
Record which systems retain the document structure and which don't
rschu1ze Mar 11, 2025
bd78570
Mark flattening databases
rschu1ze Mar 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ You can easily reproduce every test (although for some systems it may take from

Best efforts should be taken to understand the details of every tested system for a fair comparison. It is allowed to apply various [indexing methods](https://clickhouse.com/blog/json-bench-clickhouse-vs-mongodb-elasticsearch-duckdb-postgresql#some-json-paths-can-be-used-for-indexes-and-data-sorting) whenever appropriate.

It is [not allowed](https://clickhouse.com/blog/json-bench-clickhouse-vs-mongodb-elasticsearch-duckdb-postgresql#no-query-results-cache) to use query results caching or flatten JSON at the insertion time.
It is [not allowed](https://clickhouse.com/blog/json-bench-clickhouse-vs-mongodb-elasticsearch-duckdb-postgresql#no-query-results-cache) to use query results caching or flatten JSON into multiple non-JSON colums at insertion type.

Some databases do have a JSON data type but they flatten nested JSON documents at insertion time to a single level (typically using `.` as separator between levels). We consider this a grey zone. On the one hand, this removes the possibility to restore the original documents, on the other hand, flattening may in many practical situations be acceptable. The dashboard allows to filter out databases which do not retain the document structure (i.e. which flatten).

## Goals

Expand Down Expand Up @@ -85,7 +87,7 @@ For example, for clickhouse:

```
cd clickhouse
./main.sh
./main.sh

Select the dataset size to benchmark:
1) 1m (default)
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_1000m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_1000m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_100m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_100m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_10m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_10m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_1m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
1 change: 1 addition & 0 deletions clickhouse/results/m6i.8xlarge_bluesky_1m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
3 changes: 2 additions & 1 deletion duckdb/results/m6i.8xlarge_bluesky_1000m.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand All @@ -28,4 +29,4 @@
"1 hr 2 min 26.50 sec, 1 hr 2 min 15.85 sec, 1 hr 2 min 19.26 sec",
"1 hr 2 min 28.53 sec, 1 hr 2 min 16.93 sec, 1 hr 2 min 20.10 sec"
]
}
}
3 changes: 2 additions & 1 deletion duckdb/results/m6i.8xlarge_bluesky_100m.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand All @@ -28,4 +29,4 @@
"6 min 13.36 sec, 7.55 sec, 7.53 sec",
"6 min 13.56 sec, 8.07 sec, 8.06 sec"
]
}
}
3 changes: 2 additions & 1 deletion duckdb/results/m6i.8xlarge_bluesky_10m.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand All @@ -28,4 +29,4 @@
"37.36 sec, 1.06 sec, 1.06 sec",
"37.44 sec, 1.18 sec, 1.21 sec"
]
}
}
3 changes: 2 additions & 1 deletion duckdb/results/m6i.8xlarge_bluesky_1m.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand All @@ -28,4 +29,4 @@
"2.80 sec, 250.00 msec, 247.00 msec",
"2.82 sec, 259.00 msec, 256.00 msec"
]
}
}
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_1000m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_1000m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_100m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_100m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_10m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_10m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_1m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 1000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_no_source_1m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "no",
"tags": [
],
"dataset_size": 1000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_1000m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_1000m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_100m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_100m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_10m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_10m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions elasticsearch/results/m6i.8xlarge_bluesky_source_1m_best_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
3 changes: 2 additions & 1 deletion elasticsearch/results/m6i.8xlarge_bluesky_source_1m_default_compression.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 16000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand All @@ -28,4 +29,4 @@
"48.00 msec, 50.00 msec, 49.00 msec",
"57.00 msec, 59.00 msec, 58.00 msec"
]
}
}
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_1000m_snappy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_1000m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_100m_snappy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_100m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_10m_snappy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_10m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_1m_snappy.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
1 change: 1 addition & 0 deletions mongodb/results/m6i.8xlarge_bluesky_1m_zstd.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_1000m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_1000m_pglz.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_100m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_100m_pglz.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 100000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_10m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_10m_pglz.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 10000000,
Expand Down
1 change: 1 addition & 0 deletions postgresql/results/m6i.8xlarge_bluesky_1m_lz4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
"machine": "m6i.8xlarge, 10000gib gp3",
"cluster_size": 1,
"comment": "",
"retains_structure": "yes",
"tags": [
],
"dataset_size": 1000000,
Expand Down
Loading