fix(eap-spans): Reduce number of buckets we use for attributes#6148
Merged
Conversation
|
This PR has a migration; here is the generated SQL -- start migrations
-- forward migration events_analytics_platform : 0001_spans
Local op: CREATE TABLE IF NOT EXISTS eap_spans_local (organization_id UInt64, project_id UInt64, service String CODEC (ZSTD(1)), trace_id UUID, span_id UInt64, parent_span_id UInt64 CODEC (ZSTD(1)), segment_id UInt64 CODEC (ZSTD(1)), segment_name String CODEC (ZSTD(1)), is_segment UInt8 CODEC (T64, ZSTD(1)), _sort_timestamp DateTime CODEC (DoubleDelta, ZSTD(1)), start_timestamp DateTime64(6) CODEC (DoubleDelta, ZSTD(1)), end_timestamp DateTime64(6) CODEC (DoubleDelta, ZSTD(1)), duration_ms UInt32 CODEC (DoubleDelta, ZSTD(1)), exclusive_time_ms Float64 CODEC (ZSTD(1)), retention_days UInt16 CODEC (DoubleDelta, ZSTD(1)), name String CODEC (ZSTD(1)), sampling_factor Float64 CODEC (ZSTD(1)), sampling_weight Float64 CODEC (ZSTD(1)), sign Int8 CODEC (DoubleDelta(1)), attr_str_0 Map(String, String) CODEC (ZSTD(1)), attr_str_1 Map(String, String) CODEC (ZSTD(1)), attr_str_2 Map(String, String) CODEC (ZSTD(1)), attr_str_3 Map(String, String) CODEC (ZSTD(1)), attr_str_4 Map(String, String) CODEC (ZSTD(1)), attr_str_5 Map(String, String) CODEC (ZSTD(1)), attr_str_6 Map(String, String) CODEC (ZSTD(1)), attr_str_7 Map(String, String) CODEC (ZSTD(1)), attr_str_8 Map(String, String) CODEC (ZSTD(1)), attr_str_9 Map(String, String) CODEC (ZSTD(1)), attr_str_10 Map(String, String) CODEC (ZSTD(1)), attr_str_11 Map(String, String) CODEC (ZSTD(1)), attr_str_12 Map(String, String) CODEC (ZSTD(1)), attr_str_13 Map(String, String) CODEC (ZSTD(1)), attr_str_14 Map(String, String) CODEC (ZSTD(1)), attr_str_15 Map(String, String) CODEC (ZSTD(1)), attr_str_16 Map(String, String) CODEC (ZSTD(1)), attr_str_17 Map(String, String) CODEC (ZSTD(1)), attr_str_18 Map(String, String) CODEC (ZSTD(1)), attr_str_19 Map(String, String) CODEC (ZSTD(1)), attr_num_0 Map(String, Float64) CODEC (ZSTD(1)), attr_num_1 Map(String, Float64) CODEC (ZSTD(1)), attr_num_2 Map(String, Float64) CODEC (ZSTD(1)), attr_num_3 Map(String, Float64) CODEC (ZSTD(1)), attr_num_4 Map(String, Float64) CODEC (ZSTD(1)), attr_num_5 Map(String, Float64) CODEC (ZSTD(1)), attr_num_6 Map(String, Float64) CODEC (ZSTD(1)), attr_num_7 Map(String, Float64) CODEC (ZSTD(1)), attr_num_8 Map(String, Float64) CODEC (ZSTD(1)), attr_num_9 Map(String, Float64) CODEC (ZSTD(1)), attr_num_10 Map(String, Float64) CODEC (ZSTD(1)), attr_num_11 Map(String, Float64) CODEC (ZSTD(1)), attr_num_12 Map(String, Float64) CODEC (ZSTD(1)), attr_num_13 Map(String, Float64) CODEC (ZSTD(1)), attr_num_14 Map(String, Float64) CODEC (ZSTD(1)), attr_num_15 Map(String, Float64) CODEC (ZSTD(1)), attr_num_16 Map(String, Float64) CODEC (ZSTD(1)), attr_num_17 Map(String, Float64) CODEC (ZSTD(1)), attr_num_18 Map(String, Float64) CODEC (ZSTD(1)), attr_num_19 Map(String, Float64) CODEC (ZSTD(1))) ENGINE ReplicatedCollapsingMergeTree('/clickhouse/tables/events_analytics_platform/{shard}/default/eap_spans_local', '{replica}', sign) PRIMARY KEY (organization_id, _sort_timestamp, trace_id) ORDER BY (organization_id, _sort_timestamp, trace_id, span_id) PARTITION BY (toMonday(_sort_timestamp)) TTL _sort_timestamp + toIntervalDay(retention_days) SETTINGS index_granularity=8192;
Distributed op: CREATE TABLE IF NOT EXISTS eap_spans_dist (organization_id UInt64, project_id UInt64, service String CODEC (ZSTD(1)), trace_id UUID, span_id UInt64, parent_span_id UInt64 CODEC (ZSTD(1)), segment_id UInt64 CODEC (ZSTD(1)), segment_name String CODEC (ZSTD(1)), is_segment UInt8 CODEC (T64, ZSTD(1)), _sort_timestamp DateTime CODEC (DoubleDelta, ZSTD(1)), start_timestamp DateTime64(6) CODEC (DoubleDelta, ZSTD(1)), end_timestamp DateTime64(6) CODEC (DoubleDelta, ZSTD(1)), duration_ms UInt32 CODEC (DoubleDelta, ZSTD(1)), exclusive_time_ms Float64 CODEC (ZSTD(1)), retention_days UInt16 CODEC (DoubleDelta, ZSTD(1)), name String CODEC (ZSTD(1)), sampling_factor Float64 CODEC (ZSTD(1)), sampling_weight Float64 CODEC (ZSTD(1)), sign Int8 CODEC (DoubleDelta(1)), attr_str_0 Map(String, String) CODEC (ZSTD(1)), attr_str_1 Map(String, String) CODEC (ZSTD(1)), attr_str_2 Map(String, String) CODEC (ZSTD(1)), attr_str_3 Map(String, String) CODEC (ZSTD(1)), attr_str_4 Map(String, String) CODEC (ZSTD(1)), attr_str_5 Map(String, String) CODEC (ZSTD(1)), attr_str_6 Map(String, String) CODEC (ZSTD(1)), attr_str_7 Map(String, String) CODEC (ZSTD(1)), attr_str_8 Map(String, String) CODEC (ZSTD(1)), attr_str_9 Map(String, String) CODEC (ZSTD(1)), attr_str_10 Map(String, String) CODEC (ZSTD(1)), attr_str_11 Map(String, String) CODEC (ZSTD(1)), attr_str_12 Map(String, String) CODEC (ZSTD(1)), attr_str_13 Map(String, String) CODEC (ZSTD(1)), attr_str_14 Map(String, String) CODEC (ZSTD(1)), attr_str_15 Map(String, String) CODEC (ZSTD(1)), attr_str_16 Map(String, String) CODEC (ZSTD(1)), attr_str_17 Map(String, String) CODEC (ZSTD(1)), attr_str_18 Map(String, String) CODEC (ZSTD(1)), attr_str_19 Map(String, String) CODEC (ZSTD(1)), attr_num_0 Map(String, Float64) CODEC (ZSTD(1)), attr_num_1 Map(String, Float64) CODEC (ZSTD(1)), attr_num_2 Map(String, Float64) CODEC (ZSTD(1)), attr_num_3 Map(String, Float64) CODEC (ZSTD(1)), attr_num_4 Map(String, Float64) CODEC (ZSTD(1)), attr_num_5 Map(String, Float64) CODEC (ZSTD(1)), attr_num_6 Map(String, Float64) CODEC (ZSTD(1)), attr_num_7 Map(String, Float64) CODEC (ZSTD(1)), attr_num_8 Map(String, Float64) CODEC (ZSTD(1)), attr_num_9 Map(String, Float64) CODEC (ZSTD(1)), attr_num_10 Map(String, Float64) CODEC (ZSTD(1)), attr_num_11 Map(String, Float64) CODEC (ZSTD(1)), attr_num_12 Map(String, Float64) CODEC (ZSTD(1)), attr_num_13 Map(String, Float64) CODEC (ZSTD(1)), attr_num_14 Map(String, Float64) CODEC (ZSTD(1)), attr_num_15 Map(String, Float64) CODEC (ZSTD(1)), attr_num_16 Map(String, Float64) CODEC (ZSTD(1)), attr_num_17 Map(String, Float64) CODEC (ZSTD(1)), attr_num_18 Map(String, Float64) CODEC (ZSTD(1)), attr_num_19 Map(String, Float64) CODEC (ZSTD(1))) ENGINE Distributed(`cluster_one_sh`, default, eap_spans_local, reinterpretAsUInt128(trace_id));
Local op: ALTER TABLE eap_spans_local ADD INDEX IF NOT EXISTS bf_trace_id trace_id TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_0 mapKeys(attr_str_0) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_1 mapKeys(attr_str_1) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_2 mapKeys(attr_str_2) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_3 mapKeys(attr_str_3) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_4 mapKeys(attr_str_4) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_5 mapKeys(attr_str_5) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_6 mapKeys(attr_str_6) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_7 mapKeys(attr_str_7) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_8 mapKeys(attr_str_8) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_9 mapKeys(attr_str_9) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_10 mapKeys(attr_str_10) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_11 mapKeys(attr_str_11) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_12 mapKeys(attr_str_12) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_13 mapKeys(attr_str_13) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_14 mapKeys(attr_str_14) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_15 mapKeys(attr_str_15) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_16 mapKeys(attr_str_16) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_17 mapKeys(attr_str_17) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_18 mapKeys(attr_str_18) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_19 mapKeys(attr_str_19) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_0 mapValues(attr_str_0) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_1 mapValues(attr_str_1) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_2 mapValues(attr_str_2) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_3 mapValues(attr_str_3) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_4 mapValues(attr_str_4) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_5 mapValues(attr_str_5) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_6 mapValues(attr_str_6) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_7 mapValues(attr_str_7) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_8 mapValues(attr_str_8) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_9 mapValues(attr_str_9) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_10 mapValues(attr_str_10) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_11 mapValues(attr_str_11) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_12 mapValues(attr_str_12) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_13 mapValues(attr_str_13) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_14 mapValues(attr_str_14) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_15 mapValues(attr_str_15) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_16 mapValues(attr_str_16) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_17 mapValues(attr_str_17) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_18 mapValues(attr_str_18) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_str_val_19 mapValues(attr_str_19) TYPE ngrambf_v1(4, 1024, 10, 1) GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_0 mapKeys(attr_num_0) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_1 mapKeys(attr_num_1) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_2 mapKeys(attr_num_2) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_3 mapKeys(attr_num_3) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_4 mapKeys(attr_num_4) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_5 mapKeys(attr_num_5) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_6 mapKeys(attr_num_6) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_7 mapKeys(attr_num_7) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_8 mapKeys(attr_num_8) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_9 mapKeys(attr_num_9) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_10 mapKeys(attr_num_10) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_11 mapKeys(attr_num_11) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_12 mapKeys(attr_num_12) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_13 mapKeys(attr_num_13) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_14 mapKeys(attr_num_14) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_15 mapKeys(attr_num_15) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_16 mapKeys(attr_num_16) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_17 mapKeys(attr_num_17) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_18 mapKeys(attr_num_18) TYPE bloom_filter GRANULARITY 1, ADD INDEX IF NOT EXISTS bf_attr_num_19 mapKeys(attr_num_19) TYPE bloom_filter GRANULARITY 1;
-- end forward migration events_analytics_platform : 0001_spans
-- backward migration events_analytics_platform : 0001_spans
Local op: DROP TABLE IF EXISTS eap_spans_local;
Distributed op: DROP TABLE IF EXISTS eap_spans_dist;
-- end backward migration events_analytics_platform : 0001_spans |
|
Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time. ❌ Failed Test Results:Completed 1215 tests with View the full list of failed testspytest
|
evanh
approved these changes
Jul 26, 2024
john-z-yang
reviewed
Jul 26, 2024
Comment on lines
+452
to
+453
| if storage_key == "events_analytics_platform": | ||
| print(schema) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
After some insertion tests, it turns out we're using too much memory to insert with 200 buckets, so reducing this now.