Refactor CapnProto format to improve input/output performance #49752

Avogar · 2023-05-10T18:43:06Z

Changelog category (leave one):

Performance Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Rewrite CapnProto input/output format to improve its performance. Map column names and CapnProto fields case insensitive, fix reading/writing of nested structure fields.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

robot-clickhouse · 2023-05-10T18:52:25Z

This is an automated comment for commit 24d70a2 with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🔴 failure

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	🟢 success
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟢 success
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	🟢 success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	🟢 success
Docker image for servers	The check to build and optionally push the mentioned image to docker hub	🟢 success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	🟢 success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integrational tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	🟢 success
Install packages	Checks that the built packages are installable in a clear environment	🟢 success
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	🟢 success
Mergeable Check	Checks if all other necessary checks are successful	🟢 success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	🟢 success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success
SQLancer	Fuzzing tests that detect logical bugs with SQLancer tool	🟢 success
Sqllogic	Run clickhouse on the sqllogic test set against sqlite and checks that all statements are passed	🟢 success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	🟢 success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	🔴 failure
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	🟢 success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	🟢 success
Unit tests	Runs the unit tests for different release types	🟢 success
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	🟢 success

Avogar · 2023-05-12T14:42:22Z

Benchmark for hits table:
Before this PR:

avogar-dev :) select * from test.hits limit 1000000 into outfile 'data.capnp' settings format_schema='hits_schema:Message'

SELECT *
FROM test.hits
LIMIT 1000000
INTO OUTFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: 3633d222-f294-484a-8c65-690fdfa3bb8e

1000000 rows in set. Elapsed: 50.638 sec. Processed 1.00 million rows, 937.17 MB (19.75 thousand rows/s., 18.51 MB/s.)

avogar-dev :) insert into test.hits from infile 'data.capnp' settings format_schema='hits_schema:Message'

INSERT INTO test.hits FROM INFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: f373b3c6-6f92-4efa-ba8a-bb01daa0183a

Ok.

1000000 rows in set. Elapsed: 36.840 sec. Processed 1.00 million rows, 937.16 MB (27.14 thousand rows/s., 25.44 MB/s.)

After this PR:

avogar-dev :) select * from test.hits limit 1000000 into outfile 'data.capnp' settings format_schema='hits_schema:Message'

SELECT *
FROM test.hits
LIMIT 1000000
INTO OUTFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: 1c17dfd9-9d65-4073-b0e9-951762ce4d3b


1000000 rows in set. Elapsed: 11.876 sec. Processed 1.00 million rows, 937.17 MB (84.20 thousand rows/s., 78.91 MB/s.)

avogar-dev :) insert into test.hits from infile 'data.capnp' settings format_schema='hits_schema:Message'

INSERT INTO test.hits FROM INFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: 23ff7c36-dd4a-4861-b7a2-2828e81cbc44

Ok.

1000000 rows in set. Elapsed: 11.951 sec. Processed 1.00 million rows, 937.16 MB (83.67 thousand rows/s., 78.41 MB/s.)

x5 improvement for output, x3 improvement for input. But still 10s is too long for 1000000 rows, but it's already the limitation of capnp library, seems like it's not well optimized for generic messages writing and reading (too many unnecessary copies per each message).

Avogar · 2023-05-12T14:43:47Z

src/Formats/CapnProtoSchema.cpp

@@ -0,0 +1,298 @@
+#include <Formats/CapnProtoSchema.h>


Code from this file is not new, it's copied from old file CapnProtoUtils.cpp. The new code is in CapnProtoSerializer.cpp

src/Formats/CapnProtoSerializer.cpp

Avogar · 2023-05-17T18:54:14Z

By updating capnp library (just add additional arguments for some methods to avoid copying, will test it more and make a patch) I can achive x1.5-x2 performance improvement. But the main problem with CapnProto input/output format is that we first serialize/deserialize message in memory and then read data from it. Perfect solution will be to read/write message directly from/into buffer, but it will require writing our own CapnProto message serializer/deserializer (as it's done for Protobuf format). But I am not ready for it now.

…o better-capnproto-3

Avogar · 2023-05-31T19:51:13Z

New results:

avogar-dev :) select * from test.hits limit 1000000 into outfile 'data.capnp' settings format_schema='hits_schema:Message'

SELECT *
FROM test.hits
LIMIT 1000000
INTO OUTFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: 1c17dfd9-9d65-4073-b0e9-951762ce4d3b


1000000 rows in set. Elapsed: 3.539 sec. Processed 1.00 million rows, 937.17 MB (287.00 thousand rows/s., 268.91 MB/s.)

avogar-dev :) insert into test.hits from infile 'data.capnp' settings format_schema='hits_schema:Message'

INSERT INTO test.hits FROM INFILE 'data.capnp'
SETTINGS format_schema = 'hits_schema:Message'

Query id: 23ff7c36-dd4a-4861-b7a2-2828e81cbc44

Ok.

1000000 rows in set. Elapsed: 4.006 sec. Processed 1.00 million rows, 889.45 MB (249.65 thousand rows/s., 222.06 MB/s.)

Total improvement now ~15x for output and ~10x for input.

@al13n321 I think it's ready for review

al13n321

Looks reasonable, nice results!

al13n321 · 2023-05-12T21:35:31Z

src/Core/Settings.h

@@ -959,7 +959,7 @@ class IColumn;
    M(Bool, output_format_orc_string_as_string, false, "Use ORC String type instead of Binary for String columns", 0) \
    M(ORCCompression, output_format_orc_compression_method, "lz4", "Compression method for ORC output format. Supported codecs: lz4, snappy, zlib, zstd, none (uncompressed)", 0) \
    \
-    M(EnumComparingMode, format_capn_proto_enum_comparising_mode, FormatSettings::EnumComparingMode::BY_VALUES, "How to map ClickHouse Enum and CapnProto Enum", 0) \
+    M(CapnProtoEnumComparingMode, format_capn_proto_enum_comparising_mode, FormatSettings::CapnProtoEnumComparingMode::BY_VALUES, "How to map ClickHouse Enum and CapnProto Enum", 0) \


format_capn_proto_enum_comparising_mode

Unfortunately, this setting exists for a long time and we cannot change it's name(

tests/queries/0_stateless/02735_capnp_case_insensitive_names_matcing.reference

src/Formats/CapnProtoSerializer.cpp

al13n321 · 2023-06-06T02:37:05Z

src/Formats/CapnProtoSerializer.cpp

+            if (enum_comparing_mode == FormatSettings::CapnProtoEnumComparingMode::BY_VALUES)
+                assert_cast<ColumnVector<EnumType> &>(column).insertValue(static_cast<EnumType>(capnp_enum_value));
+            else
+                assert_cast<ColumnVector<EnumType> &>(column).insertValue(capnp_to_ch_values[capnp_enum_value]);


al13n321 · 2023-06-06T02:41:56Z

src/Formats/CapnProtoSerializer.cpp

+        std::unordered_map<EnumType, UInt16> ch_to_capnp_values;
+        std::unordered_map<UInt16, EnumType> capnp_to_ch_values;


These could be std::vectors, for speed. Since all indices are at most 16 bits, it'll be at most 128 KB (+ maybe a 64 KB null mask to detect unexpected indices), so don't event need a fallback to hashtable for sparse enums.

Makes sense, I will try to change to vectors

I decided to leave it as map, because by using vectors we won't be able to determine if we have invalid enum value

src/Formats/CapnProtoSerializer.cpp

…-capnproto-3

src/Formats/CapnProtoSerializer.cpp

Refactor CapnProto format to improve input/output performance

ac7c54a

robot-clickhouse added the pr-performance Pull request with some performance improvements label May 10, 2023

Avogar and others added 6 commits May 10, 2023 21:07

Fix style

167516b

Fix style

344885d

Fix style

e3cb1e4

Merge branch 'master' into better-capnproto-3

680d359

Fix build

288988b

Fix special builds

1daa981

Avogar commented May 12, 2023

View reviewed changes

src/Formats/CapnProtoSerializer.cpp Outdated Show resolved Hide resolved

Avogar added 2 commits May 12, 2023 18:53

Fix special build

7698776

Fix special build

c9f90fb

al13n321 self-assigned this May 13, 2023

Merge branch 'master' into better-capnproto-3

b61b4a1

Avogar and others added 13 commits May 23, 2023 13:42

Merge branch 'master' into better-capnproto-3

d7f0b6c

Refactor CapnProto format to improve input/output performance

e66f627

Fix style

c2eada7

Fix style

cc7cfa0

Fix style

1347dc4

Fix build

a89a8b8

Fix special builds

5f1ca61

Fix special build

94ef089

Fix special build

f76fc5e

Merge branch 'better-capnproto-3' of github.com:Avogar/ClickHouse int…

c75843e

…o better-capnproto-3

Make better

ea395e9

Better

c962631

Merge branch 'master' into better-capnproto-3

898d1f3

robot-ch-test-poll1 added the submodule changed At least one submodule changed in this PR. label May 31, 2023

Avogar and others added 4 commits May 31, 2023 21:52

Clean up

4b46486

Clean up

4987c4b

Fix style

8b34a30

Fix tests

883350d

al13n321 approved these changes Jun 6, 2023

View reviewed changes

Avogar added 2 commits June 8, 2023 11:16

Merge branch 'master' of github.com:ClickHouse/ClickHouse into better…

cc03652

…-capnproto-3

Make better

47b0c2a

Avogar force-pushed the better-capnproto-3 branch from f39862c to 47b0c2a Compare June 9, 2023 13:02

al13n321 approved these changes Jun 10, 2023

View reviewed changes

src/Formats/CapnProtoSerializer.cpp Outdated Show resolved Hide resolved

Fix

24d70a2

Avogar merged commit 8fdcd91 into ClickHouse:master Jun 13, 2023
256 of 257 checks passed

baibaichen mentioned this pull request Jun 14, 2023

Clang 15 Build failed after #49752 #50963

Closed

Avogar mentioned this pull request Jun 14, 2023

Change submodule capnproto to it's fork in ClickHouse #50987

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor CapnProto format to improve input/output performance #49752

Refactor CapnProto format to improve input/output performance #49752

Avogar commented May 10, 2023 •

edited

robot-clickhouse commented May 10, 2023 •

edited by robot-ch-test-poll

Avogar commented May 12, 2023

Avogar May 12, 2023 •

edited

Avogar commented May 17, 2023 •

edited

Avogar commented May 31, 2023 •

edited

al13n321 left a comment

al13n321 May 12, 2023

Avogar Jun 8, 2023

al13n321 Jun 6, 2023

al13n321 Jun 6, 2023

Avogar Jun 8, 2023

Avogar Jun 9, 2023

		std::unordered_map<EnumType, UInt16> ch_to_capnp_values;
		std::unordered_map<UInt16, EnumType> capnp_to_ch_values;

Refactor CapnProto format to improve input/output performance #49752

Refactor CapnProto format to improve input/output performance #49752

Conversation

Avogar commented May 10, 2023 • edited

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

robot-clickhouse commented May 10, 2023 • edited by robot-ch-test-poll

Avogar commented May 12, 2023

Avogar May 12, 2023 • edited

Choose a reason for hiding this comment

Avogar commented May 17, 2023 • edited

Avogar commented May 31, 2023 • edited

al13n321 left a comment

Choose a reason for hiding this comment

al13n321 May 12, 2023

Choose a reason for hiding this comment

Avogar Jun 8, 2023

Choose a reason for hiding this comment

al13n321 Jun 6, 2023

Choose a reason for hiding this comment

al13n321 Jun 6, 2023

Choose a reason for hiding this comment

Avogar Jun 8, 2023

Choose a reason for hiding this comment

Avogar Jun 9, 2023

Choose a reason for hiding this comment

Avogar commented May 10, 2023 •

edited

robot-clickhouse commented May 10, 2023 •

edited by robot-ch-test-poll

Avogar May 12, 2023 •

edited

Avogar commented May 17, 2023 •

edited

Avogar commented May 31, 2023 •

edited