Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign up*: Support both Prost and rust-protobuf libraries #5379
Conversation
| @@ -1,5 +1,8 @@ | |||
| // Copyright 2019 TiKV Project Authors. Licensed under Apache-2.0. | |||
|
|
|||
| #[cfg(feature = "prost-codec")] | |||
| use prost; | |||
| use protobuf::ProtobufError; | |||
This comment has been minimized.
This comment has been minimized.
siddontang
Sep 2, 2019
Contributor
em, do we need to use protobuf feature here?
seem if we pass prost-codec feature, we still need to import protobuf?
This comment has been minimized.
This comment has been minimized.
nrc
Sep 2, 2019
Author
Contributor
I use the protobuf::Message trait (and supporting error types) to support both Prost and rust-proto, so we still need the protobuf imports in the Prost case.
This comment has been minimized.
This comment has been minimized.
| failpoints = ["fail/failpoints"] | ||
| prost-codec = [ |
This comment has been minimized.
This comment has been minimized.
siddontang
Sep 3, 2019
Contributor
em, I see nearly all the workspaces use these, seem too verbose, but I also find no way to simply them.
This comment has been minimized.
This comment has been minimized.
BusyJay
Oct 16, 2019
Contributor
Since features are unions, enabling one should be sufficient for all dependencies.
| @@ -96,8 +89,8 @@ zipf = "5.0.1" | |||
| bitflags = "1.0.1" | |||
| fail = "0.3" | |||
| uuid = { version = "0.6", features = [ "serde", "v4" ] } | |||
| grpcio = { version = "0.5.0-alpha.3", features = [ "openssl-vendored" ] } | |||
| raft = "0.6.0-alpha" | |||
| grpcio = { version = "0.5.0-alpha.3", features = [ "openssl-vendored" ], default-features = false } | |||
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
nrc
Sep 3, 2019
Author
Contributor
Because when we use grpc-rs etc., we must use either prost-codec or protobuf-codec (which is the default), but not both.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
How to bench this PR with building TiKV "PROST=1"? |
This comment has been minimized.
This comment has been minimized.
|
PTAL @breeswish @overvenus |
This comment has been minimized.
This comment has been minimized.
|
Hi @nrc, can we use |
This comment has been minimized.
This comment has been minimized.
|
/release PROST=1 make dist_release |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
/bench |
This comment has been minimized.
This comment has been minimized.
|
@nrc Oops, benchmark failed to start a TiDB cluster using this commit, there were some problem. |
This comment has been minimized.
This comment has been minimized.
|
friendly ping @nrc |
This comment has been minimized.
This comment has been minimized.
|
/release PROST=1 make dist_release |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
/bench |
This comment has been minimized.
This comment has been minimized.
|
CI failed with PTAL @breeswish |
This comment has been minimized.
This comment has been minimized.
|
/bench |
This comment has been minimized.
This comment has been minimized.
|
/release PROST=1 make dist_release |
This comment has been minimized.
This comment has been minimized.
|
/test |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
@@ Benchmark Diff @@
================================================================================
tidb: 0f2434660c509ec0b493d8c7ee0bbbf8e8549184
--- tikv: ff82aa9eba331585aec1c6cdf9e1584512bccb34
+++ tikv: b294b71f9a4afd4ab1c47c2c452048d511954224
pd: c7c572a7dc9710aca57cb0bddf4d9bca6c4a111b
================================================================================
test-1: < oltp_insert >
* QPS : 20840.65 ± 0.5553% (std=70.69) delta: -1.91%
* AvgMs : 12.28 ± 0.5376% (std=0.04) delta: 1.93%
* PercentileMs99 : 42.92 ± 1.0903% (std=0.38) delta: 0.73%
test-2: < oltp_update_non_index >
* QPS : 29243.96 ± 0.1096% (std=20.95) delta: -0.69%
* AvgMs : 8.75 ± 0.1143% (std=0.01) delta: 0.71%
* PercentileMs99 : 30.70 ± 2.1758% (std=0.41) delta: -0.35%
test-3: < oltp_read_write >
* QPS : 36624.11 ± 0.2892% (std=63.44) delta: -0.78%
* AvgMs : 140.37 ± 0.2821% (std=0.25) delta: 0.79%
* PercentileMs99 : 262.64 ± 0.0000% (std=0.00) delta: 1.82%
test-4: < oltp_point_select >
* QPS : 73650.82 ± 0.2890% (std=134.00) delta: -0.62%
* AvgMs : 3.47 ± 1.0387% (std=0.02) delta: 0.35%
* PercentileMs99 : 7.48 ± 1.0425% (std=0.06) delta: 0.00%
test-5: < oltp_update_index >
* QPS : 16519.31 ± 0.1599% (std=17.75) delta: -2.55%
* AvgMs : 15.50 ± 0.1613% (std=0.02) delta: 2.62%
* PercentileMs99 : 49.21 ± 0.0000% (std=0.00) delta: 1.80%
|
|
This PR frequently facing conflicts. Approve it so that it can be merged sooner. |
To use Prost, set the `PROST` env var, e.g.,: `PROST=1 make dev`. If using Cargo, use `--no-default-features --features prost-codec`. The most notable change is threading the prost-codec/protobuf-codec through the Cargo.tomls of all crates. In addition, in order to make this work I had to move integraton tests and benchmarks into their own crate (`tests`). This is because Cargo features do not interact perfectly with dev-dependencies. We're using a Git dep for Prost in order to get some optimisations which are on master, but not in the latest release. We can change to a crates.io dep when there is another release. We must allow the `identity_conversion` lint because there are some conversions which are meaningful with rust-protobuf, but no-ops with Prost. The changes to src/coprocessor/endpoint.rs are because Prost does not permit setting a custom recursion limit. We only did this for tests previously. We now use the default recursion limit all the time for both codecs; the test must be adjusted so that we hit the higher limit. Signed-off-by: Nick Cameron <nrc@ncameron.org>
|
|
This comment has been minimized.
This comment has been minimized.
|
/merge |
This comment has been minimized.
This comment has been minimized.
|
/run-all-tests |
This comment has been minimized.
This comment has been minimized.
|
@nrc merge failed. |
This comment has been minimized.
This comment has been minimized.
|
/merge |
This comment has been minimized.
This comment has been minimized.
|
Your auto merge job has been accepted, waiting for 5964 |
This comment has been minimized.
This comment has been minimized.
|
/run-all-tests |
This comment has been minimized.
This comment has been minimized.
|
@nrc merge failed. |
This comment has been minimized.
This comment has been minimized.
|
/merge |
This comment has been minimized.
This comment has been minimized.
|
/run-all-tests |
Signed-off-by: Nick Cameron <nrc@ncameron.org>
Signed-off-by: Nick Cameron <nrc@ncameron.org>

nrc commentedSep 2, 2019
To use Prost, set the
PROSTenv var, e.g.,:PROST=1 make dev. If using Cargo,use
--no-default-features --features prost-codec.The most notable change is threading the prost-codec/protobuf-codec through the
Cargo.tomls of all crates. In addition, in order to make this work I had to move
integraton tests and benchmarks into their own crate (
tests). This is becauseCargo features do not interact perfectly with dev-dependencies.
We're using a Git dep for Prost in order to get some optimisations which are on
master, but not in the latest release. We can change to a crates.io dep when there
is another release.
We must allow the
identity_conversionlint because there are some conversionswhich are meaningful with rust-protobuf, but no-ops with Prost.
The changes to src/coprocessor/endpoint.rs are because Prost does not permit
setting a custom recursion limit. We only did this for tests previously. We
now use the default recursion limit all the time for both codecs; the test must
be adjusted so that we hit the higher limit.
PTAL @breeswish @BusyJay @overvenus
What have you changed?
Add a feature flag for Prost which builds TiKV and its deps using Prost rather than rust-protobuf as the protobuf codec.
What is the type of the changes?
How is the PR tested?
make devDoes this PR affect documentation (docs) or should it be mentioned in the release notes?
Should have dev docs (to come) and mentioned in release docs.
Does this PR affect
tidb-ansible?No
Refer to a related PR or issue link (optional)
#2452