Add MySQL client protocol (text protocol, transactions, prepared statements) on clean-room auth (#2093)#3330
Conversation
Clean-room implementation of the MySQL connection-phase authentication handshake, derived from the public MySQL protocol documentation with no GPL lineage: mysql_native_password and caching_sha2_password scrambles, HandshakeV10/ HandshakeResponse41 codec, and length-encoded integer/string plus packet-header wire helpers. Handles the lenenc NULL (0xFB) marker and rejects an oversize auth_response.
…statements Port the MySQL protocol client (issue apache#2093) onto the clean-room auth codec and protobuf 3.21 (NonreflectableMessage): COM_QUERY text protocol, interactive transactions via connection affinity, and prepared statements. Wire caching_sha2_password (fast-auth, full-auth RSA, and secure-transport cleartext) into the live client. Fix the lenenc 9-byte length marker in pack_encode_length (0xFD -> 0xFE) per the MySQL protocol spec.
… Controller cleanup - Add clean-room integration tests (transactions, prepared statements, pooled connection concurrency, connection-type) run against a self-spawned mysqld. - Fix: a failed COM_STMT_PREPARE now returns the ERR packet to the caller and keeps the connection alive, instead of closing the socket. - Warn when a prepared statement runs on a 'short' connection (re-prepares on every execute; prefer 'pooled'). - Replace Controller's mysql-specific _mysql_stmt with a generic opaque per-RPC slot so no protocol type leaks onto the shared Controller. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
b598971 to
a292dbc
Compare
…ted images Move all MySQL sources (mysql.*, mysql_command/reply/common/transaction/ statement*, mysql_protocol.*, mysql_authenticator.*) into src/brpc/policy/mysql/ alongside the clean-room auth codec; update all includes, build globs, and install rules. Remove three benchmark images inherited from the apache#2093 port and the doc section referencing them. No behavior change: full build green; all 19 mysql unit/integration tests pass; a 30-case standalone end-to-end run was independently verified against mysqld. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
d3989c8 to
be457a0
Compare
… cases) + ASF headers - Binary DATETIME/TIME: gate the microsecond bytes on the packet length, not the column's declared decimals (over-read / result-set desync). - COM_STMT_SEND_LONG_DATA: frame stmt_id/param_id inside the packet; fix chunk offset. - COM_STMT_EXECUTE: emit the trailing 0-length packet for 16MiB-aligned payloads. - OK/EOF status & warnings: decode via mysql_uint2korr (big-endian safe). - Row NULL-bitmap: arena-allocate instead of a stack VLA; cap column_count. - Auth: bounds-check the parsed auth string; size-bound StringPiece uses. - Prepared stmt: prune stale per-socket stmt_id map entries; count only real '?' placeholders (skip quotes/comments). - MysqlResponse::Clear and MysqlRequest copy/Swap: reset/copy all members. - Controller::ResetPods: release _bind_sock on controller reuse. - Standardize mysql file license headers to the ASF form; clarify auth comment. All 19 unit/integration tests still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
be457a0 to
0bb55ee
Compare
|
@chenBright @wwbmmm — marking this ready for review. Full MySQL client protocol for brpc — COM_QUERY text protocol, interactive transactions, and prepared statements — built on a clean-room authentication codec (no GPL lineage), addressing #2093.
The clean-room auth codec is also up as a smaller, focused PR (#3310, Stage 1a) if you'd prefer to review that foundation first. |
There was a problem hiding this comment.
Pull request overview
This PR adds a first-class MySQL client protocol implementation to brpc, including clean-room authentication, text protocol querying, transaction connection-affinity, and prepared statements support, along with new integration/unit tests and examples.
Changes:
- Introduce
PROTOCOL_MYSQLand register a MySQL client protocol stack (pack/parse/process) with connection-phase authentication handling. - Add MySQL client APIs (
MysqlRequest,MysqlResponse, prepared statements, transactions) and supporting wire/protocol utilities. - Add MySQL-focused tests, docs, and examples; extend build/test integration (CMake/Bazel/Makefile).
Reviewed changes
Copilot reviewed 50 out of 50 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/mysql/README.md | Documents auth-handshake integration test modes and setup. |
| test/mysql/brpc_mysql_connection_type_unittest.cpp | Adds integration test for SHORT connection behavior with prepared statements and plain queries. |
| test/mysql/brpc_mysql_auth_packet_unittest.cpp | Adds unit tests for MySQL auth packet wire helpers (lenenc ints/strings, headers). |
| test/CMakeLists.txt | Ensures MySQL unit tests are included in the CMake test glob. |
| test/BUILD.bazel | Adds a Bazel cc_test target covering MySQL unit tests. |
| src/brpc/socket.h | Exposes fd_version() for ABA-guarding per-connection statement caching. |
| src/brpc/socket.cpp | Increments fd version on fd reset; allows empty writes for auth-handshake flows. |
| src/brpc/policy/mysql/mysql.h | Introduces MySQL request/response message types and statement stub plumbing. |
| src/brpc/policy/mysql/mysql.cpp | Implements MysqlRequest/MysqlResponse serialization and parsing entrypoints. |
| src/brpc/policy/mysql/mysql_transaction.h | Adds MySQL transaction API and options for connection-affinity usage. |
| src/brpc/policy/mysql/mysql_transaction.cpp | Implements transaction start/commit/rollback with socket reserve/use semantics. |
| src/brpc/policy/mysql/mysql_statement.h | Adds prepared statement API and statement-id caching interface. |
| src/brpc/policy/mysql/mysql_statement.cpp | Implements stmt-id cache keyed by socket id + fd version; placeholder counting. |
| src/brpc/policy/mysql/mysql_statement_inl.h | Provides DBD map helpers for stmt-id caching and related gflag. |
| src/brpc/policy/mysql/mysql_protocol.h | Declares MySQL protocol functions (parse/process/serialize/pack). |
| src/brpc/policy/mysql/mysql_protocol.cpp | Implements client-side MySQL protocol codec, including auth handshake and prepare/execute flows. |
| src/brpc/policy/mysql/mysql_common.h | Adds MySQL constants/types and collation mapping used by the client implementation. |
| src/brpc/policy/mysql/mysql_common.cpp | Defines common MySQL defaults and type-to-string helper. |
| src/brpc/policy/mysql/mysql_command.h | Declares MySQL command packet builders for query/prepare/execute/long-data. |
| src/brpc/policy/mysql/mysql_command.cpp | Implements MySQL packetization and prepared statement wire builders. |
| src/brpc/policy/mysql/mysql_authenticator.h | Adds MysqlAuthenticator carrying user/password/schema/params/collation. |
| src/brpc/policy/mysql/mysql_authenticator.cpp | Implements credential serialization and clean-room scrambles (native + caching_sha2 fast path). |
| src/brpc/policy/mysql/mysql_auth_scramble.h | Declares clean-room auth scramble and RSA/cleartext helpers. |
| src/brpc/policy/mysql/mysql_auth_scramble.cpp | Implements native and caching_sha2 scrambles + RSA-OAEP encryption with OpenSSL EVP. |
| src/brpc/policy/mysql/mysql_auth_packet.h | Adds wire helpers for lenenc ints/strings, packet headers, NUL strings. |
| src/brpc/policy/mysql/mysql_auth_packet.cpp | Implements the MySQL wire-format helpers used by handshake parsing/building. |
| src/brpc/policy/mysql/mysql_auth_handshake.h | Declares clean-room handshake packet codecs (greeting/response/switch/more-data). |
| src/brpc/policy/mysql/mysql_auth_handshake.cpp | Implements parsing/building for connection-phase handshake payloads. |
| src/brpc/options.proto | Adds PROTOCOL_MYSQL enum value. |
| src/brpc/global.cpp | Registers the MySQL protocol with the global protocol registry. |
| src/brpc/details/controller_private_accessor.h | Adds private controller hooks for socket reserve/use and opaque session data. |
| src/brpc/controller.h | Introduces bind-socket actions/state to support MySQL transaction affinity and session data. |
| src/brpc/controller.cpp | Implements socket reserve/use behavior across RPCs for pooled/short connection types. |
| Makefile | Includes src/brpc/policy/mysql in the build directories list. |
| example/mysql_c++/mysqlclient_press.cpp | Adds libmysqlclient-based press tool (comparison baseline). |
| example/mysql_c++/mysql_tx.cpp | Adds brpc MySQL transaction example. |
| example/mysql_c++/mysql_stmt.cpp | Adds brpc MySQL prepared statement example. |
| example/mysql_c++/mysql_press.cpp | Adds brpc MySQL press example for CRUD. |
| example/mysql_c++/mysql_go_press.go | Adds Go MySQL press example (comparison baseline). |
| example/mysql_c++/mysql_cli.cpp | Adds brpc-based interactive MySQL CLI example. |
| example/mysql_c++/CMakeLists.txt | Adds CMake build for MySQL examples and dependency discovery. |
| docs/cn/mysql_client.md | Adds/updates Chinese documentation describing brpc MySQL client usage. |
Comments suppressed due to low confidence (1)
src/brpc/policy/mysql/mysql_authenticator.h:92
- The trailing
#endifcomment refers to COUCHBASE, which doesn’t match this header guard and is confusing when grepping/include-debugging. Update it toBRPC_POLICY_MYSQL_AUTHENTICATOR_H.
} // namespace policy
} // namespace brpc
#endif // BRPC_POLICY_COUCHBASE_AUTHENTICATOR_H
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| extern const char* MysqlDefaultCollation; | ||
| extern const char* MysqlBinaryCollation; | ||
| const std::map<std::string, uint8_t> MysqlCollations = { | ||
| {"big5_chinese_ci", 1}, | ||
| {"latin2_czech_cs", 2}, |
| size_t MysqlResponse::ByteSizeLong() const { | ||
| return _cached_size_; | ||
| } | ||
|
|
||
| void MysqlResponse::MergeFrom(const MysqlResponse& from) { | ||
| CHECK_NE(&from, this); | ||
| } |
| // * PreparedStatementUnderShortMustError (PRIMARY): | ||
| // build a SHORT channel, prepare "SELECT ? AS v", bind one INT param, | ||
| // CallMethod. Must ERROR (cntl.Failed() OR reply(0).is_error()); must | ||
| // NOT return a correct result set; must NOT crash. Looped a few times. | ||
| // | ||
| // * PlainQueryUnderShortMustSucceed (POSITIVE CONTROL): | ||
| // same SHORT channel; a stateless COM_QUERY "SELECT 7 AS v" must SUCCEED | ||
| // and return 7. Proves SHORT is fine for stateless queries; only the | ||
| // connection-scoped prepared-statement handle breaks under SHORT. |
| @@ -0,0 +1,556 @@ | |||
| [MySQL](https://www.mysql.com/)是著名的开源的关系型数据库,为了使用户更快捷地访问mysql并充分利用bthread的并发能力,brpc直接支持mysql协议。示例程序:[example/mysql_c++](https://github.com/brpc/brpc/tree/master/example/mysql_c++/) | |||
|
|
|||
| **注意**:只支持MySQL 4.1 及之后的版本的文本协议,支持事务,不支持Prepared statement。目前支持的鉴权方式为mysql_native_password,使用事务的时候不支持single模式。 | |||
| DEFINE_int32(mysql_statment_map_size, | ||
| 100, | ||
| "Mysql statement map size, usually equal to max bthread number"); |
Implements a MySQL client protocol for brpc built on a clean-room authentication codec (no GPL lineage), addressing #2093.
Scope
mysql_native_passwordandcaching_sha2_password(fast-auth + full-auth RSA / secure-transport cleartext), derived from the public MySQL protocol documentation.stmt_idcaching and transparent re-prepare when a statement lands on a fresh connection.Shared-core footprint (kept redis-like / minimal)
The only generic
Controller/Socketadditions are: a socket reserve/use hook andfd_version(ABA guard) for transaction/prepared-statement affinity, an auth empty-write guard for MySQL's server-greets-first handshake, and protocol registration. No protocol-specific type lives on the sharedController(the prepared-statement pointer rides a generic opaque per-RPC slot).Tests
Clean-room integration tests (transactions, prepared statements, pooled-connection concurrency, connection-type boundary) run against a self-spawned
mysqld; 19/19 passing locally. Concurrency coverage includes per-transaction connection pinning under overlap, abort/auto-rollback under contention, and prepared re-prepare when a connection is stolen by a transaction.🤖 Generated with Claude Code