Skip to content

Add MySQL client protocol (text protocol, transactions, prepared statements) on clean-room auth (#2093)#3330

Open
rajvarun77 wants to merge 5 commits into
apache:masterfrom
rajvarun77:mysql-text-protocol-clean-room
Open

Add MySQL client protocol (text protocol, transactions, prepared statements) on clean-room auth (#2093)#3330
rajvarun77 wants to merge 5 commits into
apache:masterfrom
rajvarun77:mysql-text-protocol-clean-room

Conversation

@rajvarun77
Copy link
Copy Markdown
Contributor

Implements a MySQL client protocol for brpc built on a clean-room authentication codec (no GPL lineage), addressing #2093.

Scope

  • Clean-room connection-phase authmysql_native_password and caching_sha2_password (fast-auth + full-auth RSA / secure-transport cleartext), derived from the public MySQL protocol documentation.
  • COM_QUERY text protocol — OK / ERR / EOF / text result-set parsing.
  • Interactive transactions via connection affinity (reserve/use a pooled connection; SINGLE rejected, auto-rollback on drop).
  • Prepared statements (binary protocol) with per-connection stmt_id caching and transparent re-prepare when a statement lands on a fresh connection.

Shared-core footprint (kept redis-like / minimal)

The only generic Controller/Socket additions are: a socket reserve/use hook and fd_version (ABA guard) for transaction/prepared-statement affinity, an auth empty-write guard for MySQL's server-greets-first handshake, and protocol registration. No protocol-specific type lives on the shared Controller (the prepared-statement pointer rides a generic opaque per-RPC slot).

Tests

Clean-room integration tests (transactions, prepared statements, pooled-connection concurrency, connection-type boundary) run against a self-spawned mysqld; 19/19 passing locally. Concurrency coverage includes per-transaction connection pinning under overlap, abort/auto-rollback under contention, and prepared re-prepare when a connection is stolen by a transaction.

🤖 Generated with Claude Code

rajvarun77 and others added 3 commits June 4, 2026 13:07
Clean-room implementation of the MySQL connection-phase authentication handshake,
derived from the public MySQL protocol documentation with no GPL lineage:
mysql_native_password and caching_sha2_password scrambles, HandshakeV10/
HandshakeResponse41 codec, and length-encoded integer/string plus packet-header
wire helpers. Handles the lenenc NULL (0xFB) marker and rejects an oversize
auth_response.
…statements

Port the MySQL protocol client (issue apache#2093) onto the clean-room auth codec and
protobuf 3.21 (NonreflectableMessage): COM_QUERY text protocol, interactive
transactions via connection affinity, and prepared statements. Wire
caching_sha2_password (fast-auth, full-auth RSA, and secure-transport cleartext)
into the live client. Fix the lenenc 9-byte length marker in pack_encode_length
(0xFD -> 0xFE) per the MySQL protocol spec.
… Controller cleanup

- Add clean-room integration tests (transactions, prepared statements, pooled
  connection concurrency, connection-type) run against a self-spawned mysqld.
- Fix: a failed COM_STMT_PREPARE now returns the ERR packet to the caller and keeps
  the connection alive, instead of closing the socket.
- Warn when a prepared statement runs on a 'short' connection (re-prepares on every
  execute; prefer 'pooled').
- Replace Controller's mysql-specific _mysql_stmt with a generic opaque per-RPC slot
  so no protocol type leaks onto the shared Controller.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rajvarun77 rajvarun77 force-pushed the mysql-text-protocol-clean-room branch from b598971 to a292dbc Compare June 4, 2026 19:37
…ted images

Move all MySQL sources (mysql.*, mysql_command/reply/common/transaction/
statement*, mysql_protocol.*, mysql_authenticator.*) into src/brpc/policy/mysql/
alongside the clean-room auth codec; update all includes, build globs, and
install rules. Remove three benchmark images inherited from the apache#2093 port and
the doc section referencing them.

No behavior change: full build green; all 19 mysql unit/integration tests pass;
a 30-case standalone end-to-end run was independently verified against mysqld.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rajvarun77 rajvarun77 force-pushed the mysql-text-protocol-clean-room branch 5 times, most recently from d3989c8 to be457a0 Compare June 4, 2026 22:09
… cases) + ASF headers

- Binary DATETIME/TIME: gate the microsecond bytes on the packet length, not the
  column's declared decimals (over-read / result-set desync).
- COM_STMT_SEND_LONG_DATA: frame stmt_id/param_id inside the packet; fix chunk offset.
- COM_STMT_EXECUTE: emit the trailing 0-length packet for 16MiB-aligned payloads.
- OK/EOF status & warnings: decode via mysql_uint2korr (big-endian safe).
- Row NULL-bitmap: arena-allocate instead of a stack VLA; cap column_count.
- Auth: bounds-check the parsed auth string; size-bound StringPiece uses.
- Prepared stmt: prune stale per-socket stmt_id map entries; count only real '?'
  placeholders (skip quotes/comments).
- MysqlResponse::Clear and MysqlRequest copy/Swap: reset/copy all members.
- Controller::ResetPods: release _bind_sock on controller reuse.
- Standardize mysql file license headers to the ASF form; clarify auth comment.

All 19 unit/integration tests still pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rajvarun77 rajvarun77 force-pushed the mysql-text-protocol-clean-room branch from be457a0 to 0bb55ee Compare June 4, 2026 22:16
@rajvarun77 rajvarun77 marked this pull request as ready for review June 4, 2026 22:20
@rajvarun77
Copy link
Copy Markdown
Contributor Author

@chenBright @wwbmmm — marking this ready for review.

Full MySQL client protocol for brpc — COM_QUERY text protocol, interactive transactions, and prepared statements — built on a clean-room authentication codec (no GPL lineage), addressing #2093.

  • Auth (clean-room): mysql_native_password + caching_sha2_password (fast-auth + full-auth RSA / secure-transport cleartext), derived from the public MySQL protocol docs.
  • Minimal shared-core footprint (redis-like): the only generic Controller/Socket additions are a socket reserve/use hook + fd_version (ABA guard) for transaction/prepared-statement affinity, an auth empty-write guard for the server-greets-first handshake, and protocol registration. No protocol-specific type lives on the shared Controller.
  • Tests: clean-room integration tests (transactions, prepared statements, pooled-connection concurrency, connection-type, binary TIME/DATETIME) run against a self-spawned mysqld — 20/20 passing locally; a standalone 30-case end-to-end run was independently verified.

The clean-room auth codec is also up as a smaller, focused PR (#3310, Stage 1a) if you'd prefer to review that foundation first.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a first-class MySQL client protocol implementation to brpc, including clean-room authentication, text protocol querying, transaction connection-affinity, and prepared statements support, along with new integration/unit tests and examples.

Changes:

  • Introduce PROTOCOL_MYSQL and register a MySQL client protocol stack (pack/parse/process) with connection-phase authentication handling.
  • Add MySQL client APIs (MysqlRequest, MysqlResponse, prepared statements, transactions) and supporting wire/protocol utilities.
  • Add MySQL-focused tests, docs, and examples; extend build/test integration (CMake/Bazel/Makefile).

Reviewed changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
test/mysql/README.md Documents auth-handshake integration test modes and setup.
test/mysql/brpc_mysql_connection_type_unittest.cpp Adds integration test for SHORT connection behavior with prepared statements and plain queries.
test/mysql/brpc_mysql_auth_packet_unittest.cpp Adds unit tests for MySQL auth packet wire helpers (lenenc ints/strings, headers).
test/CMakeLists.txt Ensures MySQL unit tests are included in the CMake test glob.
test/BUILD.bazel Adds a Bazel cc_test target covering MySQL unit tests.
src/brpc/socket.h Exposes fd_version() for ABA-guarding per-connection statement caching.
src/brpc/socket.cpp Increments fd version on fd reset; allows empty writes for auth-handshake flows.
src/brpc/policy/mysql/mysql.h Introduces MySQL request/response message types and statement stub plumbing.
src/brpc/policy/mysql/mysql.cpp Implements MysqlRequest/MysqlResponse serialization and parsing entrypoints.
src/brpc/policy/mysql/mysql_transaction.h Adds MySQL transaction API and options for connection-affinity usage.
src/brpc/policy/mysql/mysql_transaction.cpp Implements transaction start/commit/rollback with socket reserve/use semantics.
src/brpc/policy/mysql/mysql_statement.h Adds prepared statement API and statement-id caching interface.
src/brpc/policy/mysql/mysql_statement.cpp Implements stmt-id cache keyed by socket id + fd version; placeholder counting.
src/brpc/policy/mysql/mysql_statement_inl.h Provides DBD map helpers for stmt-id caching and related gflag.
src/brpc/policy/mysql/mysql_protocol.h Declares MySQL protocol functions (parse/process/serialize/pack).
src/brpc/policy/mysql/mysql_protocol.cpp Implements client-side MySQL protocol codec, including auth handshake and prepare/execute flows.
src/brpc/policy/mysql/mysql_common.h Adds MySQL constants/types and collation mapping used by the client implementation.
src/brpc/policy/mysql/mysql_common.cpp Defines common MySQL defaults and type-to-string helper.
src/brpc/policy/mysql/mysql_command.h Declares MySQL command packet builders for query/prepare/execute/long-data.
src/brpc/policy/mysql/mysql_command.cpp Implements MySQL packetization and prepared statement wire builders.
src/brpc/policy/mysql/mysql_authenticator.h Adds MysqlAuthenticator carrying user/password/schema/params/collation.
src/brpc/policy/mysql/mysql_authenticator.cpp Implements credential serialization and clean-room scrambles (native + caching_sha2 fast path).
src/brpc/policy/mysql/mysql_auth_scramble.h Declares clean-room auth scramble and RSA/cleartext helpers.
src/brpc/policy/mysql/mysql_auth_scramble.cpp Implements native and caching_sha2 scrambles + RSA-OAEP encryption with OpenSSL EVP.
src/brpc/policy/mysql/mysql_auth_packet.h Adds wire helpers for lenenc ints/strings, packet headers, NUL strings.
src/brpc/policy/mysql/mysql_auth_packet.cpp Implements the MySQL wire-format helpers used by handshake parsing/building.
src/brpc/policy/mysql/mysql_auth_handshake.h Declares clean-room handshake packet codecs (greeting/response/switch/more-data).
src/brpc/policy/mysql/mysql_auth_handshake.cpp Implements parsing/building for connection-phase handshake payloads.
src/brpc/options.proto Adds PROTOCOL_MYSQL enum value.
src/brpc/global.cpp Registers the MySQL protocol with the global protocol registry.
src/brpc/details/controller_private_accessor.h Adds private controller hooks for socket reserve/use and opaque session data.
src/brpc/controller.h Introduces bind-socket actions/state to support MySQL transaction affinity and session data.
src/brpc/controller.cpp Implements socket reserve/use behavior across RPCs for pooled/short connection types.
Makefile Includes src/brpc/policy/mysql in the build directories list.
example/mysql_c++/mysqlclient_press.cpp Adds libmysqlclient-based press tool (comparison baseline).
example/mysql_c++/mysql_tx.cpp Adds brpc MySQL transaction example.
example/mysql_c++/mysql_stmt.cpp Adds brpc MySQL prepared statement example.
example/mysql_c++/mysql_press.cpp Adds brpc MySQL press example for CRUD.
example/mysql_c++/mysql_go_press.go Adds Go MySQL press example (comparison baseline).
example/mysql_c++/mysql_cli.cpp Adds brpc-based interactive MySQL CLI example.
example/mysql_c++/CMakeLists.txt Adds CMake build for MySQL examples and dependency discovery.
docs/cn/mysql_client.md Adds/updates Chinese documentation describing brpc MySQL client usage.
Comments suppressed due to low confidence (1)

src/brpc/policy/mysql/mysql_authenticator.h:92

  • The trailing #endif comment refers to COUCHBASE, which doesn’t match this header guard and is confusing when grepping/include-debugging. Update it to BRPC_POLICY_MYSQL_AUTHENTICATOR_H.
}  // namespace policy
}  // namespace brpc

#endif  // BRPC_POLICY_COUCHBASE_AUTHENTICATOR_H


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +29 to +33
extern const char* MysqlDefaultCollation;
extern const char* MysqlBinaryCollation;
const std::map<std::string, uint8_t> MysqlCollations = {
{"big5_chinese_ci", 1},
{"latin2_czech_cs", 2},
Comment on lines +439 to +445
size_t MysqlResponse::ByteSizeLong() const {
return _cached_size_;
}

void MysqlResponse::MergeFrom(const MysqlResponse& from) {
CHECK_NE(&from, this);
}
Comment on lines +40 to +48
// * PreparedStatementUnderShortMustError (PRIMARY):
// build a SHORT channel, prepare "SELECT ? AS v", bind one INT param,
// CallMethod. Must ERROR (cntl.Failed() OR reply(0).is_error()); must
// NOT return a correct result set; must NOT crash. Looped a few times.
//
// * PlainQueryUnderShortMustSucceed (POSITIVE CONTROL):
// same SHORT channel; a stateless COM_QUERY "SELECT 7 AS v" must SUCCEED
// and return 7. Proves SHORT is fine for stateless queries; only the
// connection-scoped prepared-statement handle breaks under SHORT.
Comment thread docs/cn/mysql_client.md
@@ -0,0 +1,556 @@
[MySQL](https://www.mysql.com/)是著名的开源的关系型数据库,为了使用户更快捷地访问mysql并充分利用bthread的并发能力,brpc直接支持mysql协议。示例程序:[example/mysql_c++](https://github.com/brpc/brpc/tree/master/example/mysql_c++/)

**注意**:只支持MySQL 4.1 及之后的版本的文本协议,支持事务,不支持Prepared statement。目前支持的鉴权方式为mysql_native_password,使用事务的时候不支持single模式。
Comment on lines +26 to +28
DEFINE_int32(mysql_statment_map_size,
100,
"Mysql statement map size, usually equal to max bthread number");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants