Ruben’s MySQL codec. #98

Closed
wants to merge 36 commits into
from
Commits
Jump to file
+3,381 −1
Split
@@ -17,7 +17,7 @@ object ExternalMemcached { self =>
p.waitFor()
require(p.exitValue() == 0, "memcached binary must be present.")
}
-
+
private[this] def findAddress() {
var tries = 100
while (address == None && tries >= 0) {
@@ -0,0 +1,67 @@
+A MySQL client built for finagle.
+
+---
+## Overview
+
+*This is meant to give a very brief overview of the MySQL Client/Server protocol and reference relevant source code within finagle-mysql. For an exposition of the MySQL Client/Server protocol, refer to [MySQL Forge](http://forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol)
+and [Understanding MySQL Internal](http://my.safaribooksonline.com/book/databases/mysql/0596009577/client-server-communication/orm9780596009571-chp-4).*
+
+**Packets** - The basic unit of communication for the MySQL Client/Server protocol is an application-layer packet. A MySQL packet consists of a header (size and sequence number) followed by a body. Packets can be fragmented across frames during transmission. To simplify the decoding of results received from the server, the codec includes a packet frame decoder on the pipeline.
+
+`* protocol/Packet.scala, codec/PacketFrameDecoder.scala`
+
+**Handshake and Authentication** - When a client connects to the server, the server sends a greeting. This greeting contains information about the server version, protocol, capabilities, etc. The client replies with a login request containing, among other information, authentication credentials. To ensure connections are authenticated before they are issued by finagle, the codec implements prepareConnFactory with an AuthenticationProxy.
+
+`* protocol/Handshake.scala, Codec.scala`
+
+**Capabilities** - The client and server express their capability succinctly as bit vectors. Each set bit in the vector represents what the client/server is capable of or willing to do. Finagle-mysql provides constants of all the capability flags available (at the time of writing) and a comprehensive way to build capability bit vectors.
+
+ // This clients capabilities
+ val clientCapability = Capability(
+ LongFlag,
+ Transactions,
+ Protocol41,
+ FoundRows,
+ Interactive,
+ LongPassword,
+ ConnectWithDB,
+ SecureConnection,
+ LocalFiles
+ )
+`* protocol/Capability.scala, Codec.scala`
+
+ Note: This client only supports protocol version 4.1 (available in MySQL version 4.1 and above). This is strictly enforced during authentication with a MySQL server.
+
+**Requests** - Most requests sent to the server are Command Requests. They contain a command byte and any arguments that are specific to the command. Command Requests are sent to the server as a MySQL Packet. The first byte of the packet body must contain a valid command byte followed by the arguments. Within finagle-mysql, each Request object has a data field which defines the body of the Packet. Requests are translated into a logical MySQL Packet by the toChannelBuffer method when they reach the Encoder.
+
+`* protocol/Request.scala`
+
+**Results** - finagle-mysql translates packets received from the server into Scala objects. Each result object has a relevant decode method that translates the packet(s) into the object according to the protocol. Result packets can be distinguished by their first byte. Some result packets denote the start of a longer transmission and need to be defragged by the decoder.
+
+ResultSets are returned from the server for queries that return Rows. A Row can be String encoded or Binary encoded depending on the Request used to execute the query. For example, a QueryRequest uses the String protocol and a PreparedStatement uses the binary protocol.
+
+`* codec/Endec.scala, protocol/{Result.scala, ResultSet.scala, PreparedStatement.scala}`
+
+**Value** - finagle-mysql provides a Value ADT that can represent all values returned from MySQL. However, this does not include logic to decode every data type. For unsupported values finagle-mysql will return a RawStringValue and RawBinaryValue for the String and Binary protocols, respectively. Other note worthy Value objects include NullValue (SQL NULL) and EmptyValue.
+
+The following code depicts a robust, safe, and idiomatic way to extract and deconstruct a Value from a Row.
+
+ // The row.valueOf(...) method returns an Option[Value].
+ val userId: Option[Long] = row.valueOf("id") map {
+ case LongValue(id) => id
+ case _ => -1
+ }
+
+Pattern matching all possible values of the Value ADT gives great flexibility and control. For example, it allows the programmer to handle NullValues and EmptyValues with specific application logic.
+
+`* protocol/Value.scala`
+
+**Byte Buffers** - The BufferReader and BufferWriter interfaces provide convenient methods for reading/writing primitive data types exchanged between the client/server. This includes all primitive numeric types and strings (null-terminated and length coded). All Buffer methods are side-effecting, that is, each call to a read*/write* method will increase the current read and write position. Note, the bytes exchanged between the client/server are encoded in little-endian byte order.
+
+`* protocol/Buffer.scala`
+
+**Charset** - Currently, finagle-mysql only supports UTF-8 character encodings. This is strictly enforced when authenticating with a MySQL server. For more information about how to configure a MySQL instance to use UTF-8 refer to the [MySQL Documentation](http://dev.mysql.com/doc/refman/5.0/en/charset-applications.html).
+
+Note: MySQL also supports variable charsets at the table and field level. This charset data is part of the [Field Packet](http://dev.mysql.com/doc/internals/en/field-packet.html).
+
+`* protocol/Charset.scala`
View
@@ -0,0 +1,43 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-mysql</artifactId>
+ <packaging>jar</packaging>
+ <version>4.0.3-SNAPSHOT</version>
@mariusae
mariusae Aug 10, 2012 Twitter, Inc. member

version needs update.

+ <parent>
+ <groupId>com.twitter</groupId>
+ <artifactId>scala-parent</artifactId>
+ <version>0.0.2</version>
+ <relativePath>../../parents/scala-parent/pom.xml</relativePath>
+ </parent>
+ <properties>
+ <git.dir>${project.basedir}/../../.git</git.dir>
+ </properties>
+ <dependencies>
+ <!-- library dependencies -->
+ <!-- project dependencies -->
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-core</artifactId>
+ <version>4.0.3-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>util-logging</artifactId>
+ <version>4.0.1</version>
+ </dependency>
@mariusae
mariusae Aug 10, 2012 Twitter, Inc. member

ditto for these.

+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <excludes>
+ </excludes>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+</project>
@@ -0,0 +1,169 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle.builder.ClientBuilder
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.Query
+import com.twitter.finagle.Service
+import com.twitter.finagle.{ServiceFactory, Codec, CodecFactory}
+import com.twitter.util.Future
+
+object Client {
+ /**
+ * Constructs a Client given a ServiceFactory.
+ */
+ def apply(factory: ServiceFactory[Request, Result]): Client = {
+ new Client(factory)
+ }
+
+ /**
+ * Constructs a ServiceFactory using a single host.
+ * @param host a String of host:port combination.
+ * @param username the username used to authenticate to the mysql instance
+ * @param password the password used to authenticate to the mysql instance
+ * @param dbname database to initially use
+ */
+ def apply(host: String, username: String, password: String, dbname: Option[String]): Client = {
+ val factory = ClientBuilder()
+ .codec(new MySQL(username, password, dbname))
+ .hosts(host)
+ .hostConnectionLimit(1)
+ .buildFactory()
+
+ apply(factory)
+ }
+
+ def apply(host: String, username: String, password: String): Client = {
+ apply(host, username, password, None)
+ }
+
+ def apply(host: String, username: String, password: String, dbname: String): Client = {
+ apply(host, username, password, Some(dbname))
+ }
+}
+
+class Client(factory: ServiceFactory[Request, Result]) {
+ private[this] lazy val fService = factory.apply()
+
+ /**
+ * Sends a query to the server without using
+ * prepared statements.
+ * @param sql An sql statement to be executed.
+ * @return an OK Result or a ResultSet for queries that return
+ * rows.
+ */
+ def query(sql: String) = send(QueryRequest(sql)) {
+ case rs: ResultSet => Future.value(rs)
+ case ok: OK => Future.value(ok)
+ }
+
+ /**
+ * Runs a query that returns a result set. For each row
+ * in the ResultSet, call f on the row and return the results.
+ * @param sql A sql statement that returns a result set.
+ * @param f A function from ResultSet to any type T.
+ * @return a Future of Seq[T]
+ */
+ def select[T](sql: String)(f: Row => T): Future[Seq[T]] = query(sql) map {
+ case rs: ResultSet => rs.rows.map(f)
+ case ok: OK => Seq()
+ }
+
+ /**
+ * Sends a query to server to be prepared for execution.
+ * @param sql A query to be prepared on the server.
+ * @return PreparedStatement
+ */
+ def prepare(sql: String, params: Any*) = {
@mariusae
mariusae Aug 10, 2012 Twitter, Inc. member

explicitly annotate return type here.

+ val stmt = Query.expandParams(sql, params)
+ send(PrepareRequest(stmt)) {
+ case ps: PreparedStatement =>
+ ps.statement.setValue(stmt)
+ if(params.size > 0)
+ ps.parameters = Query.flatten(params).toArray
+
+ Future.value(ps)
+ }
+ }
+
+ /**
+ * Execute a prepared statement.
+ * @return an OK Result or a ResultSet for queries that return
+ * rows.
+ */
+ def execute(ps: PreparedStatement) = send(ExecuteRequest(ps)) {
+ case rs: ResultSet =>
+ ps.bindParameters()
+ Future.value(rs)
+ case ok: OK =>
+ ps.bindParameters()
+ Future.value(ok)
+ }
+
+ /**
+ * Combines the prepare and execute operations.
+ * @return a Future[(PreparedStatement, Result)] tuple.
+ */
+ def prepareAndExecute(sql: String, params: Any*) =
+ prepare(sql, params: _*) flatMap { ps =>
+ execute(ps) map {
+ res => (ps, res)
+ }
+ }
+
+
+ /**
+ * Runs a query that returns a result set. For each row
+ * in the ResultSet, call f on the row and return the results.
+ * @param ps A prepared statement.
+ * @param f A function from ResultSet to any type T.
+ * @return a Future of Seq[T]
+ */
+ def select[T](ps: PreparedStatement)(f: Row => T): Future[Seq[T]] = execute(ps) map {
+ case rs: ResultSet => rs.rows.map(f)
+ case ok: OK => Seq()
+ }
+
+ /**
+ * Combines the prepare and select operations.
+ * @return a Future[(PreparedStatement, Seq[T])] tuple.
+ */
+ def prepareAndSelect[T](sql: String, params: Any*)(f: Row => T) =
+ prepare(sql, params: _*) flatMap { ps =>
+ select(ps)(f) map {
+ seq => (ps, seq)
+ }
+ }
+
+ /**
+ * Close a prepared statement on the server.
+ * @return OK result.
+ */
+ def closeStatement(ps: PreparedStatement) = send(CloseRequest(ps)) {
+ case ok: OK => Future.value(ok)
+ }
+
+ def selectDB(schema: String) = send(UseRequest(schema)) {
+ case ok: OK => Future.value(ok)
+ }
+
+ def ping = send(PingRequest) {
+ case ok: OK => Future.value(ok)
+ }
+
+ /**
+ * Close the ServiceFactory and its underlying resources.
+ */
+ def close() = factory.close()
+
+ /**
+ * Helper function to send requests to the ServiceFactory
+ * and handle Error responses from the server.
+ */
+ private[this] def send[T](r: Request)(handler: PartialFunction[Result, Future[T]]) =
+ fService flatMap { service =>
+ service(r) flatMap (handler orElse {
+ case Error(c, s, m) => Future.exception(ServerError(c + " - " + m))
+ case result => Future.exception(ClientError("Unhandled result from server: " + result))
+ })
+ }
+}
@mariusae
mariusae Aug 10, 2012 Twitter, Inc. member

for all public interfaces here: include return types. it serves both as (in code) documentation and it also fixes the types, which is important; otherwise refinement types may be exposed unintentionally, causing API conflicts later:

http://twitter.github.com/effectivescala/#Types%20and%20Generics-Return%20type%20annotations

@@ -0,0 +1,88 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle._
+import com.twitter.finagle.mysql.codec.{PacketFrameDecoder, Endec}
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.protocol.Capability._
+import com.twitter.util.Future
+import org.jboss.netty.channel.{ChannelPipelineFactory, Channels, Channel}
+
+class MySQL(username: String, password: String, database: Option[String])
+ extends CodecFactory[Request, Result] {
+ private[this] val clientCapability = Capability(
+ LongFlag,
+ Transactions,
+ Protocol41,
+ FoundRows,
+ Interactive,
+ LongPassword,
+ ConnectWithDB,
+ SecureConnection,
+ LocalFiles
+ )
+
+ def server = throw new Exception("Not yet implemented...")
+
+ def client = Function.const {
+ new Codec[Request, Result] {
+
+ def pipelineFactory = new ChannelPipelineFactory {
+ def getPipeline = {
+ val pipeline = Channels.pipeline()
+
+ pipeline.addLast("frameDecoder", new PacketFrameDecoder)
+ pipeline.addLast("EncoderDecoder", new Endec)
+
+ pipeline
+ }
+ }
+
+ // Authenticate each connection before returning it via a ServiceFactoryProxy.
+ override def prepareConnFactory(underlying: ServiceFactory[Request, Result]) =
+ new AuthenticationProxy(underlying, username, password, database, clientCapability)
+
+ }
+ }
+}
+
+class AuthenticationProxy(
+ underlying: ServiceFactory[Request, Result],
+ username: String,
+ password: String,
+ database: Option[String],
+ clientCap: Capability)
+ extends ServiceFactoryProxy(underlying) {
+
+ def makeLoginReq(sg: ServersGreeting) =
+ LoginRequest(username, password, database, clientCap, sg.salt, sg.serverCap)
+
+ def acceptGreeting(res: Result) = res match {
+ case sg: ServersGreeting if !sg.serverCap.has(Capability.Protocol41) =>
+ Future.exception(IncompatibleServer("This client is only compatible with MySQL version 4.1 and later."))
+
+ case sg: ServersGreeting if !Charset.isUTF8(sg.charset) =>
+ Future.exception(IncompatibleServer("This client is only compatible with UTF-8 charset encoding."))
+
+ case sg: ServersGreeting =>
+ Future.value(sg)
+
+ case r =>
+ Future.exception(new ClientError("Invalid Reply type %s".format(r.getClass.getName)))
+ }
+
+ def acceptLogin(res: Result) = res match {
+ case r: OK =>
+ Future.value(res)
+
+ case Error(c, _, m) =>
+ Future.exception(ServerError("Error when authenticating the client "+ c + " - " + m))
+ }
+
+ override def apply(conn: ClientConnection) = for {
+ service <- self(conn)
+ result <- service(ClientInternalGreet)
+ sg <- acceptGreeting(result)
+ loginRes <- service(makeLoginReq(sg))
+ _ <- acceptLogin(loginRes)
+ } yield service
+}
Oops, something went wrong. Retry.