Ruben’s MySQL codec.

finagle-memcached/src/test/scala/com/twitter/finagle/memcached/integration/ExternalMemcached.scala

@@ -17,7 +17,7 @@ object ExternalMemcached { self =>
     p.waitFor()
     require(p.exitValue() == 0, "memcached binary must be present.")
   }
-
+  
   private[this] def findAddress() {
     var tries = 100
     while (address == None && tries >= 0) {

finagle-mysql/README.md

@@ -0,0 +1,67 @@
+A MySQL client built for finagle.
+
+---
+## Overview  
+
+*This is meant to give a very brief overview of the MySQL Client/Server protocol and reference relevant source code within finagle-mysql. For an exposition of the MySQL Client/Server protocol, refer to [MySQL Forge](http://forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol)
+and [Understanding MySQL Internal](http://my.safaribooksonline.com/book/databases/mysql/0596009577/client-server-communication/orm9780596009571-chp-4).*
+
+**Packets** - The basic unit of communication for the MySQL Client/Server protocol is an application-layer packet. A MySQL packet consists of a header (size and sequence number) followed by a body. Packets can be fragmented across frames during transmission. To simplify the decoding of results received from the server, the codec includes a packet frame decoder on the pipeline.
+
+`* protocol/Packet.scala, codec/PacketFrameDecoder.scala`
+
+**Handshake and Authentication** - When a client connects to the server, the server sends a greeting. This greeting contains information about the server version, protocol, capabilities, etc. The client replies with a login request containing, among other information, authentication credentials. To ensure connections are authenticated before they are issued by finagle, the codec implements prepareConnFactory with an AuthenticationProxy.
+
+`* protocol/Handshake.scala, Codec.scala`
+
+**Capabilities** - The client and server express their capability succinctly as bit vectors. Each set bit in the vector represents what the client/server is capable of or willing to do. Finagle-mysql provides constants of all the capability flags available (at the time of writing) and a comprehensive way to build capability bit vectors.
+
+    // This clients capabilities
+    val clientCapability = Capability(
+      LongFlag, 
+      Transactions, 
+      Protocol41, 
+      FoundRows, 
+      Interactive, 
+      LongPassword, 
+      ConnectWithDB, 
+      SecureConnection, 
+      LocalFiles
+    )
+`* protocol/Capability.scala, Codec.scala`
+
+ Note: This client only supports protocol version 4.1 (available in MySQL version 4.1 and above). This is strictly enforced during authentication with a MySQL server.
+
+**Requests** - Most requests sent to the server are Command Requests. They contain a command byte and any arguments that are specific to the command. Command Requests are sent to the server as a MySQL Packet. The first byte of the packet body must contain a valid command byte followed by the arguments. Within finagle-mysql, each Request object has a data field which defines the body of the Packet. Requests are translated into a logical MySQL Packet by the toChannelBuffer method when they reach the Encoder.
+
+`* protocol/Request.scala`
+
+**Results** - finagle-mysql translates packets received from the server into Scala objects. Each result object has a relevant decode method that translates the packet(s) into the object according to the protocol. Result packets can be distinguished by their first byte. Some result packets denote the start of a longer transmission and need to be defragged by the decoder.
+
+ResultSets are returned from the server for queries that return Rows. A Row can be String encoded or Binary encoded depending on the Request used to execute the query. For example, a QueryRequest uses the String protocol and a PreparedStatement uses the binary protocol.
+
+`* codec/Endec.scala, protocol/{Result.scala, ResultSet.scala, PreparedStatement.scala}`
+
+**Value** - finagle-mysql provides a Value ADT that can represent all values returned from MySQL. However, this does not include logic to decode every data type. For unsupported values finagle-mysql will return a RawStringValue and RawBinaryValue for the String and Binary protocols, respectively. Other note worthy Value objects include NullValue (SQL NULL) and EmptyValue.
+
+The following code depicts a robust, safe, and idiomatic way to extract and deconstruct a Value from a Row.
+
+    // The row.valueOf(...) method returns an Option[Value].
+    val userId: Option[Long] = row.valueOf("id") map {
+      case LongValue(id) => id
+      case _ => -1
+    }
+
+Pattern matching all possible values of the Value ADT gives great flexibility and control. For example, it allows the programmer to handle NullValues and EmptyValues with specific application logic.
+
+`* protocol/Value.scala`
+
+**Byte Buffers** - The BufferReader and BufferWriter interfaces provide convenient methods for reading/writing primitive data types exchanged between the client/server. This includes all primitive numeric types and strings (null-terminated and length coded). All Buffer methods are side-effecting, that is, each call to a read*/write* method will increase the current read and write position. Note, the bytes exchanged between the client/server are encoded in little-endian byte order.
+
+`* protocol/Buffer.scala`
+
+**Charset** - Currently, finagle-mysql only supports UTF-8 character encodings. This is strictly enforced when authenticating with a MySQL server. For more information about how to configure a MySQL instance to use UTF-8 refer to the [MySQL Documentation](http://dev.mysql.com/doc/refman/5.0/en/charset-applications.html).
+
+Note: MySQL also supports variable charsets at the table and field level. This charset data is part of the [Field Packet](http://dev.mysql.com/doc/internals/en/field-packet.html).
+
+`* protocol/Charset.scala`

finagle-mysql/pom.xml

@@ -0,0 +1,43 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+  <groupId>com.twitter</groupId>
+  <artifactId>finagle-mysql</artifactId>
+  <packaging>jar</packaging>
+  <version>4.0.3-SNAPSHOT</version>
+  <parent>
+    <groupId>com.twitter</groupId>
+    <artifactId>scala-parent</artifactId>
+    <version>0.0.2</version>
+    <relativePath>../../parents/scala-parent/pom.xml</relativePath>
+  </parent>
+  <properties>
+    <git.dir>${project.basedir}/../../.git</git.dir>
+  </properties>
+  <dependencies>
+    <!-- library dependencies -->
+    <!-- project dependencies -->
+    <dependency>
+      <groupId>com.twitter</groupId>
+      <artifactId>finagle-core</artifactId>
+      <version>4.0.3-SNAPSHOT</version>
+    </dependency>
+    <dependency>
+      <groupId>com.twitter</groupId>
+      <artifactId>util-logging</artifactId>
+      <version>4.0.1</version>
+    </dependency>
+  </dependencies>
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-surefire-plugin</artifactId>
+        <configuration>
+          <excludes>
+          </excludes>
+        </configuration>
+      </plugin>
+    </plugins>
+  </build>
+</project>

finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Client.scala

@@ -0,0 +1,169 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle.builder.ClientBuilder
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.Query
+import com.twitter.finagle.Service
+import com.twitter.finagle.{ServiceFactory, Codec, CodecFactory}
+import com.twitter.util.Future
+
+object Client {
+  /**
+   * Constructs a Client given a ServiceFactory.
+   */
+  def apply(factory: ServiceFactory[Request, Result]): Client = {
+    new Client(factory)
+  }
+
+  /**
+   * Constructs a ServiceFactory using a single host.
+   * @param host a String of host:port combination.
+   * @param username the username used to authenticate to the mysql instance
+   * @param password the password used to authenticate to the mysql instance
+   * @param dbname database to initially use
+   */
+  def apply(host: String, username: String, password: String, dbname: Option[String]): Client = {
+    val factory = ClientBuilder()
+      .codec(new MySQL(username, password, dbname))
+      .hosts(host)
+      .hostConnectionLimit(1)
+      .buildFactory()
+
+      apply(factory)
+  }
+
+  def apply(host: String, username: String, password: String): Client = {
+    apply(host, username, password, None)
+  }
+
+  def apply(host: String, username: String, password: String, dbname: String): Client = {
+    apply(host, username, password, Some(dbname))
+  }
+}
+
+class Client(factory: ServiceFactory[Request, Result]) {
+  private[this] lazy val fService = factory.apply()
+
+  /**
+   * Sends a query to the server without using
+   * prepared statements.
+   * @param sql An sql statement to be executed.
+   * @return an OK Result or a ResultSet for queries that return
+   * rows.
+   */
+  def query(sql: String) = send(QueryRequest(sql)) {
+      case rs: ResultSet => Future.value(rs)
+      case ok: OK => Future.value(ok)
+  }
+
+  /**
+   * Runs a query that returns a result set. For each row
+   * in the ResultSet, call f on the row and return the results.
+   * @param sql A sql statement that returns a result set.
+   * @param f A function from ResultSet to any type T.
+   * @return a Future of Seq[T]
+   */
+  def select[T](sql: String)(f: Row => T): Future[Seq[T]] = query(sql) map {
+    case rs: ResultSet => rs.rows.map(f)
+    case ok: OK => Seq()
+  }
+
+  /**
+   * Sends a query to server to be prepared for execution.
+   * @param sql A query to be prepared on the server.
+   * @return PreparedStatement 
+   */
+  def prepare(sql: String, params: Any*) = {
+    val stmt = Query.expandParams(sql, params)
+    send(PrepareRequest(stmt)) {
+      case ps: PreparedStatement =>
+        ps.statement.setValue(stmt)
+        if(params.size > 0)
+          ps.parameters = Query.flatten(params).toArray
+
+        Future.value(ps)
+    } 
+  }
+
+  /**
+   * Execute a prepared statement.
+   * @return an OK Result or a ResultSet for queries that return
+   * rows.
+   */
+  def execute(ps: PreparedStatement) = send(ExecuteRequest(ps)) {
+    case rs: ResultSet =>
+      ps.bindParameters() 
+      Future.value(rs)
+    case ok: OK => 
+      ps.bindParameters()
+      Future.value(ok)
+  }
+
+  /**
+   * Combines the prepare and execute operations.
+   * @return a Future[(PreparedStatement, Result)] tuple.
+   */
+  def prepareAndExecute(sql: String, params: Any*) = 
+    prepare(sql, params: _*) flatMap { ps => 
+      execute(ps) map {
+        res => (ps, res)
+      }
+    }
+
+
+  /**
+   * Runs a query that returns a result set. For each row
+   * in the ResultSet, call f on the row and return the results.
+   * @param ps A prepared statement.
+   * @param f A function from ResultSet to any type T.
+   * @return a Future of Seq[T]
+   */
+  def select[T](ps: PreparedStatement)(f: Row => T): Future[Seq[T]] = execute(ps) map {
+    case rs: ResultSet => rs.rows.map(f)
+    case ok: OK => Seq()
+  }
+
+  /**
+   * Combines the prepare and select operations.
+   * @return a Future[(PreparedStatement, Seq[T])] tuple.
+   */
+  def prepareAndSelect[T](sql: String, params: Any*)(f: Row => T) = 
+    prepare(sql, params: _*) flatMap { ps => 
+      select(ps)(f) map { 
+          seq => (ps, seq)
+      } 
+    }
+
+  /**
+   * Close a prepared statement on the server.
+   * @return OK result.
+   */
+  def closeStatement(ps: PreparedStatement) = send(CloseRequest(ps)) {
+    case ok: OK => Future.value(ok)
+  }
+
+  def selectDB(schema: String) = send(UseRequest(schema)) {
+    case ok: OK => Future.value(ok)
+  }
+
+  def ping = send(PingRequest) {
+    case ok: OK => Future.value(ok)
+  }
+
+  /**
+   * Close the ServiceFactory and its underlying resources.
+   */
+  def close() = factory.close()
+
+  /**
+   * Helper function to send requests to the ServiceFactory
+   * and handle Error responses from the server.
+   */
+  private[this] def send[T](r: Request)(handler: PartialFunction[Result, Future[T]])  = 
+    fService flatMap { service =>
+      service(r) flatMap (handler orElse {
+        case Error(c, s, m) => Future.exception(ServerError(c + " - " + m))
+        case result         => Future.exception(ClientError("Unhandled result from server: " + result))
+      })
+    }
+}

finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Codec.scala

@@ -0,0 +1,88 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle._
+import com.twitter.finagle.mysql.codec.{PacketFrameDecoder, Endec}  
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.protocol.Capability._
+import com.twitter.util.Future
+import org.jboss.netty.channel.{ChannelPipelineFactory, Channels, Channel}
+
+class MySQL(username: String, password: String, database: Option[String]) 
+  extends CodecFactory[Request, Result] {
+    private[this] val clientCapability = Capability(
+      LongFlag, 
+      Transactions, 
+      Protocol41, 
+      FoundRows, 
+      Interactive, 
+      LongPassword, 
+      ConnectWithDB, 
+      SecureConnection, 
+      LocalFiles
+    )
+
+    def server = throw new Exception("Not yet implemented...")
+
+    def client = Function.const {
+      new Codec[Request, Result] {
+
+        def pipelineFactory = new ChannelPipelineFactory {
+          def getPipeline = {
+            val pipeline = Channels.pipeline()
+
+            pipeline.addLast("frameDecoder", new PacketFrameDecoder)
+            pipeline.addLast("EncoderDecoder", new Endec)
+
+            pipeline
+          }
+        }
+
+        // Authenticate each connection before returning it via a ServiceFactoryProxy.
+        override def prepareConnFactory(underlying: ServiceFactory[Request, Result]) = 
+          new AuthenticationProxy(underlying, username, password, database, clientCapability)
+
+      }
+    }
+}
+
+class AuthenticationProxy(
+    underlying: ServiceFactory[Request, Result], 
+    username: String, 
+    password: String,
+    database: Option[String],
+    clientCap: Capability) 
+  extends ServiceFactoryProxy(underlying) {
+
+    def makeLoginReq(sg: ServersGreeting) = 
+      LoginRequest(username, password, database, clientCap, sg.salt, sg.serverCap)
+
+    def acceptGreeting(res: Result) = res match {
+      case sg: ServersGreeting if !sg.serverCap.has(Capability.Protocol41) =>
+        Future.exception(IncompatibleServer("This client is only compatible with MySQL version 4.1 and later."))
+
+      case sg: ServersGreeting if !Charset.isUTF8(sg.charset) => 
+        Future.exception(IncompatibleServer("This client is only compatible with UTF-8 charset encoding."))
+
+      case sg: ServersGreeting =>
+        Future.value(sg)
+
+      case r =>
+       Future.exception(new ClientError("Invalid Reply type %s".format(r.getClass.getName)))
+    }
+
+    def acceptLogin(res: Result) = res match {
+      case r: OK => 
+        Future.value(res)
+
+      case Error(c, _, m) => 
+        Future.exception(ServerError("Error when authenticating the client "+ c + " - " + m))
+    }
+
+    override def apply(conn: ClientConnection) = for {
+      service <- self(conn)
+      result <- service(ClientInternalGreet)
+      sg <- acceptGreeting(result)
+      loginRes <- service(makeLoginReq(sg))
+      _ <- acceptLogin(loginRes)
+    } yield service
+}