Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Ruben’s MySQL codec. #98

Closed
wants to merge 36 commits into from

7 participants

marius a. eriksen Sam Pullara Steve Gury Jeremy Cole Davi Arnaut Ruben Oanta Blake Matheny
marius a. eriksen
Collaborator

This is all Ruben’s code -- I’m submitting it here so we have a forum to do a code review!

Blake Matheny

Looking briefly at the code, naggati doesn't look like a dependency so you can probably drop this.

Collaborator

Yes, it was an old dependency that I forgot to remove. You can remove it.

roanta and others added some commits
Ruben Oanta roanta Removed unused dependency. c450b8b
Ruben Oanta roanta Refactored a bit a2be668
Ruben Oanta roanta Includes the start of a test suite, some refactoring, and bug fixes. …
…The most significant change

includes removing authentication from the codec pipeline and into a ServiceFactoryProxy on
the codec. This more elegantly ensures (through sequential composition) that authentication happens
in the correct order.
bfa7c99
Ruben Oanta roanta removed unused cached files 83d3469
Ruben Oanta roanta Moved common operations used to decode (read) and encode (write) to a…
… byte array into classes BufferReader and BufferWriter, respectively. This removed a lot of code duplication in handling the requests and responses from MySQL.
c64b77d
Ruben Oanta roanta -Fixed an issue when decoding the packet header in the Decoder that w…
…as caused by not masking each byte with 255 and resulted in negative size values.

-Changed the client capabilities to include connectWithDB. Added a check to ensure the server can accept a schema on client connect.
-Added a more extensive decode method for EOF packets.
-Corrected an issue with the take method not correctly incrementing the offset in BufferReader.
e38b7b0
Ruben Oanta roanta -Added the ability to read an unsigned byte and short in BufferReader.
-Fixed an issue where the Field result was being improperly decoded.
b575426
Ruben Oanta -Changed the codec to support more complex results. Decoupled the pac…
…ket defragmenter from

the primary decoder.
-Started implementing PreparedStatements on the codec.
-Added the ability to easily add/remove from a Capability bit mask.
-Fixed an issue where the LoginRequest client capability was improperly set when the database argument was None.
678d55c
Ruben Oanta roanta -Added decoding of prepared statements in ResultDecoder.
-Fixed readLengthCodedBinary to work properly.
7a57e46
Ruben Oanta roanta -Expanded the ResultSet class to be more useful
-Added some basic methods to the Client that use the new ResultSet capabilities.
-Many other small additions, fixes, and changes.
65c15de
Ruben Oanta roanta Changed tabs to spaces c57b59d
Ruben Oanta Implemented an initial api that supports basic queries and prepared
statements.
9fcfc5e
Ruben Oanta roanta -Changed the return type of prepareAndSelect of a Future[(PreparedSta…
…tement, Seq[T]] so that the prepared statement is still accesible after it is executed.

-Added a bindParameters() method to PreparedStatements to indicate that the parameters have been sent to the server.
-Added support for Time, Date, Datetime, and Timestamp fields. Still need to add support for sending those types to the mysql server.
feb4412
Ruben Oanta -Added writeTimestamp and writeDate to Buffer.scala
-Added id method to field that uses name or origName depending on which one is present.
-Simplified the query method in Client.
7dc508f
Ruben Oanta Added better error handling for Query.* methods. 6e5503f
...ain/scala/com/twitter/finagle/mysql/codec/MySQL.scala
((14 lines not shown))
+ def client = Function.const {
+ new Codec[Request, Result] {
+
+ def pipelineFactory = new ChannelPipelineFactory {
+ def getPipeline = {
+ val pipeline = Channels.pipeline()
+
+ pipeline.addLast("frameDecoder", new PacketFrameDecoder)
+ pipeline.addLast("resultDecoder", new ResultDecoder)
+ pipeline.addLast("requestEncoder", RequestEncoder)
+
+ pipeline
+ }
+ }
+
+ /* Authenticate each connection before returning it via a ServiceFactoryProxy. */
marius a. eriksen Collaborator

style: use // comments except for scala docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...ain/scala/com/twitter/finagle/mysql/codec/MySQL.scala
((26 lines not shown))
+ }
+ }
+
+ /* Authenticate each connection before returning it via a ServiceFactoryProxy. */
+ override def prepareConnFactory(underlying: ServiceFactory[Request, Result]) =
+ new AuthenticationProxy(underlying, username, password, database)
+
+ }
+ }
+}
+
+class AuthenticationProxy(underlying: ServiceFactory[Request, Result],
+ username: String,
+ password: String,
+ database: Option[String])
+ extends ServiceFactoryProxy(underlying) {
marius a. eriksen Collaborator

style:

class AuthenticationProxy(
    underlying: ServiceFactory[Request, Result], 
    username: String, 
    password: String,
    database: Option[String]) 
  extends ServiceFactoryProxy(underlying) {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...ain/scala/com/twitter/finagle/mysql/codec/MySQL.scala
((27 lines not shown))
+ }
+
+ /* Authenticate each connection before returning it via a ServiceFactoryProxy. */
+ override def prepareConnFactory(underlying: ServiceFactory[Request, Result]) =
+ new AuthenticationProxy(underlying, username, password, database)
+
+ }
+ }
+}
+
+class AuthenticationProxy(underlying: ServiceFactory[Request, Result],
+ username: String,
+ password: String,
+ database: Option[String])
+ extends ServiceFactoryProxy(underlying) {
+ val greet = new SimpleRequest(Command.COM_NOOP_GREET)
marius a. eriksen Collaborator

private[this] val ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((17 lines not shown))
+ */
+class PacketFrameDecoder extends FrameDecoder {
+ override def decode(ctx: ChannelHandlerContext, channel: Channel, buffer: ChannelBuffer): Packet = {
+ if(buffer.readableBytes < Packet.headerSize)
+ return null
+
+ val header = new Array[Byte](Packet.headerSize)
+ buffer.readBytes(header)
+ val br = new BufferReader(header)
+
+ val (length, seq) = (br.readInt24, br.readByte)
+
+ if(buffer.readableBytes < length)
+ return null
+
+ println("<- Decoding MySQL packet (length=%d, seq=%d)".format(length, seq))

println--

marius a. eriksen Collaborator

spurious println

marius a. eriksen Collaborator

btw-- if you want to retain the debugging information here -- it's often useful -- you should probably either (1) use logging at a debug level, or (2) pass a debug flag into the codec constructor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...ain/scala/com/twitter/finagle/mysql/codec/MySQL.scala
((34 lines not shown))
+ }
+}
+
+class AuthenticationProxy(underlying: ServiceFactory[Request, Result],
+ username: String,
+ password: String,
+ database: Option[String])
+ extends ServiceFactoryProxy(underlying) {
+ val greet = new SimpleRequest(Command.COM_NOOP_GREET)
+
+ def makeLoginReq(sg: ServersGreeting) = LoginRequest(
+ username = username,
+ password = password,
+ database = database,
+ serverCap = sg.serverCap,
+ salt = sg.salt
marius a. eriksen Collaborator

style: using x = x stutters a lot, try instead

def makeLoginReq(sg: ServerGreeting) = 
  LoginRequest(username, password, database, sg.serverCap, sg.salt)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../com/twitter/finagle/mysql/codec/RequestEncoder.scala
@@ -0,0 +1,23 @@
+package com.twitter.finagle.mysql.codec
+
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.BufferUtil
+import org.jboss.netty.buffer.{ChannelBuffers, ChannelBuffer}
+import org.jboss.netty.channel.{Channel, ChannelHandlerContext, MessageEvent, Channels, ChannelEvent}
+import org.jboss.netty.handler.codec.oneone.OneToOneEncoder
+
+object RequestEncoder extends OneToOneEncoder {
+ override def encode(context: ChannelHandlerContext, channel: Channel, message: AnyRef) = {
+ message match {
+ case req: SimpleRequest if req.cmd == Command.COM_NOOP_GREET =>
+ ChannelBuffers.EMPTY_BUFFER
+ case req: Request =>
+ println("-> Encoding " + req)

println--

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((8 lines not shown))
+ * in little endian byte order.
+ */
+
+class BufferReader(val buffer: Array[Byte], private[this] var offset: Int = 0) {
+ require(offset >= 0)
+ require(buffer != null)
+
+ def readable(width: Int = 1): Boolean = offset + width <= buffer.size
+
+ /**
+ * Reads multi-byte numeric values stored in a byte array.
+ * Starts at offset and reads offset+width bytes. The values are
+ * assumed to be stored with the low byte first at data(offset)
+ * (i.e. little endian) and the result is returned as a Long.
+ */
+ def read(width: Int): Long = {

I'm concerned that this code is too functional at such a low level. I can imagine it being a hotspot. Obviously we will see when we do performance testing but usually you want C-like code when doing operations like this one to avoid a lot of CPU and memory overhead.

marius a. eriksen Collaborator

let's measure first, cut later. i'd like to focus on correctness and clarity first, then performance later -- and of course to make sure that the codec isn't structured in a way that wouldn't allow us to squeeze all the performance we can out of it later.

marius a. eriksen Collaborator

(though i do agree-- this seems likely to be a hotspot)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((53 lines not shown))
+ */
+ def readLengthCodedBinary: Long = {
+ val firstByte = readByte
+ if(firstByte < 251)
+ firstByte
+ else
+ firstByte match {
+ case 252 => read(2)
+ case 253 => read(3)
+ case 254 => read(8)
+ case _ => -1 //NULL
+ }
+ }
+
+ def readNullTerminatedString: String = {
+ val result = new StringBuilder()

I'm surprised there is no reference to the charset here. I think this only works if the server and client are both in the same one?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((70 lines not shown))
+ result += readByte.toChar
+
+ readByte //consume null byte
+ result.toString
+ }
+
+ def readLengthCodedString: String = {
+ val size = readUnsignedByte
+
+ if(size == 0xFB)
+ return "" // NULL string.
+
+ val strBytes = new Array[Byte](size)
+ Array.copy(buffer, offset, strBytes, 0, size)
+ offset += size
+ new String(strBytes)

This definitely needs a reference to the charset encoding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((98 lines not shown))
+ if(readable(4)) {
+ year = readUnsignedShort
+ month = readUnsignedByte
+ day = readUnsignedByte
+ }
+
+ if(readable(3)) {
+ hour = readUnsignedByte
+ min = readUnsignedByte
+ sec = readUnsignedByte
+ }
+
+ if(readable(4))
+ nano = readInt
+
+ val fmt = "%04d-%02d-%02d %02d:%02d:%02d"

String.format is very inefficient. I suggest using a StringBuilder.

Edit: On second thought you might use a Calendar object instead that doesn't require us to format and then parse a string to get a timestamp.

Ruben Oanta Collaborator
roanta added a note

I was wondering where all the set* methods went for Dates. I'll change these methods to use Calendar instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((120 lines not shown))
+
+ /**
+ * Read a MySQL binary encoded Time field from the buffer.
+ */
+ def readTime: Time = {
+ val len = readUnsignedByte
+ if(len == 0)
+ return new Time(0)
+
+ val sign = if(readByte == 1) -1 else 1
+ val days = readInt
+ val hour = readUnsignedByte
+ val min = readUnsignedByte
+ val sec = readUnsignedByte
+
+ Time.valueOf("%02d:%02d:%02d".format(hour, min, sec))

Again using Calendar here?

Davi Arnaut
darnaut added a note

Also, you need to convert days to hours. I guess using Calendar will solve.

Ruben Oanta Collaborator
roanta added a note

It seems like the SQL Time type is more accurately represented by a duration because time can also be negative. Unless I am missing something, the current Connector/J driver does not support negative Time. It will throw a SQLException if you try to getTime with a negative time value in the database because it uses java.sql.Time to represent SQL Time.

EDIT: java.sql.Time also doesn't support hour values greater than 24.

Is this behavior we want to mimic? Or are we actually interested in negative Time values?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((169 lines not shown))
+ def fill(n: Int, b: Byte) = {
+ (offset until offset + n) foreach { j => buffer(j) = b ; offset += 1 }
+ this
+ }
+ def fillRest(b: Byte) = fill(buffer.size - offset, b)
+
+ def writeNullTerminatedString(str: String) = {
+ Array.copy(str.getBytes, 0, buffer, offset, str.length)
+ buffer(offset + str.length) = '\0'.toByte
+ offset += str.length + 1
+ this
+ }
+
+ def writeLengthCodedString(str: String) = {
+ buffer(offset) = str.length.toByte
+ Array.copy(str.getBytes, 0, buffer, offset+1, str.length)

Need an explicit charset. I suggest that we require UTF-8 on both sides.

marius a. eriksen Collaborator

i'm sure the protocol has something to say about this, right? probably it's even configurable? :-(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...ain/scala/com/twitter/finagle/mysql/codec/MySQL.scala
((53 lines not shown))
+ self(conn) flatMap { service => service(greet) flatMap {
+ case sg: ServersGreeting if sg.serverCap.has(Capability.protocol41) =>
+ Future.value(sg)
+
+ case sg: ServersGreeting =>
+ Future.exception(IncompatibleServerVersion)
+ } flatMap { sg => service(makeLoginReq(sg)) flatMap {
+ case result: OK =>
+ Future.value(service)
+
+ case Error(c, s, m) =>
+ Future.exception(ServerError("Error when authenticating the client "+ c + " - " + m))
+ }
+ }
+ }
+ }
marius a. eriksen Collaborator

here it’s a little difficult to keep track of the scope of the futures and flatMaps. it might help to separate out concerns a little, and flatten things out eg.:

def acceptGreeting(res: Result) = sg match {
  case sg: ServersGreeting if sg.serverCap.has(Capability.protocol41) =>
    Future.value(())
  case sg: ServersGreeting =>
    Future.exception(Incomatib..)
  case v =>
    Future.exception(new Exception("invalid reply type %s".format(v.getClass.getName))
}

def acceptLogin(res: Result) = res match {
 ...
}

// Then the login sequence is very explicit, and flattened by using a
// for comprehension.
def apply(conn: ClientConnection) = for {
  service <- self(conn)
  sg <- service(greet)
  _ <- acceptGreeting(sg)
  res <- service(makeLoginReq(sg))
  _ <- acceptLogin(res)
} yield service
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...cala/com/twitter/finagle/mysql/protocol/Request.scala
((93 lines not shown))
+ case t: Timestamp => setType(Types.TIMESTAMP)
+ case d: Date => setType(Types.DATE)
+ case b: Byte => setType(Types.TINY)
+ case s: Short => setType(Types.SHORT)
+ case i: Int => setType(Types.LONG)
+ case l: Long => setType(Types.LONGLONG)
+ case f: Float => setType(Types.FLOAT)
+ case d: Double => setType(Types.DOUBLE)
+ case NullValue(code) => setType(code)
+ case null => setType(Types.NULL)
+ case _ => throw new IllegalArgumentException("Unhandled query parameter type for " +
+ param + " type " + param.asInstanceOf[Object].getClass.getName)
+ }
+ }
+
+ private def convertToBinary(parameters: List[Any]): Array[Byte] = {

Making a new buffer, and a new writer per parameter seems pretty expensive. Can we refactor this to use a single buffer? Also, it generates multiple copies per parameter.

Ruben Oanta Collaborator
roanta added a note

Yes, this can be solved by precomputing the size the parameters will take and using 1 buffer writer. This would eliminate the copying and the object creation.

In retrospect, I am not sure how I reasoned that all this object creation was okay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((1 lines not shown))
+package com.twitter.finagle.mysql.codec
+
+import com.twitter.finagle.mysql.protocol.{Packet, BufferReader}
+import com.twitter.finagle.mysql.util.BufferUtil
+import org.jboss.netty.buffer.ChannelBuffer
+import org.jboss.netty.channel.{Channel, ChannelHandlerContext}
+import org.jboss.netty.handler.codec.frame.FrameDecoder
+
+/**
+ * MySQL packets are a length encoded set of bytes written
+ * in little endian byte order.
+ *
+ * The built in LengthFieldBasedFrameDecoder in Netty
+ * doesn't seem to support byte buffers that are encoded in
+ * little endian. Thus, a simple custom FrameDecoder is
+ * needed to defrag a ChannelBuffer into a logical MySQL packet.
marius a. eriksen Collaborator

It does: Endianess is Netty (and in Java generally) is handled by the underlying ChannelBuffer. So to do this “properly” you could either demand a little endian ChannelBuffer factory, or override the LengthFieldBasedFrameDecoder and wrap the incoming buffer with one that is little endian.

However, this is so trivial, that it’s both simpler and clearer to do it “manually” as you have done.

But remove the comment I think :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((5 lines not shown))
+import org.jboss.netty.buffer.ChannelBuffer
+import org.jboss.netty.channel.{Channel, ChannelHandlerContext}
+import org.jboss.netty.handler.codec.frame.FrameDecoder
+
+/**
+ * MySQL packets are a length encoded set of bytes written
+ * in little endian byte order.
+ *
+ * The built in LengthFieldBasedFrameDecoder in Netty
+ * doesn't seem to support byte buffers that are encoded in
+ * little endian. Thus, a simple custom FrameDecoder is
+ * needed to defrag a ChannelBuffer into a logical MySQL packet.
+ */
+class PacketFrameDecoder extends FrameDecoder {
+ override def decode(ctx: ChannelHandlerContext, channel: Channel, buffer: ChannelBuffer): Packet = {
+ if(buffer.readableBytes < Packet.headerSize)
marius a. eriksen Collaborator

whitespace: if (

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((16 lines not shown))
+ * needed to defrag a ChannelBuffer into a logical MySQL packet.
+ */
+class PacketFrameDecoder extends FrameDecoder {
+ override def decode(ctx: ChannelHandlerContext, channel: Channel, buffer: ChannelBuffer): Packet = {
+ if(buffer.readableBytes < Packet.headerSize)
+ return null
+
+ val header = new Array[Byte](Packet.headerSize)
+ buffer.readBytes(header)
+ val br = new BufferReader(header)
+
+ val (length, seq) = (br.readInt24, br.readByte)
+
+ if(buffer.readableBytes < length)
+ return null
+
marius a. eriksen Collaborator

You'll need to save the reader index here so that it is restored next time. See the FrameDecoder example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((12 lines not shown))
+ *
+ * The built in LengthFieldBasedFrameDecoder in Netty
+ * doesn't seem to support byte buffers that are encoded in
+ * little endian. Thus, a simple custom FrameDecoder is
+ * needed to defrag a ChannelBuffer into a logical MySQL packet.
+ */
+class PacketFrameDecoder extends FrameDecoder {
+ override def decode(ctx: ChannelHandlerContext, channel: Channel, buffer: ChannelBuffer): Packet = {
+ if(buffer.readableBytes < Packet.headerSize)
+ return null
+
+ val header = new Array[Byte](Packet.headerSize)
+ buffer.readBytes(header)
+ val br = new BufferReader(header)
+
+ val (length, seq) = (br.readInt24, br.readByte)
marius a. eriksen Collaborator

When tuples aren't natural, use sequence the statements instead. While correct, it isn't obvious to the reader that the order is br.readInt24, br.readByte).

Also, since these are side-effecting, they should be furnished with ()'s:

val length = br.readInt24()
val seq = br.readByte()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../twitter/finagle/mysql/codec/PacketFrameDecoder.scala
((20 lines not shown))
+ if(buffer.readableBytes < Packet.headerSize)
+ return null
+
+ val header = new Array[Byte](Packet.headerSize)
+ buffer.readBytes(header)
+ val br = new BufferReader(header)
+
+ val (length, seq) = (br.readInt24, br.readByte)
+
+ if(buffer.readableBytes < length)
+ return null
+
+ println("<- Decoding MySQL packet (length=%d, seq=%d)".format(length, seq))
+ val body = new Array[Byte](length)
+ buffer.readBytes(body)
+ BufferUtil.hex(body)
marius a. eriksen Collaborator

spurious statement?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../com/twitter/finagle/mysql/codec/RequestEncoder.scala
((2 lines not shown))
+
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.BufferUtil
+import org.jboss.netty.buffer.{ChannelBuffers, ChannelBuffer}
+import org.jboss.netty.channel.{Channel, ChannelHandlerContext, MessageEvent, Channels, ChannelEvent}
+import org.jboss.netty.handler.codec.oneone.OneToOneEncoder
+
+object RequestEncoder extends OneToOneEncoder {
+ override def encode(context: ChannelHandlerContext, channel: Channel, message: AnyRef) = {
+ message match {
+ case req: SimpleRequest if req.cmd == Command.COM_NOOP_GREET =>
+ ChannelBuffers.EMPTY_BUFFER
+ case req: Request =>
+ println("-> Encoding " + req)
+ val bytes = req.toByteArray
+ BufferUtil.hex(bytes)
marius a. eriksen Collaborator

spurious?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../com/twitter/finagle/mysql/codec/RequestEncoder.scala
((1 lines not shown))
+package com.twitter.finagle.mysql.codec
+
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.BufferUtil
+import org.jboss.netty.buffer.{ChannelBuffers, ChannelBuffer}
+import org.jboss.netty.channel.{Channel, ChannelHandlerContext, MessageEvent, Channels, ChannelEvent}
+import org.jboss.netty.handler.codec.oneone.OneToOneEncoder
+
+object RequestEncoder extends OneToOneEncoder {
+ override def encode(context: ChannelHandlerContext, channel: Channel, message: AnyRef) = {
+ message match {
+ case req: SimpleRequest if req.cmd == Command.COM_NOOP_GREET =>
+ ChannelBuffers.EMPTY_BUFFER
+ case req: Request =>
+ println("-> Encoding " + req)
+ val bytes = req.toByteArray
marius a. eriksen Collaborator

define toChannelBuffer instead, which would allow the implementation to avoid extra copies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
@@ -0,0 +1,168 @@
+package com.twitter.finagle.mysql.codec
+
+import com.twitter.finagle.mysql.ClientError
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.logging.Logger
+import org.jboss.netty.buffer.ChannelBuffer
+import org.jboss.netty.channel._
+
+sealed abstract class State
marius a. eriksen Collaborator

idiomatic to use traits: sealed trait State

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...cala/com/twitter/finagle/mysql/protocol/Request.scala
((133 lines not shown))
+
+ convert(parameters, Array[Byte]())
+ }
+
+ override val data: Array[Byte] = {
+ val cmdByte = Array(Command.COM_STMT_EXECUTE)
+ val bw = new BufferWriter(new Array[Byte](9))
+ bw.writeInt(ps.statementId)
+ bw.writeByte(flags)
+ bw.writeInt(iterationCount)
+
+ val paramsList = ps.parameters.toList
+ val nullBytes = makeNullBitmap(paramsList, 0, BigInt(0))
+ val newParamsBound: Byte = if(ps.hasNewParameters) 1 else 0
+
+ val result = Array.concat(cmdByte, bw.buffer, nullBytes, Array(newParamsBound))

More copies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Result.scala
((31 lines not shown))
+ }
+}
+
+/**
+ * Represents the Error Packet received from the server
+ * and the data sent along with it.
+ */
+case class Error(code: Short, sqlState: String, message: String) extends Result
+
+object Error {
+ def decode(packet: Packet) = {
+ //start reading after flag byte
+ val br = new BufferReader(packet.body, 1)
+ val code = br.readShort
+ val state = new String(br.take(6))
+ val msg = new String(br.takeRest)

I'm not sure if we can assume that error codes and messages are in ASCII.

Davi Arnaut
darnaut added a note

It uses the same character set as query results (character_set_results).

Ruben Oanta Collaborator
roanta added a note

Is this also the same as server_language sent during handshaking?

Davi Arnaut
darnaut added a note

The variable is initialized to the charset number sent by the client.

See Connection Character Sets and Collations and Character Set for Error Messages for more details.

Davi Arnaut
darnaut added a note

As suggested above, better to enforce UTF-8 both ways.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
((24 lines not shown))
+ * There are specific packets received from MySQL that can
+ * be easily decoded based on their first byte. However, more complex
+ * results need to be defragged as they arrive in the pipeline.
+ * To accomplish this, this handler needs to contain some state.
+ *
+ * Some of state is synchronized because it is shared between handleDownstream
+ * and handleUpstream events which are usually executed
+ * on separate threads.
+ */
+class ResultDecoder extends SimpleChannelHandler {
+ private val log = Logger("finagle-mysql")
+ private var state: State = WaitingForGreeting
+ @volatile
+ private var expectPrepareOK: Boolean = false
+ @volatile
+ private var expectBinaryResults: Boolean = false
marius a. eriksen Collaborator

volatiles on same line: @volatile private[this] var ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
((62 lines not shown))
+ }
+
+ override def writeRequested(ctx: ChannelHandlerContext, evt: MessageEvent): Unit = {
+ if(!evt.getMessage.isInstanceOf[ChannelBuffer]) {
+ ctx.sendDownstream(evt)
+ return
+ }
+
+ val buffer = evt.getMessage.asInstanceOf[ChannelBuffer]
+ if(buffer.capacity < 5) {
+ ctx.sendDownstream(evt)
+ return
+ }
+
+ //Do we need to block requests over the same
+ //pipeline when we are defragging a result?
marius a. eriksen Collaborator

whitespace

// Do ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
((56 lines not shown))
+ result map { Channels.fireMessageReceived(ctx, _) }
+
+ case unknown =>
+ Channels.disconnect(ctx.getChannel)
+ log.error("ResultDecoder: Expected packet and received: " + unknown)
+ }
+ }
+
+ override def writeRequested(ctx: ChannelHandlerContext, evt: MessageEvent): Unit = {
+ if(!evt.getMessage.isInstanceOf[ChannelBuffer]) {
+ ctx.sendDownstream(evt)
+ return
+ }
+
+ val buffer = evt.getMessage.asInstanceOf[ChannelBuffer]
+ if(buffer.capacity < 5) {
marius a. eriksen Collaborator

this is mysterious to the reader: comment on this behavior. also you probably did mean: buffer.readableBytes? are there valid packets that are fewer than 5 bytes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...main/scala/com/twitter/finagle/mysql/util/Query.scala
((41 lines not shown))
+ def flatten(params: Seq[Any]): Seq[Any] = params flatMap {
+ case p: Product => flatten(p.productIterator.toSeq)
+ case i: Iterable[_] => flatten(i.toSeq)
+ case x => Seq(x)
+ }
+
+ /**
+ * Replace each wildcard instance in sql with the output from
+ * running f(param(i))
+ */
+ private def replace(sql: String, params: Seq[Any], f: Any => String): String = {
+ if(params.isEmpty) {
+ return sql
+ }
+
+ val matcher = Query.wildcard.matcher(sql)

Does this regular expression work if I have ? within quotes? I also don't know why we are doing the replacement rather than leaving it to the server? This seems pretty dangerous.

Ruben Oanta Collaborator
roanta added a note

For prepared queries, the '?' are just expanded and the replacement happens on the server. This allows for calling prepare("select * from table where id in (?)", 1 to 10) as opposed to prepare("select * from table where id in (?,?,?,?...", 1 to 10)

For regular queries, the parameters have to be replaced. This also allows for cleaner client apis that don't require StringBuilders and String concatenation to build a sql statement.

I would like this to not be in the API at all. It is very difficult to get this right such that you don't introduce SQL injection attacks like the current implementation allows.

Ruben Oanta Collaborator
roanta added a note

As in, a rogue library that has a malicious toString method?

We can completely remove it but the problem of sql injection is still there if the a user builds their own sql statement.

Ruben Oanta Collaborator
roanta added a note

That makes sense. I agree it would need more work to be more robust and safe. But if you don't think it is worth having we can scrap it and let the user explicitly build the queries.

marius a. eriksen Collaborator

I agree with Sam -- let's not include this in the API. Let's make it pure SQL. There are many other approaches to building queries, including DSLs that exploit type safety -- but those can and should be built on top of finagle-mysql

Not totally sure I'm following the above conversation, but if we're to use this in production it will need to support some form of prepared statement substitution. Server-side prepared statements are slower for many use cases and much more difficult to resource-manage. It's quite easy to accidentally run the server out of prepared statement handles and then everybody loses. I generally recommend with MySQL Connector/J JDBC driver to always set useServerPrepStmts=false, causing the client-side to always do the substitution.

It sounds to me then that we need a well tested way of doing prepared statement encoding client side. In order to support this, the MySQL driver appears to have a SQL parser along with all the escaping rules once the context of the ? is well understood through parsing.

http://www.docjar.com/html/api/com/mysql/jdbc/PreparedStatement.java.html

@jeremycole The other possibility is more carefully monitoring server side prepared statement usage. Which one do you think is more doable?

Marius' idea of using a DSL to generate them might also serve the same purpose without having to parse statements but you would still need to do proper escaping of values.

In general I would say to avoid using server-side prepared statements at all. Even if they are perfectly monitored, they are still in very many cases generating 3 round trips (prepare, execute, close) when 1 would suffice (query). In most of our apps that we care about, there would be huge numbers of prepared statements in order to pre-prepare them all (as table name is not an allowed placeholder, and they are mostly sharded into N tables); so the round trip multiplication can't be avoided by pre-preparing all possible statements.

So since a good client-side prepared statement handler is necessary, may as well use it everywhere, even when it's not strictly needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
((4 lines not shown))
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.logging.Logger
+import org.jboss.netty.buffer.ChannelBuffer
+import org.jboss.netty.channel._
+
+sealed abstract class State
+case object Idle extends State
+case object WaitingForGreeting extends State
+case class PacketDefragger(
+ firstPacket: Option[Packet] = None,
+ setOne: Option[List[Packet]] = None,
+ setTwo: Option[List[Packet]] = None,
+ setOneExpected: Boolean = true,
+ setTwoExpected: Boolean = true,
+ decoder: (Packet, List[Packet], List[Packet]) => Result
+ ) extends State
marius a. eriksen Collaborator

consistent naming (what is the decoder doing?): DefraggingPackets

marius a. eriksen Collaborator

I think this would be clearer if it were split into more states. Right now there seems to be a lot of mutually exclusive configurations of PacketDefragger. Here's a suggestion based on my very primitive understanding of the protocol: depending on the type of the result set, there may be one or two set of packets that are to be decoded. First simplification: I think the Option in the sets are redundant. It's only None whenever there is no data, so use Nil instead to represent this state? Second it seems that you expect a number of packet sets; 0, 1, or 2, and that their order, but not their position is relevant (this understanding may not be accurate). So it seems that what you really want to arrive at is a Seq[Seq[Packet]]

case class Defragging(
  expected: Int,
  packets: Seq[Seq[Packet]]
)

Then your transitions look like

Defragging(2, Nil)

after EOF for the first set

Defragging(1, Seq(Seq(first, set, of, packets)))

...

does that make sense? My intuition is that this approach would simplify greatly.

Ruben Oanta Collaborator
roanta added a note

I've implemented your suggestions. Removing the noise caused by wrapping the sets in Option[...] and not passing around the decoder in the Defragging class cleaned up the code a lot.

The changes actually reduced the amount of cases by 1, but I think it did result in more understandable code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sam Pullara

I would like to see benchmarks against the normal MySQL driver as part of this effort. For performance, memory usage and CPU usage.

...a/com/twitter/finagle/mysql/codec/ResultDecoder.scala
((74 lines not shown))
+ }
+
+ //Do we need to block requests over the same
+ //pipeline when we are defragging a result?
+ if(state != Idle) {
+ log.warning("Cannot process a writeRequest when ResultDecoder is not idle.")
+ return
+ }
+
+ //set flags that indicate expected results.
+ val cmdByte = buffer.getByte(4)
+ expectPrepareOK = (cmdByte == Command.COM_STMT_PREPARE)
+ expectBinaryResults = (cmdByte == Command.COM_STMT_EXECUTE)
+
+ ctx.sendDownstream(evt)
+ }
marius a. eriksen Collaborator

This suggests that the decoder is not separable from the encoder. Perhaps you should just conflate them in the code? It would avoid this extra decoding step and make the code clearer, I think.

Ruben Oanta Collaborator
roanta added a note

In some cases, they are not separable. I don't think it would be a problem to conflate them especially since the encoder is so simple.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((24 lines not shown))
+ val n = (offset until offset + width).zipWithIndex.foldLeft(0L) {
+ case (result, (b,i)) => result | ((buffer(b) & 0xFFL) << (i*8))
+ }
+ offset += width
+ n
+ }
+
+ def readByte = read(1).toByte
+ def readUnsignedByte = read(1).toShort
+ def readShort = read(2).toShort
+ def readUnsignedShort = read(2).toInt
+ def readInt24 = read(3).toInt
+ def readInt = read(4).toInt
+ def readLong = read(8)
+ def readFloat = JFloat.intBitsToFloat(readInt)
+ def readDouble = JDouble.longBitsToDouble(readLong)
marius a. eriksen Collaborator

these should all have ()s since they modify state (offset)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
@@ -0,0 +1,188 @@
+package com.twitter.finagle.mysql.protocol
+
+import java.lang.{Float => JFloat, Double => JDouble}
+import java.sql.{Time, Date, Timestamp}
+
+/**
+ * Defines classes to read and write to/from a byte buffer
+ * in little endian byte order.
+ */
+
+class BufferReader(val buffer: Array[Byte], private[this] var offset: Int = 0) {
marius a. eriksen Collaborator

(for future performance optimization) - we should try to use ChannelBuffers as much as possible throughout as it allows us to reduce the amount of copying we do.

Ruben Oanta Collaborator
roanta added a note

Agreed. I visited the idea of having this just wrap a netty ChannelBuffer, but I remember running into a lot of issues. It could have been my lack of experience of working with netty Buffers / nio Buffers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...a/com/twitter/finagle/mysql/protocol/Capability.scala
((5 lines not shown))
+ val foundRows = 0x0002 /* Found instead of affected rows */
+ val longFlag = 0x0004 /* Get all column flags */
+ val connectWithDB = 0x0008 /* One can specify db on connect */
+ val noSchema = 0x0010 /* Don't allow database.table.column */
+ val compress = 0x0020 /* Can use compression protocol */
+ val ODBC = 0x0040 /* Odbc client */
+ val localFiles = 0x0080 /* Can use LOAD DATA LOCAL */
+ val ignoreSpace = 0x0100 /* Ignore spaces before '(' */
+ val protocol41 = 0x0200 /* New 4.1 protocol */
+ val interactive = 0x0400 /* This is an interactive client */
+ val ssl = 0x0800 /* Switch to SSL after handshake */
+ val ignoreSigPipe = 0x1000 /* IGNORE sigpipes */
+ val transactions = 0x2000 /* Client knows about transactions */
+ val secureConnection = 0x8000 /* New 4.1 authentication */
+ val multiStatements = 0x10000 /* Enable/disable multi-stmt support */
+ val multiResults = 0x20000 /* Enable/disable multi-results */
marius a. eriksen Collaborator

Constants should start with UpperCase -- this is actually not only a matter of style, but the pattern matching does not bind uppercase names, so it allows you to match against constants without further ado. And I also believe the optimizer only folds UppercaseConstants.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...la/com/twitter/finagle/mysql/protocol/Handshake.scala
((35 lines not shown))
+ threadId,
+ Array.concat(salt1, salt2),
+ serverCap,
+ language,
+ status
+ )
+ }
+}
+
+/**
+ * Reply to ServerGreeting sent during handshaking phase.
+ */
+case class LoginRequest(
+ clientCap: Capability = Capability(0xA68F),
+ maxPacket: Int = 0x10000000,
+ charset: Byte = 33.toByte, // TODO case class
marius a. eriksen Collaborator

comment on what this constant is, or better yet, name it.

It's also worth noting that MySQL conflates character sets and collations at the protocol level, so 33 is "utf8_general_ci" collation, and you could see many other values here for utf8 character set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Packet.scala
@@ -0,0 +1,40 @@
+package com.twitter.finagle.mysql.protocol
+
+/**
+ * Represents a logical packet received from MySQL.
+ * A MySQL packet consists of a header,
+ * 3-bytes containing the size of the body and a
+ * 1 byte seq number, followed by the body.
+ */
+
+object Packet {
+ val headerSize = 0x04
+ val okByte = 0x00.toByte
+ val errorByte = 0xFF.toByte
+ val eofByte = 0xFE.toByte
marius a. eriksen Collaborator

constants are Capitalized

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Packet.scala
((17 lines not shown))
+ val size = packetSize
+ val number = seq
+ val body = new Array[Byte](size)
+ }
+
+ def apply(packetSize: Int, seq: Byte, data: Array[Byte]) = new Packet {
+ val size = packetSize
+ val number = seq
+ val body = data
+ }
+}
+
+trait Packet {
+ val size: Int
+ val number: Byte //used for sanity checks on server side
+ val body: Array[Byte]
marius a. eriksen Collaborator

any reason to not use a case class?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...witter/finagle/mysql/protocol/PreparedStatement.scala
@@ -0,0 +1,61 @@
+package com.twitter.finagle.mysql.protocol
+
+import com.twitter.util.Promise
+
+trait PreparedStatement extends Result {
+ val statementId: Int
+ val numberOfParams: Int
+ val statement: Promise[String] = new Promise[String]()
marius a. eriksen Collaborator

case class?

Ruben Oanta Collaborator
roanta added a note

I wanted to leave room for different implementations.

marius a. eriksen Collaborator

then we can make room when they come :)

Ruben Oanta Collaborator
roanta added a note

case class it is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Result.scala
((9 lines not shown))
+ * - COM_PING
+ * - COM_QUERY (INSERT, UPDATE, or ALTER TABLE)
+ * - COM_REFRESH
+ * - COM_REGISTER_SLAVE
+ */
+case class OK(affectedRows: Long,
+ insertId: Long,
+ serverStatus: Int,
+ warningCount: Int,
+ message: String) extends Result
+
+object OK {
+ def decode(packet: Packet) = {
+ //start reading after flag byte
+ val br = new BufferReader(packet.body, 1)
+ new OK(
marius a. eriksen Collaborator

`new' not needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen
Collaborator

Some general comments from my review:

  • try to use ChannelBuffers throughout as they make it easier to optimize away allocations &c.; especially for down the road
  • make sure you test the frame decoder for fragmented packets
  • it would be great to have package docs that give a brief overview of the protocol, the structure of the code, and links to protocol documentation
  • apply whitespace consistently throughout: if (, // a comment

Not for now, but possibly a simplification in the future: it would also be possible to use a Dispatcher to do packet defragmentation. In some sense, it's a more natural abstraction: the pipeline codec produces and consumes Packets, and is stateless. The dispatcher consumes and produces requests/responses, and is stateful. Besides a cleaner separation of responsibilities, you also have the full power of composable futures at hand, which makes it possible to write imperative-looking code. You could do stuff like:

readPacket() flatMap { packet =>
  if (fragmented(packet))
    defrag(packet)
  else
    decode(packet)
}

etc.

...la/com/twitter/finagle/mysql/protocol/Handshake.scala
((33 lines not shown))
+ protocol,
+ version,
+ threadId,
+ Array.concat(salt1, salt2),
+ serverCap,
+ language,
+ status
+ )
+ }
+}
+
+/**
+ * Reply to ServerGreeting sent during handshaking phase.
+ */
+case class LoginRequest(
+ clientCap: Capability = Capability(0xA68F),
Davi Arnaut
darnaut added a note

Would be better to refer to capabilities by name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...src/main/scala/com/twitter/finagle/mysql/Client.scala
((30 lines not shown))
+ .hostConnectionLimit(1)
+ .buildFactory()
+
+ apply(factory)
+ }
+
+ def apply(host: String, username: String, password: String): Client = {
+ apply(host, username, password, None)
+ }
+
+ def apply(host: String, username: String, password: String, dbname: String): Client = {
+ apply(host, username, password, Some(dbname))
+ }
+
+ class Client(factory: ServiceFactory[Request, Result]) {
+ private lazy val fService = factory.apply()
Steve Gury Collaborator

private[this] val fService = factory.apply()
2 comments:

  • use private[this] by default
  • The lazy feature here is not very useful
Ruben Oanta Collaborator
roanta added a note

Why is it so important to make private members private to the instance?

Also, my reasoning for the lazy val here was allowing the creating of a client without doing the connection leg work until the user actually used the client. A replacement for having an explicit 'connect()' method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Steve Gury stevegury commented on the diff
project/Build.scala
@@ -235,6 +235,17 @@ object Finagle extends Build {
sourceDirectory <<= baseDirectory(_/"src29"),
libraryDependencies ++= Seq("junit" % "junit" % "4.8.1" % "test", util("hashing"))
).dependsOn(finagleCore)
+
+ lazy val finagleMySQL = Project(
+ id = "finagle-mysql",
+ base = file("finagle-mysql"),
+ settings = Project.defaultSettings ++
+ StandardProject.newSettings ++
+ sharedSettings
+ ).settings(
+ name := "finagle-mysql",
+ libraryDependencies ++= Seq(util("logging"))
+ ).dependsOn(finagleCore)
Steve Gury Collaborator

You should also create/update the pom.xml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...la/com/twitter/finagle/mysql/protocol/ResultSet.scala
((107 lines not shown))
+ val br = new BufferReader(row)
+ val values: IndexedSeq[String] = (0 until indexMap.size) map { _ => br.readLengthCodedString }
+
+ def findColumnIndex(name: String) = indexMap.get(name)
+
+ /**
+ * The readLengthCodedString method returns an empty string when
+ * a null value is returned from the database. This method handles the
+ * empty string and returns an Option[String] to avoid attempting to cast an
+ * empty string.
+ *
+ * This also has a nice side-effect of translating null values
+ * into Option values.
+ */
+ private def getValue(index: Int): Option[String] =
+ if(values(index) == "") None else Some(values(index))

Do I understand correctly that this conflates NULL and empty string by always translating empty string to None? If a legitimate empty string were returned would it result in None and thus be indistinguishable from NULL? That would not be a good thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((48 lines not shown))
+
+ /**
+ * Read MySQL data field - a variable length encoded binary.
+ * Depending on the first byte, read a different width from
+ * the data array.
+ */
+ def readLengthCodedBinary: Long = {
+ val firstByte = readByte
+ if(firstByte < 251)
+ firstByte
+ else
+ firstByte match {
+ case 252 => read(2)
+ case 253 => read(3)
+ case 254 => read(8)
+ case _ => -1 //NULL

This is not really correct. I would think it better to explicitly case 251 as SQL NULL, and throw an exception for 255, as it implies corrupted data or out of sync client/server, since it's not a valid value. Continuing your read after getting a 255 here would end up with an exception in some other unrelated place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((65 lines not shown))
+ }
+
+ def readNullTerminatedString: String = {
+ val result = new StringBuilder()
+ while(buffer(offset) != 0)
+ result += readByte.toChar
+
+ readByte //consume null byte
+ result.toString
+ }
+
+ def readLengthCodedString: String = {
+ val size = readUnsignedByte
+
+ if(size == 0xFB)
+ return "" // NULL string.

Blank string and SQL NULL should never be conflated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((61 lines not shown))
+ case 253 => read(3)
+ case 254 => read(8)
+ case _ => -1 //NULL
+ }
+ }
+
+ def readNullTerminatedString: String = {
+ val result = new StringBuilder()
+ while(buffer(offset) != 0)
+ result += readByte.toChar
+
+ readByte //consume null byte
+ result.toString
+ }
+
+ def readLengthCodedString: String = {

This seems to not actually implement the length-coded string. What am I missing?

Ruben Oanta Collaborator
roanta added a note

I read the spec wrong. This of course managed to work because I haven't dealt with string lengths that don't fit into a unsigned byte. I'll fix it to use the length coded binary.

Ruben Oanta Collaborator
roanta added a note

Is there an upper bound on the length of a string? Can I assume that the size will never need to be represented with 8 bytes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((83 lines not shown))
+ Array.copy(buffer, offset, strBytes, 0, size)
+ offset += size
+ new String(strBytes)
+ }
+
+ /**
+ * Read a MySQL binary encoded Timestamp from the buffer.
+ */
+ def readTimestamp: Timestamp = {
+ val len = readUnsignedByte
+ if(len == 0)
+ return new Timestamp(0)
+
+ var year, month, day, hour, min, sec, nano = 0
+
+ if(readable(4)) {

If it is not readable, then what? Certainly this needs an else and should probably throw an exception of some sort.

Ruben Oanta Collaborator
roanta added a note

I need to find a better way to represent corrupt data. From what I understand, it isn't idiomatic to throw exceptions that are not future encoded within Finagle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
...la/com/twitter/finagle/mysql/protocol/ResultSet.scala
@@ -0,0 +1,301 @@
+package com.twitter.finagle.mysql.protocol
+
+import com.twitter.logging.Logger
+import scala.math.BigInt
+import java.sql.{Timestamp, Date, Time}
+
+/**
+ * ResultSets are returned from the server for any
+ * query except for INSERT, UPDATE, or ALTER TABLE.

There are a lot of other SQL queries which don't return result sets. To avoid confusion it's probably better not to try to list them here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
.../scala/com/twitter/finagle/mysql/protocol/Types.scala
((29 lines not shown))
+ val STRING = 0xfe;
+ val GEOMETRY = 0xff;
+}
+
+sealed case class NullValue(typeCode: Int)
+object NullValues {
+ import Types._
+
+ val legalValues = Set(DECIMAL, TINY, SHORT, LONG, FLOAT, DOUBLE, NULL, TIMESTAMP, LONGLONG,
+ INT24, DATE, DATETIME, YEAR, NEWDATE, VARCHAR, BIT, NEWDECIMAL, ENUM,
+ SET, TINY_BLOB, MEDIUM_BLOB, LONG_BLOB, BLOB, VAR_STRING, STRING, GEOMETRY)
+
+ val NullString = NullValues(VARCHAR)
+ val NullInt = NullValues(LONG)
+ val NullDouble = NullValues(DOUBLE)
+ val NullBoolean = NullValues(BIT)

In terms of naming, this is not really correct. Type BIT represents a bitmap of 1 or more bits. It's really an integer type of varying length, and only in the minimal case of BIT/BIT(1) is it potentially a boolean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jeremy Cole jeremycole commented on the diff
...main/scala/com/twitter/finagle/mysql/util/Query.scala
@@ -0,0 +1,75 @@
+package com.twitter.finagle.mysql.util
+
+import java.util.regex.Pattern
+
+class TooFewQueryParametersException(t: Throwable) extends Exception(t)
+class TooManyQueryParametersException(t: Throwable) extends Exception(t)
+
+object Query {
+ val wildcard = Pattern.compile("\\?")

This is, of course, far too simple. I would argue that regular expression substitution is itself too complex to be part of this library though, and should be delegated to something else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
roanta added some commits
Ruben Oanta roanta Includes most of the changes suggested during the code review. c6f7c9e
Ruben Oanta roanta fixed multi-line comment spacing. d8c27ed
Ruben Oanta roanta Fixed the README to use markdown + some comment changes. 65c53d4
Ruben Oanta roanta Ensured client/server are using UTF8 b59501c
Ruben Oanta roanta added .md extension to README 9aff55c
Ruben Oanta roanta - Added the ability to read length coded bytes from the Buffer
- Other small changes and fixes.
c05ab0d
Ruben Oanta roanta - Changed the Row interface to use a Value ADT. This adds type safety…
… to results and more explicitly expresses undecoded/raw values.
632f301
Ruben Oanta roanta Fixed small mistakes in readLengthCodedBytes and readLengthCodedString cd5fd4d
Ruben Oanta roanta - Changed the BufferReader and BufferWriter classes to interfaces to …
…accommodate easily changing the implementation when profiling performance.

- Made some general memory optimizations to the BufferReader class.
- Removed complex decoding of types from the BufferReader/BufferWriter implementations. It seemed more correct to decode/encode only primitives on Buffers.
- Moved decoding more complex types onto the Types object.
- Other small changes and fixes.
5db3c76
Ruben Oanta roanta Changed tabs to spaces. bdac4aa
Ruben Oanta roanta Avoid extra copying in readLengthCodedString 2eac63b
Ruben Oanta roanta Updates to the README, Row interface, and Example. e5f0880
Ruben Oanta roanta - Changed default implementations of BufferReader and BufferWriter to…
… wrap Netty ChannelBuffers.

- Updated references in unit tests.
- Added reference to defaultCharset in LoginRequest.encryptPassword.
70774b9
Ruben Oanta roanta Cleaned up Buffer.scala 9943d26
Ruben Oanta roanta - new unit tests + an integration test
- moved codec entry point code into seperate methods to allow for easier testing without needing to mock netty objects.
- added a new example that shows a table create / insert / query.
- Cleaned up Buffer.scala
- Added a ping method to client that runs a COM_PING request on the server.
- Removed deprecated requests for COM_CREATE_DB and COM_DROP_DB
- Fixed bugs with Date / Datetime / Timestamp offsets.
- Fixed a bug with executing a prepared statement that had bound parameters.
- Other small fixes and changes.
a229f5b
Ruben Oanta roanta tabs -> spaces aaa5613
Ruben Oanta roanta - Added a synthetic response for CloseRequest which doesn't get a res…
…ponse from the server.

- Better handling for unsupported LONG_BLOBs
- Closed prepared statement in Example.scala
- style fixes and comment fixes.
935f86e
Ruben Oanta roanta Fixed 8 byte length coded binary test to reflect current implementation. 8dae5a2
Ruben Oanta roanta Removed onSuccess from insertResults. 6758be5
marius a. eriksen mariusae commented on the diff
finagle-mysql/pom.xml
@@ -0,0 +1,43 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-mysql</artifactId>
+ <packaging>jar</packaging>
+ <version>4.0.3-SNAPSHOT</version>
marius a. eriksen Collaborator

version needs update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
finagle-mysql/pom.xml
((14 lines not shown))
+ <properties>
+ <git.dir>${project.basedir}/../../.git</git.dir>
+ </properties>
+ <dependencies>
+ <!-- library dependencies -->
+ <!-- project dependencies -->
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-core</artifactId>
+ <version>4.0.3-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>util-logging</artifactId>
+ <version>4.0.1</version>
+ </dependency>
marius a. eriksen Collaborator

ditto for these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...src/main/scala/com/twitter/finagle/mysql/Client.scala
((61 lines not shown))
+ * in the ResultSet, call f on the row and return the results.
+ * @param sql A sql statement that returns a result set.
+ * @param f A function from ResultSet to any type T.
+ * @return a Future of Seq[T]
+ */
+ def select[T](sql: String)(f: Row => T): Future[Seq[T]] = query(sql) map {
+ case rs: ResultSet => rs.rows.map(f)
+ case ok: OK => Seq()
+ }
+
+ /**
+ * Sends a query to server to be prepared for execution.
+ * @param sql A query to be prepared on the server.
+ * @return PreparedStatement
+ */
+ def prepare(sql: String, params: Any*) = {
marius a. eriksen Collaborator

explicitly annotate return type here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...src/main/scala/com/twitter/finagle/mysql/Client.scala
((154 lines not shown))
+ * Close the ServiceFactory and its underlying resources.
+ */
+ def close() = factory.close()
+
+ /**
+ * Helper function to send requests to the ServiceFactory
+ * and handle Error responses from the server.
+ */
+ private[this] def send[T](r: Request)(handler: PartialFunction[Result, Future[T]]) =
+ fService flatMap { service =>
+ service(r) flatMap (handler orElse {
+ case Error(c, s, m) => Future.exception(ServerError(c + " - " + m))
+ case result => Future.exception(ClientError("Unhandled result from server: " + result))
+ })
+ }
+}
marius a. eriksen Collaborator

for all public interfaces here: include return types. it serves both as (in code) documentation and it also fixes the types, which is important; otherwise refinement types may be exposed unintentionally, causing API conflicts later:

http://twitter.github.com/effectivescala/#Types%20and%20Generics-Return%20type%20annotations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...rc/main/scala/com/twitter/finagle/mysql/Example.scala
((126 lines not shown))
+ preparedFuture.isReturn && insertResults.forall(_.isReturn)
+ }
+
+ def parseArgs(parsed: Map[String, Any], args: List[String]): Map[String, Any] = args match {
+ case Nil => parsed
+ case "-host" :: value :: tail =>
+ parseArgs(parsed + ("host" -> value), tail)
+ case "-port" :: value :: tail =>
+ parseArgs(parsed + ("port" -> value.toInt), tail)
+ case "-u" :: value :: tail =>
+ parseArgs(parsed + ("username" -> value), tail)
+ case "-p" :: value :: tail =>
+ parseArgs(parsed + ("password" -> value), tail)
+ case unknown :: tail => parsed
+ }
+}
marius a. eriksen Collaborator

love this. we'll move it under finagle-example when it's ready for prime time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...ain/scala/com/twitter/finagle/mysql/codec/Endec.scala
((162 lines not shown))
+ None
+
+ // second set complete - no sets can follow.
+ case (Defragging(2, Seq(header, xs, ys)), Packet.EofByte) =>
+ transition(Idle)
+ Some(defragDecoder(header(0), xs.reverse, ys.reverse))
+
+ // prepend onto second set
+ case (Defragging(2, Seq(header, xs, ys)), _) =>
+ transition(Defragging(2, Seq(header, xs, packet +: ys)))
+ None
+
+ case _ =>
+ throw new ClientError("Endec: Unexpected state when defragmenting packets.")
+ }
+}
marius a. eriksen Collaborator

very nice. FOR LATER: i think we could improve this a bit further by embedding expectOK and defragDecoder into the state as well; but let’s leave it as-is for now. (I find it much easier to understand than the previous version)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...scala/com/twitter/finagle/mysql/protocol/Buffer.scala
((2 lines not shown))
+
+import com.twitter.finagle.mysql.ClientError
+import org.jboss.netty.buffer.ChannelBuffer
+import org.jboss.netty.buffer.ChannelBuffers._
+import java.nio.charset.{Charset => JCharset}
+import java.nio.ByteOrder
+
+/**
+ * The BufferReader and BufferWriter interfaces provide methods for
+ * reading/writing primitive data types exchanged between the client/server.
+ * This includes all primitive numeric types and strings (null-terminated and length coded).
+ * All Buffer methods are side-effecting. That is, each call to a read* or write*
+ * method will increase the current offset.
+ *
+ * Both BufferReader and BufferWriter assume bytes are written
+ * in little endian. This conforms with the MySQL protocol.
marius a. eriksen Collaborator

The reader will be curious as to why you wouldn’t just use a ChannelBuffer (why didn’t you?)

Ruben Oanta Collaborator
roanta added a note

There are several reason why I thought it would be better to wrap ChannelBuffer and provide a separate interface.

  1. Each time a user intends to read from a MySQL packet, a ChannelBuffer needs to be created with the correct ByteOrder. It just seemed more naturally to offer an interface specific to the codec that assures this.

  2. There are specific methods that a ChannelBuffer doesn't offer and are at the core of the protocol (readLengthCodedString/Bytes and writeLengthCodedString/Bytes). This could have easily been offered as a method in a an object ex. Buffer.readLengthCodedString(c: ChannelBuffer). Again, I thought it would be more naturally to provide this as part of an interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...witter/finagle/mysql/protocol/PreparedStatement.scala
@@ -0,0 +1,74 @@
+package com.twitter.finagle.mysql.protocol
+
+import com.twitter.util.Promise
+
+case class PreparedStatement(statementId: Int, numberOfParams: Int) extends Result {
+ val statement: Promise[String] = new Promise[String]()
+ private[this] var params: Array[Any] = new Array[Any](numberOfParams)
+ private[this] var hasNewParams: Boolean = false
+
+ def parameters: Array[Any] = params
+ def hasNewParameters: Boolean = hasNewParams
+
+ def bindParameters() = hasNewParams = false
marius a. eriksen Collaborator

for side effecting methods like this, convention dicatates the use of {}:

def bindParameters() { hasNewParams = false }

otherwise it’s difficult to discern whether you mean to do do the assignment, or if it’s a bug and you instead meant:

def bindParameters() = hasNewParams == false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...la/com/twitter/finagle/mysql/protocol/ResultSet.scala
((64 lines not shown))
+ */
+ def indexOf(columnName: String): Option[Int]
+
+ /**
+ * Retrieves the Value in the column with the
+ * given name.
+ * @param columnName name of the column.
+ * @return Some(Value) if the column
+ * exists with the given name. Otherwise, None.
+ */
+ def valueOf(columnName: String): Option[Value] =
+ valueOf(indexOf(columnName))
+
+ protected def valueOf(columnIndex: Option[Int]): Option[Value] =
+ for (idx <- columnIndex) yield values(idx)
+}
marius a. eriksen Collaborator

it might make sense to name "valueOf" -> "apply".

by convention in scala, maps use that for their keys, then you can access columns thus:

val row: Row
row("theColumn")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen mariusae commented on the diff
...main/scala/com/twitter/finagle/mysql/util/Query.scala
((37 lines not shown))
+ while (matcher.find) {
+ try {
+ matcher.appendReplacement(result, expand(params(i)))
+ } catch {
+ case e: ArrayIndexOutOfBoundsException =>
+ throw new TooFewQueryParametersException(e)
+ case e: NoSuchElementException =>
+ throw new TooFewQueryParametersException(e)
+ }
+ i += 1
+ }
+
+ matcher.appendTail(result)
+ result.toString
+ }
+}
marius a. eriksen Collaborator

since this parameter expansion technique is currently unsafe; can we drop this feature alltogether until we figure out a better way? we could simply require the user to provide fully expanded parameter lists for now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
marius a. eriksen
Collaborator

I’ve pulled this internally; should show up here soon (probably Monday). Not yet published.

marius a. eriksen mariusae closed this
suncelesta suncelesta referenced this pull request from a commit in suncelesta/suncelesta.github.com
Ruben Oanta [split] MySQL codec from Ruben Oanta.
Github-pull-request: twitter/finagle#98
Signed-off-by: marius a. eriksen <marius@twitter.com>

RB_ID=80076
2adabf7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 24, 2012
  1. Ruben Oanta
  2. Ruben Oanta
Commits on May 29, 2012
  1. Ruben Oanta

    Removed unused dependency.

    roanta authored
  2. Ruben Oanta

    Refactored a bit

    roanta authored
Commits on Jun 13, 2012
  1. Ruben Oanta

    Includes the start of a test suite, some refactoring, and bug fixes. …

    roanta authored
    …The most significant change
    
    includes removing authentication from the codec pipeline and into a ServiceFactoryProxy on
    the codec. This more elegantly ensures (through sequential composition) that authentication happens
    in the correct order.
  2. Ruben Oanta

    removed unused cached files

    roanta authored
Commits on Jun 19, 2012
  1. Ruben Oanta

    Moved common operations used to decode (read) and encode (write) to a…

    roanta authored
    … byte array into classes BufferReader and BufferWriter, respectively. This removed a lot of code duplication in handling the requests and responses from MySQL.
Commits on Jun 20, 2012
  1. Ruben Oanta

    -Fixed an issue when decoding the packet header in the Decoder that w…

    roanta authored
    …as caused by not masking each byte with 255 and resulted in negative size values.
    
    -Changed the client capabilities to include connectWithDB. Added a check to ensure the server can accept a schema on client connect.
    -Added a more extensive decode method for EOF packets.
    -Corrected an issue with the take method not correctly incrementing the offset in BufferReader.
Commits on Jun 24, 2012
  1. Ruben Oanta

    -Added the ability to read an unsigned byte and short in BufferReader.

    roanta authored
    -Fixed an issue where the Field result was being improperly decoded.
Commits on Jul 3, 2012
  1. -Changed the codec to support more complex results. Decoupled the pac…

    Ruben Oanta authored
    …ket defragmenter from
    
    the primary decoder.
    -Started implementing PreparedStatements on the codec.
    -Added the ability to easily add/remove from a Capability bit mask.
    -Fixed an issue where the LoginRequest client capability was improperly set when the database argument was None.
Commits on Jul 5, 2012
  1. Ruben Oanta

    -Added decoding of prepared statements in ResultDecoder.

    roanta authored
    -Fixed readLengthCodedBinary to work properly.
Commits on Jul 9, 2012
  1. Ruben Oanta

    -Expanded the ResultSet class to be more useful

    roanta authored
    -Added some basic methods to the Client that use the new ResultSet capabilities.
    -Many other small additions, fixes, and changes.
  2. Ruben Oanta

    Changed tabs to spaces

    roanta authored
Commits on Jul 20, 2012
  1. Implemented an initial api that supports basic queries and prepared

    Ruben Oanta authored
    statements.
  2. Ruben Oanta

    -Changed the return type of prepareAndSelect of a Future[(PreparedSta…

    roanta authored
    …tement, Seq[T]] so that the prepared statement is still accesible after it is executed.
    
    -Added a bindParameters() method to PreparedStatements to indicate that the parameters have been sent to the server.
    -Added support for Time, Date, Datetime, and Timestamp fields. Still need to add support for sending those types to the mysql server.
Commits on Jul 22, 2012
  1. -Added writeTimestamp and writeDate to Buffer.scala

    Ruben Oanta authored
    -Added id method to field that uses name or origName depending on which one is present.
    -Simplified the query method in Client.
Commits on Jul 23, 2012
  1. Added better error handling for Query.* methods.

    Ruben Oanta authored
Commits on Jul 27, 2012
  1. Ruben Oanta
Commits on Jul 28, 2012
  1. Ruben Oanta
  2. Ruben Oanta
  3. Ruben Oanta
Commits on Jul 29, 2012
  1. Ruben Oanta

    added .md extension to README

    roanta authored
Commits on Jul 31, 2012
  1. Ruben Oanta

    - Added the ability to read length coded bytes from the Buffer

    roanta authored
    - Other small changes and fixes.
  2. Ruben Oanta

    - Changed the Row interface to use a Value ADT. This adds type safety…

    roanta authored
    … to results and more explicitly expresses undecoded/raw values.
  3. Ruben Oanta
  4. Ruben Oanta

    - Changed the BufferReader and BufferWriter classes to interfaces to …

    roanta authored
    …accommodate easily changing the implementation when profiling performance.
    
    - Made some general memory optimizations to the BufferReader class.
    - Removed complex decoding of types from the BufferReader/BufferWriter implementations. It seemed more correct to decode/encode only primitives on Buffers.
    - Moved decoding more complex types onto the Types object.
    - Other small changes and fixes.
  5. Ruben Oanta

    Changed tabs to spaces.

    roanta authored
Commits on Aug 1, 2012
  1. Ruben Oanta
  2. Ruben Oanta
  3. Ruben Oanta

    - Changed default implementations of BufferReader and BufferWriter to…

    roanta authored
    … wrap Netty ChannelBuffers.
    
    - Updated references in unit tests.
    - Added reference to defaultCharset in LoginRequest.encryptPassword.
Commits on Aug 2, 2012
  1. Ruben Oanta

    Cleaned up Buffer.scala

    roanta authored
Commits on Aug 6, 2012
  1. Ruben Oanta

    - new unit tests + an integration test

    roanta authored
    - moved codec entry point code into seperate methods to allow for easier testing without needing to mock netty objects.
    - added a new example that shows a table create / insert / query.
    - Cleaned up Buffer.scala
    - Added a ping method to client that runs a COM_PING request on the server.
    - Removed deprecated requests for COM_CREATE_DB and COM_DROP_DB
    - Fixed bugs with Date / Datetime / Timestamp offsets.
    - Fixed a bug with executing a prepared statement that had bound parameters.
    - Other small fixes and changes.
  2. Ruben Oanta

    tabs -> spaces

    roanta authored
Commits on Aug 8, 2012
  1. Ruben Oanta

    - Added a synthetic response for CloseRequest which doesn't get a res…

    roanta authored
    …ponse from the server.
    
    - Better handling for unsupported LONG_BLOBs
    - Closed prepared statement in Example.scala
    - style fixes and comment fixes.
  2. Ruben Oanta
  3. Ruben Oanta
This page is out of date. Refresh to see the latest.
Showing with 3,381 additions and 1 deletion.
  1. +1 −1  finagle-memcached/src/test/scala/com/twitter/finagle/memcached/integration/ExternalMemcached.scala
  2. +67 −0 finagle-mysql/README.md
  3. +43 −0 finagle-mysql/pom.xml
  4. +169 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Client.scala
  5. +88 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Codec.scala
  6. +141 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Example.scala
  7. +5 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Exceptions.scala
  8. +177 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/codec/Endec.scala
  9. +44 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/codec/PacketFrameDecoder.scala
  10. +469 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Buffer.scala
  11. +57 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Capability.scala
  12. +63 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Charset.scala
  13. +93 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Handshake.scala
  14. +42 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Packet.scala
  15. +74 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/PreparedStatement.scala
  16. +186 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Request.scala
  17. +69 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Result.scala
  18. +179 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/ResultSet.scala
  19. +107 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Type.scala
  20. +242 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/protocol/Value.scala
  21. +17 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/util/BufferUtil.scala
  22. +52 −0 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/util/Query.scala
  23. +177 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/integration/ClientSpec.scala
  24. +271 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/codec/EndecSpec.scala
  25. +36 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/codec/PacketFrameDecoderSpec.scala
  26. +139 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/BufferSpec.scala
  27. +40 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/CapabilitySpec.scala
  28. +59 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/LoginRequestSpec.scala
  29. +18 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/PacketSpec.scala
  30. +195 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/RequestSpec.scala
  31. +50 −0 finagle-mysql/src/test/scala/com/twitter/finagle/mysql/unit/protocol/ServersGreetingSpec.scala
  32. +11 −0 project/Build.scala
2  finagle-memcached/src/test/scala/com/twitter/finagle/memcached/integration/ExternalMemcached.scala
View
@@ -17,7 +17,7 @@ object ExternalMemcached { self =>
p.waitFor()
require(p.exitValue() == 0, "memcached binary must be present.")
}
-
+
private[this] def findAddress() {
var tries = 100
while (address == None && tries >= 0) {
67 finagle-mysql/README.md
View
@@ -0,0 +1,67 @@
+A MySQL client built for finagle.
+
+---
+## Overview
+
+*This is meant to give a very brief overview of the MySQL Client/Server protocol and reference relevant source code within finagle-mysql. For an exposition of the MySQL Client/Server protocol, refer to [MySQL Forge](http://forge.mysql.com/wiki/MySQL_Internals_ClientServer_Protocol)
+and [Understanding MySQL Internal](http://my.safaribooksonline.com/book/databases/mysql/0596009577/client-server-communication/orm9780596009571-chp-4).*
+
+**Packets** - The basic unit of communication for the MySQL Client/Server protocol is an application-layer packet. A MySQL packet consists of a header (size and sequence number) followed by a body. Packets can be fragmented across frames during transmission. To simplify the decoding of results received from the server, the codec includes a packet frame decoder on the pipeline.
+
+`* protocol/Packet.scala, codec/PacketFrameDecoder.scala`
+
+**Handshake and Authentication** - When a client connects to the server, the server sends a greeting. This greeting contains information about the server version, protocol, capabilities, etc. The client replies with a login request containing, among other information, authentication credentials. To ensure connections are authenticated before they are issued by finagle, the codec implements prepareConnFactory with an AuthenticationProxy.
+
+`* protocol/Handshake.scala, Codec.scala`
+
+**Capabilities** - The client and server express their capability succinctly as bit vectors. Each set bit in the vector represents what the client/server is capable of or willing to do. Finagle-mysql provides constants of all the capability flags available (at the time of writing) and a comprehensive way to build capability bit vectors.
+
+ // This clients capabilities
+ val clientCapability = Capability(
+ LongFlag,
+ Transactions,
+ Protocol41,
+ FoundRows,
+ Interactive,
+ LongPassword,
+ ConnectWithDB,
+ SecureConnection,
+ LocalFiles
+ )
+`* protocol/Capability.scala, Codec.scala`
+
+ Note: This client only supports protocol version 4.1 (available in MySQL version 4.1 and above). This is strictly enforced during authentication with a MySQL server.
+
+**Requests** - Most requests sent to the server are Command Requests. They contain a command byte and any arguments that are specific to the command. Command Requests are sent to the server as a MySQL Packet. The first byte of the packet body must contain a valid command byte followed by the arguments. Within finagle-mysql, each Request object has a data field which defines the body of the Packet. Requests are translated into a logical MySQL Packet by the toChannelBuffer method when they reach the Encoder.
+
+`* protocol/Request.scala`
+
+**Results** - finagle-mysql translates packets received from the server into Scala objects. Each result object has a relevant decode method that translates the packet(s) into the object according to the protocol. Result packets can be distinguished by their first byte. Some result packets denote the start of a longer transmission and need to be defragged by the decoder.
+
+ResultSets are returned from the server for queries that return Rows. A Row can be String encoded or Binary encoded depending on the Request used to execute the query. For example, a QueryRequest uses the String protocol and a PreparedStatement uses the binary protocol.
+
+`* codec/Endec.scala, protocol/{Result.scala, ResultSet.scala, PreparedStatement.scala}`
+
+**Value** - finagle-mysql provides a Value ADT that can represent all values returned from MySQL. However, this does not include logic to decode every data type. For unsupported values finagle-mysql will return a RawStringValue and RawBinaryValue for the String and Binary protocols, respectively. Other note worthy Value objects include NullValue (SQL NULL) and EmptyValue.
+
+The following code depicts a robust, safe, and idiomatic way to extract and deconstruct a Value from a Row.
+
+ // The row.valueOf(...) method returns an Option[Value].
+ val userId: Option[Long] = row.valueOf("id") map {
+ case LongValue(id) => id
+ case _ => -1
+ }
+
+Pattern matching all possible values of the Value ADT gives great flexibility and control. For example, it allows the programmer to handle NullValues and EmptyValues with specific application logic.
+
+`* protocol/Value.scala`
+
+**Byte Buffers** - The BufferReader and BufferWriter interfaces provide convenient methods for reading/writing primitive data types exchanged between the client/server. This includes all primitive numeric types and strings (null-terminated and length coded). All Buffer methods are side-effecting, that is, each call to a read*/write* method will increase the current read and write position. Note, the bytes exchanged between the client/server are encoded in little-endian byte order.
+
+`* protocol/Buffer.scala`
+
+**Charset** - Currently, finagle-mysql only supports UTF-8 character encodings. This is strictly enforced when authenticating with a MySQL server. For more information about how to configure a MySQL instance to use UTF-8 refer to the [MySQL Documentation](http://dev.mysql.com/doc/refman/5.0/en/charset-applications.html).
+
+Note: MySQL also supports variable charsets at the table and field level. This charset data is part of the [Field Packet](http://dev.mysql.com/doc/internals/en/field-packet.html).
+
+`* protocol/Charset.scala`
43 finagle-mysql/pom.xml
View
@@ -0,0 +1,43 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-mysql</artifactId>
+ <packaging>jar</packaging>
+ <version>4.0.3-SNAPSHOT</version>
marius a. eriksen Collaborator

version needs update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ <parent>
+ <groupId>com.twitter</groupId>
+ <artifactId>scala-parent</artifactId>
+ <version>0.0.2</version>
+ <relativePath>../../parents/scala-parent/pom.xml</relativePath>
+ </parent>
+ <properties>
+ <git.dir>${project.basedir}/../../.git</git.dir>
+ </properties>
+ <dependencies>
+ <!-- library dependencies -->
+ <!-- project dependencies -->
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>finagle-core</artifactId>
+ <version>4.0.3-SNAPSHOT</version>
+ </dependency>
+ <dependency>
+ <groupId>com.twitter</groupId>
+ <artifactId>util-logging</artifactId>
+ <version>4.0.1</version>
+ </dependency>
marius a. eriksen Collaborator

ditto for these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ </dependencies>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <excludes>
+ </excludes>
+ </configuration>
+ </plugin>
+ </plugins>
+ </build>
+</project>
169 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Client.scala
View
@@ -0,0 +1,169 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle.builder.ClientBuilder
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.util.Query
+import com.twitter.finagle.Service
+import com.twitter.finagle.{ServiceFactory, Codec, CodecFactory}
+import com.twitter.util.Future
+
+object Client {
+ /**
+ * Constructs a Client given a ServiceFactory.
+ */
+ def apply(factory: ServiceFactory[Request, Result]): Client = {
+ new Client(factory)
+ }
+
+ /**
+ * Constructs a ServiceFactory using a single host.
+ * @param host a String of host:port combination.
+ * @param username the username used to authenticate to the mysql instance
+ * @param password the password used to authenticate to the mysql instance
+ * @param dbname database to initially use
+ */
+ def apply(host: String, username: String, password: String, dbname: Option[String]): Client = {
+ val factory = ClientBuilder()
+ .codec(new MySQL(username, password, dbname))
+ .hosts(host)
+ .hostConnectionLimit(1)
+ .buildFactory()
+
+ apply(factory)
+ }
+
+ def apply(host: String, username: String, password: String): Client = {
+ apply(host, username, password, None)
+ }
+
+ def apply(host: String, username: String, password: String, dbname: String): Client = {
+ apply(host, username, password, Some(dbname))
+ }
+}
+
+class Client(factory: ServiceFactory[Request, Result]) {
+ private[this] lazy val fService = factory.apply()
+
+ /**
+ * Sends a query to the server without using
+ * prepared statements.
+ * @param sql An sql statement to be executed.
+ * @return an OK Result or a ResultSet for queries that return
+ * rows.
+ */
+ def query(sql: String) = send(QueryRequest(sql)) {
+ case rs: ResultSet => Future.value(rs)
+ case ok: OK => Future.value(ok)
+ }
+
+ /**
+ * Runs a query that returns a result set. For each row
+ * in the ResultSet, call f on the row and return the results.
+ * @param sql A sql statement that returns a result set.
+ * @param f A function from ResultSet to any type T.
+ * @return a Future of Seq[T]
+ */
+ def select[T](sql: String)(f: Row => T): Future[Seq[T]] = query(sql) map {
+ case rs: ResultSet => rs.rows.map(f)
+ case ok: OK => Seq()
+ }
+
+ /**
+ * Sends a query to server to be prepared for execution.
+ * @param sql A query to be prepared on the server.
+ * @return PreparedStatement
+ */
+ def prepare(sql: String, params: Any*) = {
marius a. eriksen Collaborator

explicitly annotate return type here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ val stmt = Query.expandParams(sql, params)
+ send(PrepareRequest(stmt)) {
+ case ps: PreparedStatement =>
+ ps.statement.setValue(stmt)
+ if(params.size > 0)
+ ps.parameters = Query.flatten(params).toArray
+
+ Future.value(ps)
+ }
+ }
+
+ /**
+ * Execute a prepared statement.
+ * @return an OK Result or a ResultSet for queries that return
+ * rows.
+ */
+ def execute(ps: PreparedStatement) = send(ExecuteRequest(ps)) {
+ case rs: ResultSet =>
+ ps.bindParameters()
+ Future.value(rs)
+ case ok: OK =>
+ ps.bindParameters()
+ Future.value(ok)
+ }
+
+ /**
+ * Combines the prepare and execute operations.
+ * @return a Future[(PreparedStatement, Result)] tuple.
+ */
+ def prepareAndExecute(sql: String, params: Any*) =
+ prepare(sql, params: _*) flatMap { ps =>
+ execute(ps) map {
+ res => (ps, res)
+ }
+ }
+
+
+ /**
+ * Runs a query that returns a result set. For each row
+ * in the ResultSet, call f on the row and return the results.
+ * @param ps A prepared statement.
+ * @param f A function from ResultSet to any type T.
+ * @return a Future of Seq[T]
+ */
+ def select[T](ps: PreparedStatement)(f: Row => T): Future[Seq[T]] = execute(ps) map {
+ case rs: ResultSet => rs.rows.map(f)
+ case ok: OK => Seq()
+ }
+
+ /**
+ * Combines the prepare and select operations.
+ * @return a Future[(PreparedStatement, Seq[T])] tuple.
+ */
+ def prepareAndSelect[T](sql: String, params: Any*)(f: Row => T) =
+ prepare(sql, params: _*) flatMap { ps =>
+ select(ps)(f) map {
+ seq => (ps, seq)
+ }
+ }
+
+ /**
+ * Close a prepared statement on the server.
+ * @return OK result.
+ */
+ def closeStatement(ps: PreparedStatement) = send(CloseRequest(ps)) {
+ case ok: OK => Future.value(ok)
+ }
+
+ def selectDB(schema: String) = send(UseRequest(schema)) {
+ case ok: OK => Future.value(ok)
+ }
+
+ def ping = send(PingRequest) {
+ case ok: OK => Future.value(ok)
+ }
+
+ /**
+ * Close the ServiceFactory and its underlying resources.
+ */
+ def close() = factory.close()
+
+ /**
+ * Helper function to send requests to the ServiceFactory
+ * and handle Error responses from the server.
+ */
+ private[this] def send[T](r: Request)(handler: PartialFunction[Result, Future[T]]) =
+ fService flatMap { service =>
+ service(r) flatMap (handler orElse {
+ case Error(c, s, m) => Future.exception(ServerError(c + " - " + m))
+ case result => Future.exception(ClientError("Unhandled result from server: " + result))
+ })
+ }
+}
marius a. eriksen Collaborator

for all public interfaces here: include return types. it serves both as (in code) documentation and it also fixes the types, which is important; otherwise refinement types may be exposed unintentionally, causing API conflicts later:

http://twitter.github.com/effectivescala/#Types%20and%20Generics-Return%20type%20annotations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
88 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Codec.scala
View
@@ -0,0 +1,88 @@
+package com.twitter.finagle.mysql
+
+import com.twitter.finagle._
+import com.twitter.finagle.mysql.codec.{PacketFrameDecoder, Endec}
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.finagle.mysql.protocol.Capability._
+import com.twitter.util.Future
+import org.jboss.netty.channel.{ChannelPipelineFactory, Channels, Channel}
+
+class MySQL(username: String, password: String, database: Option[String])
+ extends CodecFactory[Request, Result] {
+ private[this] val clientCapability = Capability(
+ LongFlag,
+ Transactions,
+ Protocol41,
+ FoundRows,
+ Interactive,
+ LongPassword,
+ ConnectWithDB,
+ SecureConnection,
+ LocalFiles
+ )
+
+ def server = throw new Exception("Not yet implemented...")
+
+ def client = Function.const {
+ new Codec[Request, Result] {
+
+ def pipelineFactory = new ChannelPipelineFactory {
+ def getPipeline = {
+ val pipeline = Channels.pipeline()
+
+ pipeline.addLast("frameDecoder", new PacketFrameDecoder)
+ pipeline.addLast("EncoderDecoder", new Endec)
+
+ pipeline
+ }
+ }
+
+ // Authenticate each connection before returning it via a ServiceFactoryProxy.
+ override def prepareConnFactory(underlying: ServiceFactory[Request, Result]) =
+ new AuthenticationProxy(underlying, username, password, database, clientCapability)
+
+ }
+ }
+}
+
+class AuthenticationProxy(
+ underlying: ServiceFactory[Request, Result],
+ username: String,
+ password: String,
+ database: Option[String],
+ clientCap: Capability)
+ extends ServiceFactoryProxy(underlying) {
+
+ def makeLoginReq(sg: ServersGreeting) =
+ LoginRequest(username, password, database, clientCap, sg.salt, sg.serverCap)
+
+ def acceptGreeting(res: Result) = res match {
+ case sg: ServersGreeting if !sg.serverCap.has(Capability.Protocol41) =>
+ Future.exception(IncompatibleServer("This client is only compatible with MySQL version 4.1 and later."))
+
+ case sg: ServersGreeting if !Charset.isUTF8(sg.charset) =>
+ Future.exception(IncompatibleServer("This client is only compatible with UTF-8 charset encoding."))
+
+ case sg: ServersGreeting =>
+ Future.value(sg)
+
+ case r =>
+ Future.exception(new ClientError("Invalid Reply type %s".format(r.getClass.getName)))
+ }
+
+ def acceptLogin(res: Result) = res match {
+ case r: OK =>
+ Future.value(res)
+
+ case Error(c, _, m) =>
+ Future.exception(ServerError("Error when authenticating the client "+ c + " - " + m))
+ }
+
+ override def apply(conn: ClientConnection) = for {
+ service <- self(conn)
+ result <- service(ClientInternalGreet)
+ sg <- acceptGreeting(result)
+ loginRes <- service(makeLoginReq(sg))
+ _ <- acceptLogin(loginRes)
+ } yield service
+}
141 finagle-mysql/src/main/scala/com/twitter/finagle/mysql/Example.scala
View
@@ -0,0 +1,141 @@
+import com.twitter.conversions.time._
+import com.twitter.util.Try
+import com.twitter.finagle.mysql._
+import com.twitter.finagle.mysql.protocol._
+import com.twitter.util.Future
+import java.net.InetSocketAddress
+import java.sql.Date
+
+case class SwimmingRecord(
+ event: String,
+ time: Float,
+ name: String,
+ nationality: String,
+ date: Date
+) extends {
+
+ def toArray = Array(event, time, name, nationality, date)
+
+ override def toString = {
+ def q(s: String) = "'" + s + "'"
+ "(" + q(event) + "," + time + "," + q(name) + "," + q(nationality) + "," + q(date.toString) + ")"
+ }
+}
+
+object SwimmingRecord {
+ val createTableSQL =
+ """CREATE TEMPORARY TABLE IF NOT EXISTS `finagle-mysql-example` (
+ `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
+ `event` varchar(30) DEFAULT NULL,
+ `time` float DEFAULT NULL,
+ `name` varchar(40) DEFAULT NULL,
+ `nationality` varchar(20) DEFAULT NULL,
+ `date` date DEFAULT NULL,
+ PRIMARY KEY (`id`)
+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8"""
+
+ val records = List(
+ SwimmingRecord("50 m freestyle", 20.91F, "Cesar Cielo", "Brazil", Date.valueOf("2009-12-18")),
+ SwimmingRecord("100 m freestyle", 46.91F, "Cesar Cielo", "Brazil", Date.valueOf("2009-08-02")),
+ SwimmingRecord("50 m backstroke", 24.04F, "Liam Tancock", "Great Britain", Date.valueOf("2009-08-02")),
+ SwimmingRecord("100 m backstroke", 51.94F, "Aaron Peirsol", "United States", Date.valueOf("2009-07-08")),
+ SwimmingRecord("50 m butterfly", 22.43F, "Rafael Munoz", "Spain", Date.valueOf("2009-05-05")),
+ SwimmingRecord("100 m butterfly", 49.82F, "Michael Phelps", "United States", Date.valueOf("2009-07-29"))
+ )
+}
+
+object Main {
+ def main(args: Array[String]): Unit = {
+ val options = parseArgs(Map(), args.toList)
+ val host = options.getOrElse("host", "localhost").asInstanceOf[String]
+ val port = options.getOrElse("port", 3306).asInstanceOf[Int]
+ val username = options.getOrElse("username", "<user>").asInstanceOf[String]
+ val password = options.getOrElse("password", "<password>").asInstanceOf[String]
+ val dbname = "test"
+
+ val client = Client(host+":"+port, username, password, dbname)
+ if (createTable(client) && insertValues(client)) {
+ val query = "SELECT * FROM `finagle-mysql-example` WHERE `date` BETWEEN '2009-06-01' AND '2009-8-31'"
+ val qres: Future[Seq[_]] = client.select(query) { row =>
+ // row.valueOf returns an Option[Value]
+ val event = row.valueOf("event") map {
+ case StringValue(s) => s
+ case _ => "Default"
+ }
+
+ val nationality = row.valueOf("nationality") map {
+ case StringValue(s) => s
+ case _ => "Default"
+ }
+
+ val date = row.valueOf("date") map {
+ case DateValue(d) => d
+ case _ => new Date(0)