New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec #171

Merged
merged 58 commits into from Jan 11, 2017

Conversation

Projects
None yet
3 participants
@tolbertam
Contributor

tolbertam commented Aug 9, 2016

For #76, Adds protocol v3 and v4 support. This is a work in progress, but it's functional and currently all tests pass.

Summary of the major changes:

  • Use scodec for serialization/deserialization. All serde has been moved into a standalone codec module with scodec being the only dependency.
  • Change Prime store implementations to keep the request intact instead of deconstructing on input and reconstructing when receiving via the route APIs. This should keep things simpler. Added a memoized prime member on the Then implementations to convert the inputs to a Prime.

There should be no functional changes for users with one slight exception. Previously if no Prime is found for a query, query parameters would not be recorded in the activity log for batch and prepared statements. Now it records the query parameters as if they were varchars. There may also be some small nuances in conversions from json strings/numbers/booleans/etc. into the native type for a given data type, but i haven't identified any yet.

What is remaining:

  • Update cql-antlr to include the new types added in v3/v4 (tuples, udt, time, date, smallint, tinyint, etc.). The codecs exist for these, but need to be tested. I think there might be some barriers to supporting UDTs (covered in #159), so depending on the difficulty level I may defer this. (scassandra/cql-antlr#4)
  • Clean up, document and add more tests for the codec module. scodec is really nice, but its API is intimidating from my view. It is also not a complete implementation of the protocol, but the existing implementation isn't either (can maybe do more as follow ons).
  • Replace ServerStubrRunnerIntegrationTest / PrimingJsonHelperTest. I removed these earlier, need to replace them with new alternatives.
  • Add/update headers (pending #170 merge)
  • Clean up commit history
@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Sep 5, 2016

Member

How's this going?

Are you at the Cassandra summit? Might be nice to review together

Member

chbatey commented Sep 5, 2016

How's this going?

Are you at the Cassandra summit? Might be nice to review together

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Sep 5, 2016

Contributor

Hi @chbatey, it's going pretty well! Existing tests are passing, what's left is adding additional tests in the codec module, adding additional tests for the new types and maybe some clean up. Hopefully I can complete that really soon.

I'll be at the summit, I'd definitely be up for a review, that'd be fun!

Contributor

tolbertam commented Sep 5, 2016

Hi @chbatey, it's going pretty well! Existing tests are passing, what's left is adding additional tests in the codec module, adding additional tests for the new types and maybe some clean up. Hopefully I can complete that really soon.

I'll be at the summit, I'd definitely be up for a review, that'd be fun!

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Sep 5, 2016

Contributor

Rebased against master and added license headers. Working on finishing up tests in the codec module and updating rest of the code to support the v3/v4 types. I'll probably get that all going and clean up the commit history today.

Contributor

tolbertam commented Sep 5, 2016

Rebased against master and added license headers. Working on finishing up tests in the codec module and updating rest of the code to support the v3/v4 types. I'll probably get that all going and clean up the commit history today.

@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Oct 11, 2016

Member

Busy man!

Member

chbatey commented Oct 11, 2016

Busy man!

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Oct 11, 2016

Contributor

rebased on master to get the cql-antlr changes. I'll proceed to wire in the new types, and from there that should leave just some testing/doc updates/cleanup!

Contributor

tolbertam commented Oct 11, 2016

rebased on master to get the cql-antlr changes. I'll proceed to wire in the new types, and from there that should leave just some testing/doc updates/cleanup!

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Oct 13, 2016

Contributor

Getting closer, completed adding the new types and failures. Going to spend some time tomorrow evening updating the docs. I'll then do some cleanup / optimization of the codec code and clean up the commit history. If all goes well should complete sometime this weekend.

Contributor

tolbertam commented Oct 13, 2016

Getting closer, completed adding the new types and failures. Going to spend some time tomorrow evening updating the docs. I'll then do some cleanup / optimization of the codec code and clean up the commit history. If all goes well should complete sometime this weekend.

tolbertam added some commits Jul 12, 2015

Remove JDK7 specific annotations
Use 'AnyMatch' as catch all
Use terminate instead of shutdown.
Remove scodec-akka dependency since it requires JDK8
Inline own implicit conversions.  One small difference between this
implementation and scodec-akka is this uses reflection instead of a
java class.
Various fixes for protocol version nuances.
Have collection-based data types consider protocol version for length.
Batch considers protocol version for whether or not to include flags.
Prepare considers protocol version for whether or not to include pks.
native PF for encoding from Any -> Native type for data type.
Use CL from statement only if prime does not include one.
Bump to 1.1.0
There are enough changes here, some of which may be (unintentionally)
breaking, bumping up the minor version seems like the right way to go
here.
Readd ServerStubRunnerIntegrationTest
Changes:

- Remove test that expected nothing for Options before Startup, we now
  return an error for any message sent before Startup (instead of just
  Query).
- Send valid payloads.  I.E. in the Startup case, it's not valid to send
  an empty body, isntead send 4 byte body with 0s for 'empty map'.
- Change ConnectionHandler to return Buffer.drop(requiredLength) instead
  of whatever the codec consumes, because the frame may for whatever
  reason have more data than needed by the Message codec.
Readd PrimingJsonHelperTest
Extend to add tests for all various result types.  Also test for
extracting PrimeCriteria.
Add some DataType tests
Change ValueCodec size to be ranged to Integer.MaxValue.
TupleCodec size to be ranged by #elements*ValueCodec.size
Move TupleCodec into its own file.
Allow OPTIONS messages to be sent before STARTUP.
Per Section 4.1.1 of the spec, the OPTIONS message can be sent before
the STARTUP message to find out options supported by the server.
Support that here.
Handle Protocol V1 query appropriately
Query flags are not present in protocol v1 so need to handle that
special case.

tolbertam added some commits Oct 9, 2016

Remove scodec-akka dependency (isn't needed)
This was accidentally revived during rebasing.

Also update java driver dependency.
Update cql-antlr to support v3 and v4 types
Also update PrimingJsonImplicits for tuples and add start of
TuplePriming test.
Nested collection support in cql-antlr.
Add nested tuple test to TuplePriming.
Add integration tests for protocol v3 and v4 types
Also do a little bit of refactoring to facilitate common code driver21
and driver30 by creating a common-v3 module.  This was required because
TupleValue is part of the driver, so to prevent the common module from
requiring driver 2.1+ a separate module was needed.

Non-driver types like tinyint, smallint, date and time are accounted for
in common, with methods in driver20 for returning those returning
UnsupportedOperationException.
Implementation and Tests for protocol v4 error messages
WriteFailure, ReadFailure, and FunctionFailure result types.
Fix bug where the queryPatternPrimes were not showing up
The /prime-query-single endpoint was not showing primes from the pattern
store.  They were still being used for prime selection on the other
hand.
Add documentation for protocol v3 and v4 changes.
Also add example for priming a query with a pattern and some other
cleanup.
Revert logic to treat RowMetadata as an Option.
Need to be able to differentiate when there are no columns and no column
metadata was requested.
Cache Row codecs and add more documentation.
Rename BadCredentials to AuthenticationError.

Switch from scalaz for Memo to guava for Cache as would like to use a
LRU cache for Row codecs since this could grow large if you have a lot
of unique metadata.
@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Oct 15, 2016

Contributor

I've spent some time the last few days adding integration tests for the new types and errors, and some more tests and documentation for the codec module.

I think i've got everything worked out and this is ready aside from the commit history. @chbatey would you prefer I squash the commits now or would you have me leave it as is for now for review?

Contributor

tolbertam commented Oct 15, 2016

I've spent some time the last few days adding integration tests for the new types and errors, and some more tests and documentation for the codec module.

I think i've got everything worked out and this is ready aside from the commit history. @chbatey would you prefer I squash the commits now or would you have me leave it as is for now for review?

@tolbertam tolbertam changed the title from [#76] [wip] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec to [#76] Protocol v3 & v4 support, serialization/deserialization rewrite using scodec Oct 16, 2016

@mgobec

This comment has been minimized.

Show comment
Hide comment
@mgobec

mgobec Oct 16, 2016

Great work @tolbertam 👍

mgobec commented Oct 16, 2016

Great work @tolbertam 👍

@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Nov 18, 2016

Member

Taking a look this weekend

Member

chbatey commented Nov 18, 2016

Taking a look this weekend

Use spray-routing-shapeless23 to prevent conflict with scodec.
spray-routing-shapeless2 depends on shapeless 2.1, where scodec
depends on shapeless 2.3.  Depending on what is being used for
dependency resolution, if shapeless 2.1 is chosen over 2.3 this causes
incompatibility issues with scodec.
@chbatey

chbatey approved these changes Jan 5, 2017

Show outdated Hide outdated server/src/main/scala/org/scassandra/server/actors/ConnectionHandler.scala
private[this] def inspectProtocolVersion(input: ByteVector): Try[(ProtocolFlags, ByteVector)] = {
next[ProtocolFlags](input).flatMap {
case ((ProtocolFlags(_, UnsupportedProtocolVersion(v)), _)) =>
Failure(new UnsupportedProtocolException(v))

This comment has been minimized.

@chbatey

chbatey Jan 5, 2017

Member

No need for new

@chbatey

chbatey Jan 5, 2017

Member

No need for new

Show outdated Hide outdated ...in/scala/org/scassandra/server/actors/NativeProtocolMessageHandler.scala
val registerHandler = registerHandlerFactory(context)
write(Ready, frame.header)
context become initialized(queryHandler, batchHandler, registerHandler)
case _ => {

This comment has been minimized.

@chbatey

chbatey Jan 5, 2017

Member

remove braces to make it consistent with the other cases

@chbatey

chbatey Jan 5, 2017

Member

remove braces to make it consistent with the other cases

Show outdated Hide outdated server/src/main/scala/org/scassandra/server/actors/ExecuteHandler.scala
case result: FatalResult => result.produceFatalError(connection)
writePrime(execute, prime, header, Some(connection), alternative=Some(Reply(VoidResult)), consistency = Some(execute.parameters.consistency))
case None =>
var errMsg = s"Could not find prepared statement with id: 0x${execute.id.toHex}"

This comment has been minimized.

@chbatey

chbatey Jan 5, 2017

Member

should be a val

@chbatey

chbatey Jan 5, 2017

Member

should be a val

This comment has been minimized.

@tolbertam

tolbertam Jan 5, 2017

Contributor

d'oh! I've been doing too much javascript lately 😄

@tolbertam

tolbertam Jan 5, 2017

Contributor

d'oh! I've been doing too much javascript lately 😄

Show outdated Hide outdated server/src/main/scala/org/scassandra/server/actors/ExecuteHandler.scala
writePrime(execute, prime, header, Some(connection), alternative=Some(Reply(VoidResult)), consistency = Some(execute.parameters.consistency))
case None =>
var errMsg = s"Could not find prepared statement with id: 0x${execute.id.toHex}"
activityLog.recordPreparedStatementExecution(errMsg, execute.parameters.consistency, Nil, Nil)

This comment has been minimized.

@chbatey

chbatey Jan 5, 2017

Member

i take it you found this useful for testing? just a little off we use an error msg as the statement text

@chbatey

chbatey Jan 5, 2017

Member

i take it you found this useful for testing? just a little off we use an error msg as the statement text

This comment has been minimized.

@tolbertam

tolbertam Jan 5, 2017

Contributor

Hrmm, I may be misunderstanding, isn't the behavior the same as before? I think the result is in this case that we record a prepared statement execution with the error message as the query text, and then an Unprepared message gets sent back to the client which will cause most drivers to repreare the statement.

@tolbertam

tolbertam Jan 5, 2017

Contributor

Hrmm, I may be misunderstanding, isn't the behavior the same as before? I think the result is in this case that we record a prepared statement execution with the error message as the query text, and then an Unprepared message gets sent back to the client which will cause most drivers to repreare the statement.

@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Jan 5, 2017

Member

Happy to merge this whenever. I'll be adding some features shortly

Member

chbatey commented Jan 5, 2017

Happy to merge this whenever. I'll be adding some features shortly

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Jan 5, 2017

Contributor

Happy to merge this whenever. I'll be adding some features shortly

Excellent, thanks for the review! I'll address your feedback this morning.

Contributor

tolbertam commented Jan 5, 2017

Happy to merge this whenever. I'll be adding some features shortly

Excellent, thanks for the review! I'll address your feedback this morning.

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Jan 5, 2017

Contributor

added commit for review feedback, thanks! Would you like me to do anything with the commit history, or would you just prefer to squash the whole thing on merge?

Contributor

tolbertam commented Jan 5, 2017

added commit for review feedback, thanks! Would you like me to do anything with the commit history, or would you just prefer to squash the whole thing on merge?

@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Jan 5, 2017

Member

Squashing into one if find!

It has been a long time since I looked at Scala lol

Member

chbatey commented Jan 5, 2017

Squashing into one if find!

It has been a long time since I looked at Scala lol

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Jan 5, 2017

Contributor

Just remembered, when it comes time to do a release with these changes, there are a few activites:

  • Merge scassandra/cql-antlr#7 to indicate that repository is deprecated in favor of cql-antlr being in this repository now.
  • Update gh-pages branch / push to read the docs since i made some doc updates.
Contributor

tolbertam commented Jan 5, 2017

Just remembered, when it comes time to do a release with these changes, there are a few activites:

  • Merge scassandra/cql-antlr#7 to indicate that repository is deprecated in favor of cql-antlr being in this repository now.
  • Update gh-pages branch / push to read the docs since i made some doc updates.
@chbatey

This comment has been minimized.

Show comment
Hide comment
Member

chbatey commented Jan 6, 2017

Thanks @tolbertam

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Jan 10, 2017

Contributor

@chbatey would you like me to merge and go through the process to release this? Can do it tomorrow if you like.

Contributor

tolbertam commented Jan 10, 2017

@chbatey would you like me to merge and go through the process to release this? Can do it tomorrow if you like.

@chbatey

This comment has been minimized.

Show comment
Hide comment
@chbatey

chbatey Jan 10, 2017

Member
Member

chbatey commented Jan 10, 2017

@tolbertam

This comment has been minimized.

Show comment
Hide comment
@tolbertam

tolbertam Jan 10, 2017

Contributor

Awesome, thanks! Will take care of that this evening :)

Contributor

tolbertam commented Jan 10, 2017

Awesome, thanks! Will take care of that this evening :)

@tolbertam tolbertam merged commit 41432b2 into scassandra:master Jan 11, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment