Skip to content

Commit

Permalink
Update to unified Java driver 4.5.0 (#20)
Browse files Browse the repository at this point in the history 
* Not sure how, but somehow I missed the commit that, you know, changed the driver version we use 🤦

* Picking up some low-hanging fruit, mostly import changes

* Some more low-hanging fruit, mainly conversions to use the shaded Guava

* Fixing a package rename that was missed in the earlier round

* Convert usages of Guava's FutureCallback (which aren't actually being used in future-handling code) to a generic callback trait

* Trying to fix as much of the base CQL stuff as possible.

Big chunks of this are based on/copied wholesale from previous work done by Arthur Landim (#17)

* Added explicit bounded Statement type to DseCqlAttributes. This type propogates through the statement/attribute level but doesn't make it up to the action level (since the actions don't seem to care about the underlying Statement type their executing... or at least they don't so far).

* A few minor fixes

* Cleaning up PagingState references, a few other minor cleanups

* A few miscellaneous fixes

* Removed unavailable methods for accessing tried and queried hosts and replaced with accessor for coordinator node. Also cleaned up a fair amount of DseResponse.

* Most of the core graph refactoring

* Fixes to DseResponseHandler... and a few other miscellaneous fixes

* Updates to Cql/GraphRequestAction to match other work done elsewhere in these changes. Still need to update the bits around the explicit interaction with the driver.

* A bit of refactring to remove some of the larger fn definitions in the action impls

* Fixing some weird import ordering issues

* Conversion to use a builder rather than a (template) statement in CqlRequestAction... should make supporting the setting of various options from DseCqlAttributes considerably easier.

* Convert graph infrastructure to use builders coming off of DseSession rather than Statement directly

* Convert CqlPreparedStatementUtil to work with Bindable instances rather than statements specifically.  Facilitates
conversion to builders generally.

Also converted CQL statements to use builders in order to address immutable statement classes

* Cleaning up some type stuff

* Converted CQL props to use only what's actually supported on the driver now.  Still need to do the same for graph.
Some things have moved into driver configuration, most notably the retry policy.

Also simplified the DseResponse API.  Primary accessor here is for the (Graph|Async)ResultSet... these vals can then
be converted to Seqs using ResultSetUtils and managed via Scala's collection API.  Preserved as many of the existing
checks as possible mainly for backward compatibility; going forward many users would migrate to a Seq-based impl.

The DseResponse API changes were (largely) a result of the distinction between ResultSet and AsyncResultSet in the 4.x
driver.

* Converted graph props to include only what's supported on the driver now.

* Converting from GraphResultSet to AsyncGraphResultSet

* Simple impls of Iterators for async RS impls

* Fixing up some type things

* Initial work on spec fixes

* Code review feedback

* Consolidating check API into something which exposes the various ResultSet impls only.  Previously defined checks
can now be implemented as transforms off of the ResultSet instance via fns defined somewhere.

* More change from code review

* Upgrading to new cassandra-unit which includes 4.2.x driver support)

* Fixing a few tests

* Specs should at least compile now

* Base compilation now succeeds... on to actually fixing tests

* Fixes for various spec failures.

Don't try to create mocks for Durations... it makes them ANGRY!

* Fighting with mocks is awesome

* All non-cassandra-unit specs should be passing now

* Various test fixes.  Includes a fix to get cassandra-util tests running again; new version of cassandra-util
calls initSession() when trying to retrieve a session from a (possibly not yet initialized) embedded instance.
As a result our old pattern of only creating the embedded instance if the session is null was bound to fail.
Since we're creating the instance in an object anyway it seems safe to do without the guard.

* CqlPreparedStatementUtil spec now passes!

* Explicitly specify traversal source for graph queries.  This doesn't actually matter in practice since cassandra-unit
only runs these simulations against a base C* instance which won't understand graph queries anyway.  Still, I'm including
it here in case this situation should change.

* At long last... we have a clean run of tests!

* Removing sbt changes that were accidentally committed

* cassandra-unit released an official 4.x release while work was ongoing

* Shift to an API based on DataType rather than ints coming off of getProtocolCode()

* Upgrade to unified driver

* Fixing compile warnings re: unchecked types due to type erasure

* Replace DseSession with CqlSession refs

* Changing _all_ refs to DseSession to CqlSession instead

* Code review feedback

* Fixing default DC impl + cleaning up handling of memoized Sessions

* Fixing exception messages

* Don't bother going to the iterator... just use what Iterable gives us

* Added option to set idempotecy based on an existing boolean val

* Addressing PR feedback

* More review feedback

* Code review feedback

* More code review feedback

* Shifting away from companion objects and towards implicit vals

* Adding docs about the new check API

* Re-adding support for per-request proxy auth

* Some more code review feedback

* Removing support for explicit specification of collection classes in the Gatling session from prepared statement support.
Olivier pointed out that the prepared statement itself can serve as a source of truth for type info here.

* Driver version bump to 4.5.0
  • Loading branch information
@absurdfarce
absurdfarce committed Mar 24, 2020
1 parent 8e8ed15 commit 17a1613
Show file tree
Hide file tree
Showing 43 changed files with 1,604 additions and 1,774 deletions.
25 changes: 13 additions & 12 deletions build.sbt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,21 @@ val gatlingVersion = "2.3.0"

scalacOptions += "-target:jvm-1.8"

libraryDependencies += "com.datastax.dse" % "dse-java-driver-core" % "1.6.8"
libraryDependencies += "com.datastax.dse" % "dse-java-driver-graph" % "1.6.8"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.18.0"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.1"
libraryDependencies += "org.hdrhistogram" % "HdrHistogram" % "2.1.10"
libraryDependencies += "com.datastax.oss" % "java-driver-core" % "4.5.0"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.18.0"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.1"
libraryDependencies += "org.hdrhistogram" % "HdrHistogram" % "2.1.10"

libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % gatlingVersion % Provided
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % gatlingVersion % Provided

libraryDependencies += "org.fusesource" % "sigar" % "1.6.4" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "org.easymock" % "easymock" % "3.5" % Test
libraryDependencies += "org.cassandraunit" % "cassandra-unit" % "4.3.1.0" % Test
libraryDependencies += "org.pegdown" % "pegdown" % "1.6.0" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.11" % Test
libraryDependencies += "com.datastax.oss" % "java-driver-query-builder" % "4.4.0" % Test

libraryDependencies += "org.fusesource" % "sigar" % "1.6.4" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "org.easymock" % "easymock" % "3.5" % Test
libraryDependencies += "org.cassandraunit" % "cassandra-unit" % "3.3.0.2" % Test
libraryDependencies += "org.pegdown" % "pegdown" % "1.6.0" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.11" % Test

resolvers += Resolver.mavenLocal
resolvers += Resolver.mavenCentral
Expand Down
8 changes: 4 additions & 4 deletions src/main/scala/com/datastax/gatling/plugin/DseProtocol.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ import java.util.concurrent.atomic.AtomicLong

import akka.Done
import akka.actor.ActorSystem
import com.datastax.driver.dse.DseSession
import com.datastax.gatling.plugin.metrics.MetricsLogger
import com.datastax.gatling.plugin.request.{CqlRequestActionBuilder, GraphRequestActionBuilder}
import com.datastax.gatling.plugin.utils.GatlingTimingSource
import com.datastax.oss.driver.api.core.CqlSession
import com.typesafe.scalalogging.StrictLogging
import io.gatling.core.CoreComponents
import io.gatling.core.config.GatlingConfiguration
Expand Down Expand Up @@ -63,7 +63,7 @@ object DseProtocol extends StrictLogging {
}
}

case class DseProtocol(session: DseSession) extends Protocol
case class DseProtocol(session: CqlSession) extends Protocol

object DseComponents {
private val componentsCache = mutable.Map[ActorSystem, DseComponents]()
Expand Down Expand Up @@ -126,10 +126,10 @@ case class DseComponents(dseProtocol: DseProtocol,


object DseProtocolBuilder {
def session(session: DseSession) = DseProtocolBuilder(session)
def session(session: CqlSession) = DseProtocolBuilder(session)
}

case class DseProtocolBuilder(session: DseSession) {
case class DseProtocolBuilder(session: CqlSession) {
def build = DseProtocol(session)
}

4 changes: 2 additions & 2 deletions src/main/scala/com/datastax/gatling/plugin/Predef.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ trait DsePredefBase extends DseCheckSupport {

implicit def protocolBuilder2DseProtocol(builder: DseProtocolBuilder): DseProtocol = builder.build

implicit def cqlRequestAttributes2ActionBuilder(builder: DseCqlAttributesBuilder): ActionBuilder = builder.build()
implicit def cqlRequestAttributes2ActionBuilder(builder: DseCqlAttributesBuilder[_,_]): ActionBuilder = builder.build()

implicit def graphRequestAttributes2ActionBuilder(builder: DseGraphAttributesBuilder): ActionBuilder = builder.build()
implicit def graphRequestAttributes2ActionBuilder(builder: DseGraphAttributesBuilder[_, _]): ActionBuilder = builder.build()
}

/**
Expand Down
51 changes: 5 additions & 46 deletions src/main/scala/com/datastax/gatling/plugin/checks/CqlChecks.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,15 @@

package com.datastax.gatling.plugin.checks

import com.datastax.driver.core.{ResultSet, Row}
import com.datastax.oss.driver.api.core.cql.{AsyncResultSet, Statement, StatementBuilder}
import com.datastax.gatling.plugin.response.CqlResponse
import io.gatling.commons.validation.{SuccessWrapper, Validation}
import io.gatling.core.check._
import io.gatling.core.check.extractor.{CountArity, CriterionExtractor, Extractor, FindAllArity, FindArity, SingleArity, _}
import io.gatling.core.check.extractor.{Extractor, SingleArity}
import io.gatling.core.session.{Expression, ExpressionSuccessWrapper, Session}

import scala.collection.mutable


/**
* This class serves as model for the CQL-specific checks.
*
Expand Down Expand Up @@ -51,50 +50,10 @@ private class CqlResponseExtractor[X](val name: String,
}
}

private abstract class ColumnValueExtractor[X] extends CriterionExtractor[CqlResponse, Any, X] {
val criterionName = "columnValue"
}

private class SingleColumnValueExtractor(val criterion: String, val occurrence: Int) extends ColumnValueExtractor[Any] with FindArity {
def extract(response: CqlResponse): Validation[Option[Any]] =
response.getCqlResultColumnValues(criterion).lift(occurrence).success
}

private class MultipleColumnValueExtractor(val criterion: String) extends ColumnValueExtractor[Seq[Any]] with FindAllArity {
def extract(response: CqlResponse): Validation[Option[Seq[Any]]] =
response.getCqlResultColumnValues(criterion).liftSeqOption.success
}

private class CountColumnValueExtractor(val criterion: String) extends ColumnValueExtractor[Int] with CountArity {
def extract(response: CqlResponse): Validation[Option[Int]] =
response.getCqlResultColumnValues(criterion).liftSeqOption.map(_.size).success
}

object CqlChecks {
val resultSet =
new CqlResponseExtractor[ResultSet](
val resultSet:CqlCheckBuilder[AsyncResultSet] =
new CqlResponseExtractor[AsyncResultSet](
"resultSet",
r => r.getCqlResultSet)
.toCheckBuilder

val allRows =
new CqlResponseExtractor[Seq[Row]](
"allRows",
r => r.getAllRows)
r => r.resultSet)
.toCheckBuilder

val oneRow =
new CqlResponseExtractor[Row](
"oneRow",
r => r.getOneRow)
.toCheckBuilder

def columnValue(columnName: Expression[String]) = {
val cqlResponseExtender: Extender[DseCqlCheck, CqlResponse] = wrapped => DseCqlCheck(wrapped)
new DefaultMultipleFindCheckBuilder[DseCqlCheck, CqlResponse, CqlResponse, Any](cqlResponseExtender, x => x.success) {
def findExtractor(occurrence: Int) = columnName.map(new SingleColumnValueExtractor(_, occurrence))
def findAllExtractor = columnName.map(new MultipleColumnValueExtractor(_))
def countExtractor = columnName.map(new CountColumnValueExtractor(_))
}
}
}
108 changes: 67 additions & 41 deletions src/main/scala/com/datastax/gatling/plugin/checks/DseCheckSupport.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,75 @@

package com.datastax.gatling.plugin.checks

import io.gatling.core.session.ExpressionSuccessWrapper
import com.datastax.dse.driver.api.core.graph.AsyncGraphResultSet
import com.datastax.oss.driver.api.core.cql.AsyncResultSet
import com.datastax.gatling.plugin.utils.ResultSetUtils

/**
* Make both CQL and Graph checks available to the DSL.
*
* Note that as of 1.3.5 (and the upgrade to the unified OSS driver it brings along) the API here has changed.
* The old check API exposed a rich set of checks for various operations including row counts and validating
* data in individual rows. Several (most?) of these checks were built on the idea that all rows were immediately
* available in memory. This design has changed in 1.3.5, so maintaining the old API would've proven quite difficult
* (if not impossible). Additionally, the rich API shields the user from the intricacies of the driver API at the
* cost of limited flexibility; implementing new functionality requires modifications to the plugin itself (or at
* least an awareness of it's innards).
*
* With 1.3.5 this relationship has been inverted. The check API has been reduced to a single check which makes the
* underlying [[AsyncResultSet]] or [[AsyncGraphResultSet]] available. Simulations can then use transform() to
* extract values and evaluate them as necessary. So, for instance, something like this:
*
* {{{
* .check(columnValue("counter_type") is 2)
* }}}
*
* now becomes:
*
* {{{
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* }}}
*
* Note that these transforms are now managed as Scala code within the simulations so they can be abstracted and
* built into libraries which can be re-used across simulations. Also note that this abstraction can be implemented
* without modifying the plugin itself.
*
* A similar pattern applies to checks based on metadata. So this:
*
* {{{
* .check(exhausted is true)
* }}}
*
* now becomes:
*
* {{{
* .check(resultSet.transform(_.hasMorePages) is false)
* .check(resultSet.transform(_.remaining) is 0)
* }}}
*
* At this point we should also note that checks are now explicitly executed in the order in which hey are declared
* in the simulation. This matters because iterating through rows will impact methods such as remaining(). So, for
* example, if you want to validate that a single row was returned and it contained a specific value you should do
* something like:
*
* {{{
* .check(resultSet.transform(_.remaining) is 1)
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* }}}
*
* and not:
*
* {{{
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* .check(resultSet.transform(_.remaining) is 1)
* }}}
*
* Finally, in general the expectation is that you won't need to realize all rows in a result set, but if for some
* reason you find this necessary this functionality is supplied in [[ResultSetUtils]]. This class also serves as
* an example of the kind of abstraction over common extraction operations discussed above.
*/
trait DseCheckSupport {

// start global checks
lazy val exhausted = GenericChecks.exhausted
lazy val applied = GenericChecks.applied
lazy val rowCount = GenericChecks.rowCount

// execution info and subsets
lazy val executionInfo = GenericChecks.executionInfo
lazy val achievedCL = GenericChecks.achievedConsistencyLevel
lazy val pagingState = GenericChecks.pagingState
lazy val queriedHost = GenericChecks.queriedHost
lazy val schemaAgreement = GenericChecks.schemaInAgreement
lazy val successfulExecutionIndex = GenericChecks.successfulExecutionIndex
lazy val triedHosts = GenericChecks.triedHosts
lazy val warnings = GenericChecks.warnings

// start CQL only checks
lazy val resultSet = CqlChecks.resultSet
lazy val allRows = CqlChecks.allRows
lazy val oneRow = CqlChecks.oneRow

// start Graph only checks
lazy val graphResultSet = GraphChecks.graphResultSet
lazy val allNodes = GraphChecks.allNodes
lazy val oneNode = GraphChecks.oneNode

def edges(columnName: String) = GraphChecks.edges(columnName)

def vertexes(columnName: String) = GraphChecks.vertexes(columnName)

def paths(columnName: String) = GraphChecks.paths(columnName)

def properties(columnName: String) = GraphChecks.paths(columnName)

def vertexProperties(columnName: String) = GraphChecks.vertexProperties(columnName)

/**
* Get a column by name returned by the CQL statement.
* Note that this statement implicitly fetches <b>all</b> rows from the result set!
*/
def columnValue(columnName: String) = CqlChecks.columnValue(columnName.expressionSuccess)
lazy val graphResultSet = GraphChecks.resultSet
}

129 changes: 0 additions & 129 deletions src/main/scala/com/datastax/gatling/plugin/checks/GenericChecks.scala
View file

This file was deleted.

Loading

0 comments on commit 17a1613

Please sign in to comment.