Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JAVA-2563 and JAVA-2564 #20

Merged
merged 65 commits into from
Mar 24, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
d746998
Not sure how, but somehow I missed the commit that, you know, changed…
absurdfarce Dec 3, 2019
c1279ec
Picking up some low-hanging fruit, mostly import changes
absurdfarce Nov 21, 2019
a83b146
Some more low-hanging fruit, mainly conversions to use the shaded Guava
absurdfarce Nov 21, 2019
3fa0fd4
Fixing a package rename that was missed in the earlier round
absurdfarce Nov 25, 2019
ecd60a7
Convert usages of Guava's FutureCallback (which aren't actually being…
absurdfarce Nov 25, 2019
ce3225b
Trying to fix as much of the base CQL stuff as possible.
absurdfarce Nov 25, 2019
f365d98
Added explicit bounded Statement type to DseCqlAttributes. This type …
absurdfarce Nov 26, 2019
192d7bf
A few minor fixes
absurdfarce Nov 26, 2019
1915246
Cleaning up PagingState references, a few other minor cleanups
absurdfarce Nov 26, 2019
da75ad1
A few miscellaneous fixes
absurdfarce Nov 26, 2019
cff2478
Removed unavailable methods for accessing tried and queried hosts and…
absurdfarce Nov 26, 2019
01b87b4
Most of the core graph refactoring
absurdfarce Nov 27, 2019
7521b1e
Fixes to DseResponseHandler... and a few other miscellaneous fixes
absurdfarce Nov 27, 2019
bf85985
Updates to Cql/GraphRequestAction to match other work done elsewhere …
absurdfarce Dec 2, 2019
c488c5a
A bit of refactring to remove some of the larger fn definitions in th…
absurdfarce Dec 3, 2019
632eea8
Fixing some weird import ordering issues
absurdfarce Dec 3, 2019
b55391f
Conversion to use a builder rather than a (template) statement in Cql…
absurdfarce Dec 4, 2019
b45d2de
Convert graph infrastructure to use builders coming off of DseSession…
absurdfarce Dec 4, 2019
0a9d372
Convert CqlPreparedStatementUtil to work with Bindable instances rath…
absurdfarce Dec 5, 2019
ef25ca6
Cleaning up some type stuff
absurdfarce Dec 5, 2019
06aab95
Converted CQL props to use only what's actually supported on the driv…
absurdfarce Dec 6, 2019
b56f341
Converted graph props to include only what's supported on the driver …
absurdfarce Dec 9, 2019
fca261f
Converting from GraphResultSet to AsyncGraphResultSet
absurdfarce Dec 9, 2019
62d8326
Simple impls of Iterators for async RS impls
absurdfarce Dec 10, 2019
3183a44
Fixing up some type things
absurdfarce Dec 10, 2019
1c06158
Initial work on spec fixes
absurdfarce Dec 11, 2019
88089e3
Code review feedback
absurdfarce Dec 18, 2019
6919b02
Merge branch 'java2563' into java2563-tests
absurdfarce Dec 18, 2019
272548b
Consolidating check API into something which exposes the various Resu…
absurdfarce Dec 18, 2019
7ed38d5
Merge branch 'java2563' into java2563-tests
absurdfarce Dec 18, 2019
b8ff9ea
More change from code review
absurdfarce Dec 19, 2019
1fb447b
Merge branch 'java2563' into java2563-tests
absurdfarce Dec 19, 2019
02fe912
Upgrading to new cassandra-unit which includes 4.2.x driver support)
absurdfarce Dec 19, 2019
bfcb171
Fixing a few tests
absurdfarce Dec 19, 2019
2f5c474
Specs should at least compile now
absurdfarce Jan 6, 2020
9da8cca
Base compilation now succeeds... on to actually fixing tests
absurdfarce Jan 7, 2020
17117d1
Fixes for various spec failures.
absurdfarce Jan 9, 2020
e00ad38
Fighting with mocks is awesome
absurdfarce Jan 9, 2020
64f9bb4
All non-cassandra-unit specs should be passing now
absurdfarce Jan 10, 2020
1bffa1a
Various test fixes. Includes a fix to get cassandra-util tests runni…
absurdfarce Jan 14, 2020
29e69f3
CqlPreparedStatementUtil spec now passes!
absurdfarce Jan 15, 2020
2f48dc2
Explicitly specify traversal source for graph queries. This doesn't …
absurdfarce Jan 17, 2020
637cd78
At long last... we have a clean run of tests!
absurdfarce Jan 19, 2020
d1be0a6
Removing sbt changes that were accidentally committed
absurdfarce Jan 19, 2020
3432a92
cassandra-unit released an official 4.x release while work was ongoing
absurdfarce Jan 19, 2020
539edf9
Shift to an API based on DataType rather than ints coming off of getP…
absurdfarce Jan 19, 2020
a70f4af
Upgrade to unified driver
absurdfarce Jan 19, 2020
545df55
Fixing compile warnings re: unchecked types due to type erasure
absurdfarce Jan 24, 2020
993aaa4
Replace DseSession with CqlSession refs
absurdfarce Jan 28, 2020
c6962bb
Changing _all_ refs to DseSession to CqlSession instead
absurdfarce Jan 28, 2020
74b668c
Code review feedback
absurdfarce Jan 28, 2020
1658c56
Fixing default DC impl + cleaning up handling of memoized Sessions
absurdfarce Jan 28, 2020
bf767a9
Fixing exception messages
absurdfarce Jan 29, 2020
41dcc4b
Don't bother going to the iterator... just use what Iterable gives us
absurdfarce Jan 29, 2020
7754017
Added option to set idempotecy based on an existing boolean val
absurdfarce Jan 30, 2020
803c2c7
Addressing PR feedback
absurdfarce Feb 4, 2020
568c3a3
More review feedback
absurdfarce Feb 4, 2020
2991fdd
Code review feedback
absurdfarce Feb 10, 2020
5929a6d
More code review feedback
absurdfarce Feb 10, 2020
0b5172e
Shifting away from companion objects and towards implicit vals
absurdfarce Feb 10, 2020
3adbda5
Adding docs about the new check API
absurdfarce Feb 10, 2020
cca2ef9
Re-adding support for per-request proxy auth
absurdfarce Feb 11, 2020
1c69582
Some more code review feedback
absurdfarce Feb 14, 2020
228a4e8
Removing support for explicit specification of collection classes in …
absurdfarce Mar 10, 2020
7d9396a
Driver version bump to 4.5.0
absurdfarce Mar 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,21 @@ val gatlingVersion = "2.3.0"

scalacOptions += "-target:jvm-1.8"

libraryDependencies += "com.datastax.dse" % "dse-java-driver-core" % "1.6.8"
libraryDependencies += "com.datastax.dse" % "dse-java-driver-graph" % "1.6.8"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.18.0"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.1"
libraryDependencies += "org.hdrhistogram" % "HdrHistogram" % "2.1.10"
libraryDependencies += "com.datastax.oss" % "java-driver-core" % "4.5.0"
libraryDependencies += "com.github.nscala-time" %% "nscala-time" % "2.18.0"
libraryDependencies += "com.fasterxml.jackson.module" %% "jackson-module-scala" % "2.9.1"
libraryDependencies += "org.hdrhistogram" % "HdrHistogram" % "2.1.10"

libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % gatlingVersion % Provided
libraryDependencies += "io.gatling.highcharts" % "gatling-charts-highcharts" % gatlingVersion % Provided

libraryDependencies += "org.fusesource" % "sigar" % "1.6.4" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "org.easymock" % "easymock" % "3.5" % Test
libraryDependencies += "org.cassandraunit" % "cassandra-unit" % "4.3.1.0" % Test
libraryDependencies += "org.pegdown" % "pegdown" % "1.6.0" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.11" % Test
libraryDependencies += "com.datastax.oss" % "java-driver-query-builder" % "4.4.0" % Test

libraryDependencies += "org.fusesource" % "sigar" % "1.6.4" % Test
libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.5" % Test
libraryDependencies += "org.easymock" % "easymock" % "3.5" % Test
libraryDependencies += "org.cassandraunit" % "cassandra-unit" % "3.3.0.2" % Test
libraryDependencies += "org.pegdown" % "pegdown" % "1.6.0" % Test
libraryDependencies += "com.typesafe.akka" %% "akka-testkit" % "2.5.11" % Test

resolvers += Resolver.mavenLocal
resolvers += Resolver.mavenCentral
Expand Down
8 changes: 4 additions & 4 deletions src/main/scala/com/datastax/gatling/plugin/DseProtocol.scala
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@ import java.util.concurrent.atomic.AtomicLong

import akka.Done
import akka.actor.ActorSystem
import com.datastax.driver.dse.DseSession
import com.datastax.gatling.plugin.metrics.MetricsLogger
import com.datastax.gatling.plugin.request.{CqlRequestActionBuilder, GraphRequestActionBuilder}
import com.datastax.gatling.plugin.utils.GatlingTimingSource
import com.datastax.oss.driver.api.core.CqlSession
import com.typesafe.scalalogging.StrictLogging
import io.gatling.core.CoreComponents
import io.gatling.core.config.GatlingConfiguration
Expand Down Expand Up @@ -63,7 +63,7 @@ object DseProtocol extends StrictLogging {
}
}

case class DseProtocol(session: DseSession) extends Protocol
case class DseProtocol(session: CqlSession) extends Protocol

object DseComponents {
private val componentsCache = mutable.Map[ActorSystem, DseComponents]()
Expand Down Expand Up @@ -126,10 +126,10 @@ case class DseComponents(dseProtocol: DseProtocol,


object DseProtocolBuilder {
def session(session: DseSession) = DseProtocolBuilder(session)
def session(session: CqlSession) = DseProtocolBuilder(session)
}

case class DseProtocolBuilder(session: DseSession) {
case class DseProtocolBuilder(session: CqlSession) {
def build = DseProtocol(session)
}

4 changes: 2 additions & 2 deletions src/main/scala/com/datastax/gatling/plugin/Predef.scala
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ trait DsePredefBase extends DseCheckSupport {

implicit def protocolBuilder2DseProtocol(builder: DseProtocolBuilder): DseProtocol = builder.build

implicit def cqlRequestAttributes2ActionBuilder(builder: DseCqlAttributesBuilder): ActionBuilder = builder.build()
implicit def cqlRequestAttributes2ActionBuilder(builder: DseCqlAttributesBuilder[_,_]): ActionBuilder = builder.build()

implicit def graphRequestAttributes2ActionBuilder(builder: DseGraphAttributesBuilder): ActionBuilder = builder.build()
implicit def graphRequestAttributes2ActionBuilder(builder: DseGraphAttributesBuilder[_, _]): ActionBuilder = builder.build()
}

/**
Expand Down
51 changes: 5 additions & 46 deletions src/main/scala/com/datastax/gatling/plugin/checks/CqlChecks.scala
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,15 @@

package com.datastax.gatling.plugin.checks

import com.datastax.driver.core.{ResultSet, Row}
import com.datastax.oss.driver.api.core.cql.{AsyncResultSet, Statement, StatementBuilder}
import com.datastax.gatling.plugin.response.CqlResponse
import io.gatling.commons.validation.{SuccessWrapper, Validation}
import io.gatling.core.check._
import io.gatling.core.check.extractor.{CountArity, CriterionExtractor, Extractor, FindAllArity, FindArity, SingleArity, _}
import io.gatling.core.check.extractor.{Extractor, SingleArity}
import io.gatling.core.session.{Expression, ExpressionSuccessWrapper, Session}

import scala.collection.mutable


/**
* This class serves as model for the CQL-specific checks.
*
Expand Down Expand Up @@ -51,50 +50,10 @@ private class CqlResponseExtractor[X](val name: String,
}
}

private abstract class ColumnValueExtractor[X] extends CriterionExtractor[CqlResponse, Any, X] {
val criterionName = "columnValue"
}

private class SingleColumnValueExtractor(val criterion: String, val occurrence: Int) extends ColumnValueExtractor[Any] with FindArity {
def extract(response: CqlResponse): Validation[Option[Any]] =
response.getCqlResultColumnValues(criterion).lift(occurrence).success
}

private class MultipleColumnValueExtractor(val criterion: String) extends ColumnValueExtractor[Seq[Any]] with FindAllArity {
def extract(response: CqlResponse): Validation[Option[Seq[Any]]] =
response.getCqlResultColumnValues(criterion).liftSeqOption.success
}

private class CountColumnValueExtractor(val criterion: String) extends ColumnValueExtractor[Int] with CountArity {
def extract(response: CqlResponse): Validation[Option[Int]] =
response.getCqlResultColumnValues(criterion).liftSeqOption.map(_.size).success
}

object CqlChecks {
val resultSet =
new CqlResponseExtractor[ResultSet](
val resultSet:CqlCheckBuilder[AsyncResultSet] =
new CqlResponseExtractor[AsyncResultSet](
"resultSet",
r => r.getCqlResultSet)
.toCheckBuilder

val allRows =
new CqlResponseExtractor[Seq[Row]](
"allRows",
r => r.getAllRows)
r => r.resultSet)
.toCheckBuilder

val oneRow =
new CqlResponseExtractor[Row](
"oneRow",
r => r.getOneRow)
.toCheckBuilder

def columnValue(columnName: Expression[String]) = {
val cqlResponseExtender: Extender[DseCqlCheck, CqlResponse] = wrapped => DseCqlCheck(wrapped)
new DefaultMultipleFindCheckBuilder[DseCqlCheck, CqlResponse, CqlResponse, Any](cqlResponseExtender, x => x.success) {
def findExtractor(occurrence: Int) = columnName.map(new SingleColumnValueExtractor(_, occurrence))
def findAllExtractor = columnName.map(new MultipleColumnValueExtractor(_))
def countExtractor = columnName.map(new CountColumnValueExtractor(_))
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,49 +6,75 @@

package com.datastax.gatling.plugin.checks

import io.gatling.core.session.ExpressionSuccessWrapper
import com.datastax.dse.driver.api.core.graph.AsyncGraphResultSet
import com.datastax.oss.driver.api.core.cql.AsyncResultSet
import com.datastax.gatling.plugin.utils.ResultSetUtils

/**
* Make both CQL and Graph checks available to the DSL.
*
* Note that as of 1.3.5 (and the upgrade to the unified OSS driver it brings along) the API here has changed.
* The old check API exposed a rich set of checks for various operations including row counts and validating
* data in individual rows. Several (most?) of these checks were built on the idea that all rows were immediately
* available in memory. This design has changed in 1.3.5, so maintaining the old API would've proven quite difficult
* (if not impossible). Additionally, the rich API shields the user from the intricacies of the driver API at the
* cost of limited flexibility; implementing new functionality requires modifications to the plugin itself (or at
* least an awareness of it's innards).
*
* With 1.3.5 this relationship has been inverted. The check API has been reduced to a single check which makes the
* underlying [[AsyncResultSet]] or [[AsyncGraphResultSet]] available. Simulations can then use transform() to
* extract values and evaluate them as necessary. So, for instance, something like this:
*
* {{{
* .check(columnValue("counter_type") is 2)
* }}}
*
* now becomes:
*
* {{{
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* }}}
*
* Note that these transforms are now managed as Scala code within the simulations so they can be abstracted and
* built into libraries which can be re-used across simulations. Also note that this abstraction can be implemented
* without modifying the plugin itself.
*
* A similar pattern applies to checks based on metadata. So this:
*
* {{{
* .check(exhausted is true)
* }}}
*
* now becomes:
*
* {{{
* .check(resultSet.transform(_.hasMorePages) is false)
* .check(resultSet.transform(_.remaining) is 0)
* }}}
*
* At this point we should also note that checks are now explicitly executed in the order in which hey are declared
* in the simulation. This matters because iterating through rows will impact methods such as remaining(). So, for
* example, if you want to validate that a single row was returned and it contained a specific value you should do
* something like:
*
* {{{
* .check(resultSet.transform(_.remaining) is 1)
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* }}}
*
* and not:
*
* {{{
* .check(resultSet.transform(rs => rs.one().getLong("counter_type")) is 2L)
* .check(resultSet.transform(_.remaining) is 1)
* }}}
*
* Finally, in general the expectation is that you won't need to realize all rows in a result set, but if for some
* reason you find this necessary this functionality is supplied in [[ResultSetUtils]]. This class also serves as
* an example of the kind of abstraction over common extraction operations discussed above.
*/
trait DseCheckSupport {

// start global checks
lazy val exhausted = GenericChecks.exhausted
lazy val applied = GenericChecks.applied
lazy val rowCount = GenericChecks.rowCount

// execution info and subsets
lazy val executionInfo = GenericChecks.executionInfo
lazy val achievedCL = GenericChecks.achievedConsistencyLevel
lazy val pagingState = GenericChecks.pagingState
lazy val queriedHost = GenericChecks.queriedHost
lazy val schemaAgreement = GenericChecks.schemaInAgreement
lazy val successfulExecutionIndex = GenericChecks.successfulExecutionIndex
lazy val triedHosts = GenericChecks.triedHosts
lazy val warnings = GenericChecks.warnings

// start CQL only checks
lazy val resultSet = CqlChecks.resultSet
lazy val allRows = CqlChecks.allRows
lazy val oneRow = CqlChecks.oneRow

// start Graph only checks
lazy val graphResultSet = GraphChecks.graphResultSet
lazy val allNodes = GraphChecks.allNodes
lazy val oneNode = GraphChecks.oneNode

def edges(columnName: String) = GraphChecks.edges(columnName)

def vertexes(columnName: String) = GraphChecks.vertexes(columnName)

def paths(columnName: String) = GraphChecks.paths(columnName)

def properties(columnName: String) = GraphChecks.paths(columnName)

def vertexProperties(columnName: String) = GraphChecks.vertexProperties(columnName)

/**
* Get a column by name returned by the CQL statement.
* Note that this statement implicitly fetches <b>all</b> rows from the result set!
*/
def columnValue(columnName: String) = CqlChecks.columnValue(columnName.expressionSuccess)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removal of the checks above reflects an API change. Going forward the intent is to access the underlying ResultSet impl directly (via the appropriate "resultSet" check) and then extract the desired information via .transform() + fns defined somewhere in the code. Users can define these transform fns in their own simulations, in library code, etc. This has the effect of making the check API simpler and more streamlined while allowing users to extend it in a natural way by defining their own fns.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the simplification goal. And it is a good time to do this, given that all user simulations will break anyway, because of the DseCluster/DseSession class change in the 4.x branch. May I ask for a small documentation where this decision is explained and the replacement code is provided for deleted checks, to make things easier for users?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extensive scaladoc has been added on this point in DseCheckSupport. @pingtimeout would you mind taking a look and confirming that this addresses the points you wanted to see covered?

lazy val graphResultSet = GraphChecks.resultSet
}

129 changes: 0 additions & 129 deletions src/main/scala/com/datastax/gatling/plugin/checks/GenericChecks.scala

This file was deleted.

Loading