Skip to content

Commit

Permalink
Feature/fix_pagination (#612)
Browse files Browse the repository at this point in the history
* Adding tests for renaming columns

* Adding serialization tests for alter additions

* Implementing list pagination

* Adding truncation

* Adding pagination tests

* Adding a readme update

* Cleaning up the implementation of timeuuid test records to prevent race conditions

* Adding order iterator tests

* Adding pagination that can deal explicitly with collections

* Adding better docs for select queries
  • Loading branch information
alexflav23 committed Jan 24, 2017
1 parent 90993a3 commit 6b3cd9b
Show file tree
Hide file tree
Showing 15 changed files with 410 additions and 167 deletions.
72 changes: 36 additions & 36 deletions README.md
Expand Up @@ -10,40 +10,6 @@ If you use phantom, please consider adding your company to our list of adopters.

![phantom](https://s3-eu-west-1.amazonaws.com/websudos/oss/logos/phantom.png "Outworkers Phantom")

2.0.0 Migration guide
=====================

As a word of introduction, this guide is brand new and there may be certain elements we have currently left out. Phantom has an immense adopter base which includes many of you using the library in ways which we do not know of. 2.0.0 completely replaces fundemental aspects of the framework to provide superior performance and reliability, and we have tested back and forth to ensure the smoothest possible transition, but please feel free to report any issues via GitHub and we will fix them straight away.

- The OSS version of phantom has as of 2.0.0 returned to the Apache V2 license and the license is here to stay.
- All packages and dependencies are now available under the `com.outworkers` organisation instead of `com.websudos`. As
part of long term re-branding efforts, we have finally felt it's time to make sure the change is consistent throughout.
- There is a new and now completely optional Bintray resolver, `Resolver.bintrayRepo("outworkers", "oss-releases")`,
that gives you free access to the latest cuts of our open source releases before they hit Maven Central. We assume
no liability for your usage of latest cuts, but we welcome feedback and we do our best to have elaborate CI processes in place.
- Manually defining a `fromRow` inside a `CassandraTable` is no longer required if your column types match your case class types.
- `EnumColumn` is now relying entirely on `Primitive.macroImpl`, which means you will not need to pass in the enumeration
as an argument to `EnumColumn` anymore. This means `object enum extends EnumColumn(this, enum: MyEnum)` is now simply
`object enum extends EnumColumn[MyEnum#Value](this)`
- All dependencies are now being published to Maven Central. This includes outworkers util and outworkers diesel,
projects which have in their own right been completely open sourced under Apache V2 and made public on GitHub.
- All dependencies on `scala-reflect` have been completely removed.
- A new, macro based mechanism now performs the same auto-discovery task that reflection used to, thanks to `macro-compat`.
- Index modifiers no longer require a type parameter, `PartitionKey`, `PrimaryKey`, `ClusteringOrder` and `Index` don't require
the column type passed anymore.
- `KeySpaceDef` has been renamed to the more appropiate `
CassandraConnector`.
- `CassandraConnector` now natively supports specifying a keyspace creation query.
- `TimeWindowCompactionStrategy` is now natively supported in the CREATE/ALTER dsl.
- Collections can now be used as part of a primary or partition key.
- Tuples are now natively supported as valid types via `TupleColumn`.
- `phantom-reactivestreams` is now simply called `phantom-streams`.
- `Database.autocreate` and `Database.autotruncate` are now no longer accessible. Use `create`, `createAsync`, `truncate` and `truncateAsync` instead.
- `Database` now requires an f-bounded type argument: `class MyDb(override val connector: CassandraConnection) extends Database[MyDb](connector)`.
- Automated Cassandra pagination via paging states has been moved to a new method called `paginateRecord`. Using `fetchRecord` with a `PagingState` is no longer possible.
This is done to distinguish the underlying consumer mechanism of parsing and fetching records from Cassandra.
- `com.outworkers.phantom.dsl.context` should be used instead of `scala.concurrent.ExecutionContext.Implicits.global`.

Available modules
=================

Expand Down Expand Up @@ -71,7 +37,6 @@ We publish phantom in 2 formats, stable releases and bleeding edge.

- Intermediary releases are available through our managed Bintray repository available at `https://dl.bintray.com/outworkers/oss-releases/`. The latest version available on our Bintray repository is indicated by the Bintray badge at the top of this readme.


### How phantom compares

To compare phantom to similar tools in the Scala/Cassandra category, you can read more [here](https://github.com/outworkers/phantom/blob/develop/comparison.md).
Expand All @@ -83,6 +48,41 @@ The latest versions are available here. The badges automatically update when a n
- Latest stable version: [![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.outworkers/phantom-dsl_2.11/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.outworkers/phantom-dsl_2.11) (Maven Central)
- Bleeding edge: [![Bintray](https://api.bintray.com/packages/outworkers/oss-releases/phantom-dsl/images/download.svg)](https://bintray.com/outworkers/oss-releases/phantom-dsl/_latestVersion) (OSS releases on Bintray)


2.0.0 Migration guide
=====================

As a word of introduction, this guide is brand new and there may be certain elements we have currently left out. Phantom has an immense adopter base which includes many of you using the library in ways which we do not know of. 2.0.0 completely replaces fundemental aspects of the framework to provide superior performance and reliability, and we have tested back and forth to ensure the smoothest possible transition, but please feel free to report any issues via GitHub and we will fix them straight away.

- The OSS version of phantom has as of 2.0.0 returned to the Apache V2 license and the license is here to stay.
- All packages and dependencies are now available under the `com.outworkers` organisation instead of `com.websudos`. As
part of long term re-branding efforts, we have finally felt it's time to make sure the change is consistent throughout.
- There is a new and now completely optional Bintray resolver, `Resolver.bintrayRepo("outworkers", "oss-releases")`,
that gives you free access to the latest cuts of our open source releases before they hit Maven Central. We assume
no liability for your usage of latest cuts, but we welcome feedback and we do our best to have elaborate CI processes in place.
- Manually defining a `fromRow` inside a `CassandraTable` is no longer required if your column types match your case class types.
- `EnumColumn` is now relying entirely on `Primitive.macroImpl`, which means you will not need to pass in the enumeration
as an argument to `EnumColumn` anymore. This means `object enum extends EnumColumn(this, enum: MyEnum)` is now simply
`object enum extends EnumColumn[MyEnum#Value](this)`
- All dependencies are now being published to Maven Central. This includes outworkers util and outworkers diesel,
projects which have in their own right been completely open sourced under Apache V2 and made public on GitHub.
- All dependencies on `scala-reflect` have been completely removed.
- A new, macro based mechanism now performs the same auto-discovery task that reflection used to, thanks to `macro-compat`.
- Index modifiers no longer require a type parameter, `PartitionKey`, `PrimaryKey`, `ClusteringOrder` and `Index` don't require
the column type passed anymore.
- `KeySpaceDef` has been renamed to the more appropiate `
CassandraConnector`.
- `CassandraConnector` now natively supports specifying a keyspace creation query.
- `TimeWindowCompactionStrategy` is now natively supported in the CREATE/ALTER dsl.
- Collections can now be used as part of a primary or partition key.
- Tuples are now natively supported as valid types via `TupleColumn`.
- `phantom-reactivestreams` is now simply called `phantom-streams`.
- `Database.autocreate` and `Database.autotruncate` are now no longer accessible. Use `create`, `createAsync`, `truncate` and `truncateAsync` instead.
- `Database` now requires an f-bounded type argument: `class MyDb(override val connector: CassandraConnection) extends Database[MyDb](connector)`.
- Automated Cassandra pagination via paging states has been moved to a new method called `paginateRecord`. Using `fetchRecord` with a `PagingState` is no longer possible.
This is done to distinguish the underlying consumer mechanism of parsing and fetching records from Cassandra.
- `com.outworkers.phantom.dsl.context` should be used instead of `scala.concurrent.ExecutionContext.Implicits.global`.

### What got completed in Phantom 2.0.0

With the rapidly evolving requirements, Cassandra releases, and competition, it was only natural we kept Phantom up to scratch. In line with a lot of user feedback, the priorities of 2.0.0 were:
Expand Down Expand Up @@ -117,7 +117,7 @@ Feedback and contributions are welcome, and we are happy to prioritise any cruci

#### Tech debt

- [ ] Correctly implement Cassandra pagination using iterators, currently setting a `fetchSize` on a query does not correctly propagate or consume the resulting iterator, which leads to API inconsistencies and `PagingState` not being set on any `ResultSet`.
- [x] Correctly implement Cassandra pagination using iterators, currently setting a `fetchSize` on a query does not correctly propagate or consume the resulting iterator, which leads to API inconsistencies and `PagingState` not being set on any `ResultSet`.
- [ ] Add a build matrix that will test phantom against multiple versions of Cassandra in Travis for Scala 2.11, with support for all major releases of Cassandra.
- [ ] Bump code coverage up to 100%

Expand Down
59 changes: 59 additions & 0 deletions docs/querying/select.md
@@ -0,0 +1,59 @@
### Select queries

This explores all the various `SELECT` functionality in CQL as described in the official reference [here](http://cassandra.apache.org/doc/latest/cql/dml.html#select).
Phantom aims to provide a 100% mapping with all the latest CQL features, but if you find anything that you need missing,
please help us by reporting it through GitHub issues.

#### Pre-requisites and setup.

To better explain the document to follow, it is easier if we reach common ground by refering to the same Cassandra table.
The below example is a simple Cassandra table with a more complex key that allows us to explore all the features of phantom
and demonstrate the available select API.

We will create a `AnalyticsEntries` table, where we hold information about a car's state over time for 2 properties we
care about, namely the `velocity` and `tirePressure`. We leverage the `TimeUUID` Cassandra type to store information
about timestamps, order the logs we receive in descending order(most recent record first), and prevent any colissions.

If we would just use a timestamp type, if we were to receive two logs for the same car at the exact same timestamp,
the entries would override each other in Cassandra, because in effect they would have the same partition key
and the same clustering key, so the whole primary key would be identical.

```scala
import com.outworkers.phantom.dsl._

case class CarMetric(
car: UUID,
id: UUID,
velocity: Double,
tirePressure: Double
)

abstract class AnalyticsEntries extends CassandraTable[AnalyticsEntries, CarMetric] with RootConnector {
object car extends UUIDColumn(this) with PartitionKey
object id extends TimeUUIDColumn(this) with ClusteringOrder with Descending
object velocity extends DoubleColumn(this)
object tirePressure extends DoubleColumn(this)
}

```

#### Available query methods.

The following is the list of available query methods on a select, and it can be used to leverage standard `SELECT` functionality
in various ways.


| Method name | Return type | Purpose |
| =========== | =========== | ======= |



#####



#### Paginating results by leveraging paging states and automated Cassandra pagination.

There are situations where you can not retrieve a whole list of results in a single go, and for that reason
Cassandra offers paging states and automated pagination. Phantom makes that functionality available through a set of overloaded
methods called `paginateRecord`.
Expand Up @@ -16,7 +16,9 @@
package com.outworkers.phantom

import com.datastax.driver.core.{Row, Session}
import com.outworkers.phantom.builder.QueryBuilder
import com.outworkers.phantom.builder.clauses.DeleteClause
import com.outworkers.phantom.builder.primitives.Primitive
import com.outworkers.phantom.builder.query.{RootCreateQuery, _}
import com.outworkers.phantom.column.AbstractColumn
import com.outworkers.phantom.connectors.KeySpace
Expand Down Expand Up @@ -91,6 +93,20 @@ abstract class CassandraTable[T <: CassandraTable[T, R], R](

final def alter()(implicit keySpace: KeySpace): AlterQuery.Default[T, R] = AlterQuery(instance)

final def alter[
RR,
NewType
](columnSelect: T => AbstractColumn[RR])(newType: Primitive[NewType])(implicit keySpace: KeySpace): AlterQuery.Default[T, RR] = {
AlterQuery.alterType[T, RR, NewType](instance, columnSelect, newType)
}

final def alter[RR](
columnSelect: T => AbstractColumn[RR],
newName: String
)(implicit keySpace: KeySpace): AlterQuery.Default[T, RR] = {
AlterQuery.alterName[T, RR](instance, columnSelect, newName)
}

final def update()(implicit keySpace: KeySpace): UpdateQuery.Default[T, R] = UpdateQuery(instance)

final def insert()(implicit keySpace: KeySpace): InsertQuery.Default[T, R] = InsertQuery(instance)
Expand Down
@@ -0,0 +1,47 @@
/*
* Copyright 2013 - 2017 Outworkers Ltd.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.outworkers.phantom.builder.ops

import java.nio.ByteBuffer
import java.util.UUID

import org.joda.time.DateTime

/**
* Typeclass used to prevent invalid type alterations at compile time.
* Cassandra only allows for specific changes of column types using ALTER Query
* which means we need to somehow enforce the existing mechanism.
*
* We do so by requesting compile time implicit evidence that any type alteration must supply,
* e.g. that Source and Target have a predefined allowed alteration.
*
* The list of allowed ops is found here: [[[https://docs.datastax.com/en/cql/3.1/cql/cql_reference/alter_table_r.html]].
*
* @tparam Source The source type of the column.
* @tparam Target The target type of the column.
*/
sealed trait Alteration[Source, Target]

trait Alterations {
implicit object IntToVarint extends Alteration[Int, BigInt]

implicit object TimeUUIDToVarint extends Alteration[UUID, UUID]

implicit object DoubleToBlob extends Alteration[Double, ByteBuffer]
implicit object IntToBlob extends Alteration[Int, ByteBuffer]
implicit object TimestampToBlob extends Alteration[DateTime, ByteBuffer]
implicit object UUIDToBlob extends Alteration[UUID, ByteBuffer]
}
Expand Up @@ -17,13 +17,12 @@ package com.outworkers.phantom.builder.query

import com.outworkers.phantom.CassandraTable
import com.outworkers.phantom.builder.ops.DropColumn
import com.outworkers.phantom.builder.primitives.Primitive
import com.outworkers.phantom.builder.query.options.{TablePropertyClause, WithBound, WithChainned, WithUnchainned}
import com.outworkers.phantom.builder.{ConsistencyBound, QueryBuilder, Unspecified}
import com.outworkers.phantom.column.AbstractColumn
import com.outworkers.phantom.connectors.KeySpace

import scala.annotation.implicitNotFound

class AlterQuery[
Table <: CassandraTable[Table, _],
Record,
Expand Down Expand Up @@ -146,11 +145,47 @@ object AlterQuery {
* @tparam R The record held in the table.
* @return A raw ALTER query, without any further options set on it.
*/
def apply[T <: CassandraTable[T, _], R](table: T)(implicit keySpace: KeySpace): AlterQuery.Default[T, R] = {
def apply[T <: CassandraTable[T, _], R](table: T)(
implicit keySpace: KeySpace
): AlterQuery.Default[T, R] = {
new AlterQuery[T, R, Unspecified, WithUnchainned](
table,
QueryBuilder.Alter.alter(QueryBuilder.keyspace(keySpace.name, table.tableName).queryString),
QueryOptions.empty
)
}

def alterType[
T <: CassandraTable[T, _],
R,
NewType
](table: T, select: T => AbstractColumn[R], newType: Primitive[NewType])(
implicit keySpace: KeySpace
): AlterQuery.Default[T, R] = {

val qb = QueryBuilder.Alter.alter(
QueryBuilder.keyspace(keySpace.name, table.tableName).queryString
)

new AlterQuery(
table,
QueryBuilder.Alter.alter(qb, select(table).name, newType.cassandraType),
QueryOptions.empty
)
}

def alterName[
T <: CassandraTable[T, _],
R
](table: T, select: T => AbstractColumn[R], newName: String)(
implicit keySpace: KeySpace
): AlterQuery.Default[T, R] = {

val qb = QueryBuilder.Alter.alter(QueryBuilder.keyspace(keySpace.name, table.tableName).queryString)
new AlterQuery(
table,
QueryBuilder.Alter.rename(qb, select(table).name, newName),
QueryOptions.empty
)
}
}

0 comments on commit 6b3cd9b

Please sign in to comment.