add Spanner package #1491

clairemcginty · 2018-10-29T18:14:19Z

Currently supports writes from Mutations and reads into Structs. I have the spanner type macros saved for sometime further down the line.

notes:

Spanner integration test is kind of slow, takes 37 seconds
SpannerIO's testId doesn't support a unit having 2 spanner reads/writes to the same DB since it's just keyed by the SpannerConfig params, which don't include table name. For writes the table name is keyed into each Mutation object and reads give you a choice of specifying a table name or a literal query
Haven't implemented Taps yet

nevillelyh · 2018-10-29T19:20:51Z

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala

+      config.getInstanceId.get(),
+      config.getDatabaseId.get(),
+      List(
+        s"CREATE TABLE ${tablePrefix}_1 (\n Key INT64, \n Value STRING(MAX) \n) PRIMARY KEY (Key)",


nit: do we really need the \ns here? Kinda makes it hard to read.

nevillelyh · 2018-10-29T19:24:31Z

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala

+  import SpannerIOIT._
+  implicit val ec: ExecutionContext = ExecutionContext.global
+
+  override def beforeAll(): Unit = {


Also make a helper function for table creation and move the logic to each individual tests? We can also change _[123] to something more meaningful in each test to make it more readable.

I thought it made since to batch all the Spanner CREATE statements for the sake of simplicity/performance -- I refactored it a bit to provide a Spanner table context per test that handles all the other setup logic

👍 no strong preference. Either way works.

nevillelyh · 2018-10-29T19:26:07Z

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala

+        .set("Value").to("bar")
+        .build())
+
+    val awaited = Await.result(result, 10.seconds)


What's the reason for these timeouts?

nevillelyh · 2018-10-29T19:27:47Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/SpannerIO.scala

+    throw new IllegalStateException("SpannerRead is read-only")
+
+  override def tap(params: ReadP): Tap[Nothing] =
+    throw new NotImplementedError("SpannerRead tap is not implemented")


Should be EmptyTap?

I chose the NotImplementedError because I thought it was something that actually should be implemented in the future for the SpannerRead op. what do you think?

Talked IRL. Database source/sinks are not immutable/atomic so reading a tap is not guaranteed to return the same data in the original SCollection.

nevillelyh · 2018-10-29T19:32:52Z

scio-spanner/src/test/scala/com/spotify/scio/spanner/SpannerIOTest.scala

+    .withInstanceId("someInstance")
+}
+
+class SpannerIOTest extends PipelineSpec with Matchers {


We have a ScioIOSpec so you don't have to handcraft test jobs ;)
https://github.com/spotify/scio/blob/master/scio-test/src/main/scala/com/spotify/scio/testing/ScioIOSpec.scala

Example:
https://github.com/spotify/scio/blob/master/scio-bigtable/src/test/scala/com/spotify/scio/bigtable/BigtableIOTest.scala

regadas · 2018-10-29T20:09:07Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/SpannerIO.scala

+  }
+
+  sealed trait ReadMethod
+  case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod


@clairemcginty

Suggested change

case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod

final case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod

regadas · 2018-10-29T20:09:47Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/SpannerIO.scala

+
+  sealed trait ReadMethod
+  case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod
+  case class FromQuery(query: String) extends ReadMethod


@clairemcginty

Suggested change

case class FromQuery(query: String) extends ReadMethod

final case class FromQuery(query: String) extends ReadMethod

regadas · 2018-10-29T20:12:47Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/SpannerIO.scala

+
+  override protected def write(data: SCollection[Mutation],
+                               params: WriteP): Future[Tap[Nothing]] = {
+    var transform = BSpannerIO.write().withSpannerConfig(config)


@clairemcginty can you do it like this?

Suggested change

var transform = BSpannerIO.write().withSpannerConfig(config)

val transform = BSpannerIO

.write()

.withSpannerConfig(config)

.withBatchSizeBytes(params.batchSizeBytes)

.withFailureMode(params.failureMode)

regadas · 2018-10-29T20:13:51Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/client/Spanner.scala

+import org.apache.beam.sdk.io.gcp.spanner.SpannerConfig
+
+object Spanner {
+  lazy val defaultInstance: Spanner = {


@clairemcginty

Suggested change

lazy val defaultInstance: Spanner = {

private lazy val defaultInstance: Spanner = {

regadas · 2018-10-29T20:14:34Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/client/Spanner.scala

+    SpannerOptions.newBuilder().build().getService
+  }
+
+  def getDatabaseClient(config: SpannerConfig,


Suggested change

def getDatabaseClient(config: SpannerConfig,

def databaseClient(config: SpannerConfig,

regadas · 2018-10-29T20:14:49Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/client/Spanner.scala

+      ))
+  }
+
+  def getAdminClient(project: String, instance: Spanner = defaultInstance): DatabaseAdminClient =


Suggested change

def getAdminClient(project: String, instance: Spanner = defaultInstance): DatabaseAdminClient =

def adminClient(project: String, instance: Spanner = defaultInstance): DatabaseAdminClient =

regadas · 2018-10-29T20:15:28Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/package.scala

+
+package object spanner {
+
+  implicit class SpannerScioContext(@transient val self: ScioContext) extends Serializable {


Suggested change

implicit class SpannerScioContext(@transient val self: ScioContext) extends Serializable {

implicit class SpannerScioContext(@transient private val self: ScioContext) extends Serializable {

regadas · 2018-10-29T20:16:40Z

scio-spanner/src/main/scala/com/spotify/scio/spanner/package.scala

+    }
+  }
+
+  implicit class SpannerSCollection(@transient val self: SCollection[Mutation])


Suggested change

implicit class SpannerSCollection(@transient val self: SCollection[Mutation])

implicit class SpannerSCollection(@transient private val self: SCollection[Mutation])

regadas

Looking good 👍 There are some suggestions.

codecov · 2018-10-30T15:38:32Z

Codecov Report

Merging #1491 into master will increase coverage by 0.07%.
The diff coverage is 88.37%.

@@            Coverage Diff             @@
##           master    #1491      +/-   ##
==========================================
+ Coverage   79.05%   79.12%   +0.07%     
==========================================
  Files         170      173       +3     
  Lines        5208     5251      +43     
  Branches      415      411       -4     
==========================================
+ Hits         4117     4155      +38     
- Misses       1091     1096       +5

Impacted Files	Coverage Δ
...com/spotify/scio/coders/instances/JavaCoders.scala	`85.71% <100%> (+0.86%)`	⬆️
.../main/scala/com/spotify/scio/spanner/package.scala	`100% <100%> (ø)`
...cala/com/spotify/scio/spanner/client/Spanner.scala	`100% <100%> (ø)`
...ain/scala/com/spotify/scio/spanner/SpannerIO.scala	`81.48% <81.48%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bef6571...e906126. Read the comment docs.

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala

* spanner IO first pass * clean up * add to root project * make scala 2.11 happy * remove redundant implicits * address comments * SpannerRead returns EmptyTap * refactor integration test

clairemcginty added 5 commits October 29, 2018 11:53

spanner IO first pass

bd79dfc

clean up

153e081

add to root project

727bdfa

make scala 2.11 happy

37695f9

remove redundant implicits

66ddf33

clairemcginty requested review from regadas and nevillelyh October 29, 2018 18:14

nevillelyh reviewed Oct 29, 2018

View reviewed changes

regadas reviewed Oct 29, 2018

View reviewed changes

address comments

a1bc3e3

SpannerRead returns EmptyTap

f465418

regadas reviewed Oct 30, 2018

View reviewed changes

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala Outdated Show resolved Hide resolved

regadas reviewed Oct 30, 2018

View reviewed changes

scio-spanner/src/it/scala/com/spotify/scio/spanner/SpannerIOIT.scala Outdated Show resolved Hide resolved

refactor integration test

e906126

regadas approved these changes Oct 30, 2018

View reviewed changes

clairemcginty merged commit b012c4f into master Oct 30, 2018

regadas deleted the claire/spanner_io branch October 30, 2018 19:44

This was referenced Oct 30, 2018

QUESTION: Google Cloud Spanner Support #903

Closed

Interface for reading and writing to Google Cloud Spanner. #1163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add Spanner package #1491

add Spanner package #1491

clairemcginty commented Oct 29, 2018

nevillelyh Oct 29, 2018

nevillelyh Oct 29, 2018

clairemcginty Oct 30, 2018

nevillelyh Oct 30, 2018

nevillelyh Oct 29, 2018

nevillelyh Oct 29, 2018

clairemcginty Oct 30, 2018

nevillelyh Oct 30, 2018

nevillelyh Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018

regadas Oct 29, 2018 •

edited

regadas Oct 29, 2018

regadas left a comment

codecov bot commented Oct 30, 2018 •

edited

	case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod
	final case class FromTable(tableName: String, columns: Seq[String]) extends ReadMethod

	case class FromQuery(query: String) extends ReadMethod
	final case class FromQuery(query: String) extends ReadMethod

	lazy val defaultInstance: Spanner = {
	private lazy val defaultInstance: Spanner = {

	def getDatabaseClient(config: SpannerConfig,
	def databaseClient(config: SpannerConfig,

	def getAdminClient(project: String, instance: Spanner = defaultInstance): DatabaseAdminClient =
	def adminClient(project: String, instance: Spanner = defaultInstance): DatabaseAdminClient =


		package object spanner {

		implicit class SpannerScioContext(@transient val self: ScioContext) extends Serializable {

	implicit class SpannerSCollection(@transient val self: SCollection[Mutation])
	implicit class SpannerSCollection(@transient private val self: SCollection[Mutation])

add Spanner package #1491

add Spanner package #1491

Conversation

clairemcginty commented Oct 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

regadas Oct 29, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

regadas left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 30, 2018 • edited

Codecov Report

regadas Oct 29, 2018 •

edited

codecov bot commented Oct 30, 2018 •

edited