Publish AnalyzedItems to Cassandra by c-w · Pull Request #28 · CatalystCode/project-fortis-spark

c-w · 2017-06-23T17:10:11Z

For now, using a test keyspace and table on the cluster that Erik set up:

CREATE KEYSPACE fortistest WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE fortistest.events (
    created_at timestamp,
    pipeline text,
    PRIMARY KEY (created_at, pipeline)
)

There will be some follow-up work required to adapt the CassandraSchema class once our events schema is finalized.

Resolves #14

We can only have a single Spark Context running per JVM so splitting the tests out over multiple specs is more hassle than it's worth spending time to figure out right now.

Smarker

Everything else looks good.

Smarker · 2017-06-23T19:21:10Z

+
+object Utils {
+  def mean(items: List[Double]): Double = {
+    items.sum / items.length


What if the length of items is 0? Wouldn't this cause an error?

Improved error handling in 37c1983.

Smarker · 2017-06-23T19:21:58Z

+  def rescale(items: List[Double], min_new: Double, max_new: Double): List[Double] = {
+    val min_old = items.min
+    val max_old = items.max
+    val coef = (max_new - min_new) / (max_old - min_old)


If max_old == min_old, the denominator would be 0. Maybe put a check for this?

Added error handling in 37c1983.

jcjimenez · 2017-06-23T19:33:38Z

@@ -0,0 +1,6 @@
+package com.microsoft.partnercatalyst.fortis.spark.transforms.gender
+
+object GenderDetector {


May I suggest adding extends Enumeration here?

Sweet. Done in 8162fe3.

jcjimenez

LGTM with the division by zero check

erikschlegel

Looks great. Nice work. Left some comments around offloading the uuid() call to cassandra.

Starting next week, I'll add the pieces to aggregate the results by place, tile and topic and make the saveToCassnadra call for each.

erikschlegel · 2017-06-23T22:07:49Z

    import transformContext._

    stream.map(tada => AnalyzedItem(
+      id = randomUUID(),


This should be called via the uuid() function in cassandra.

The advantage of creating the id early is that we have a way to track every event through the pipeline (e.g. useful when logging). Is this benefit worth explicitly creating the UUID?

erikschlegel · 2017-06-23T22:09:29Z


  private def convertToSchema(stream: DStream[RadioTranscription], transformContext: TransformContext): DStream[AnalyzedItem] = {
    stream.map(transcription => AnalyzedItem(
+      id = randomUUID(),


This should be called via the uuid() function in cassandra.

erikschlegel · 2017-06-23T22:09:38Z

        // do computer vision analysis
        val analysis = imageAnalyzer.analyze(instagram.images.standard_resolution.url)
        AnalyzedItem(
+          id = randomUUID(),


This should be called via the uuid() function in cassandra.

kevinhartman · 2017-06-24T02:36:42Z

 import scala.collection.mutable

-class StreamProviderSpec extends FlatSpec with BeforeAndAfter {
+class SparkSpec extends FlatSpec with BeforeAndAfter {


I would ideally like to keep the StreamProviderSpec separate since the StreamProvider package is an isolated component that could be moved out to a library at any time (and it'd be nice to take the test spec with it).

As per the comment on 629e5d3, we can only have a single Spark Context running per JVM so I merged all the tests for now (they were previously split but it's non-trivial to get that to work). If this becomes an issue, we can spend the time to figure out how to split the tests.

I briefly looked into using the spark-testing-base package but they don't have anything built-in for Spark Streaming. I started working on a streaming extension but got some odd errors so preferred to just push this for now. I'll look more into this when I get a free minute.

c-w added 12 commits June 23, 2017 10:06

Add wiring for Cassandra sink

bbd3053

Simplify calling of Cassandra sink

3c84f12

Fill-in real value for keyspaceName

7582e35

Integrate with test schema

c75e93f

Fix typo

4e30b78

Fix partial match warning

d6ff036

Fix null pointer

0808459

Add implicit cassandra dependency

1a2d9bf

Add test to verify cassandra connection

f1a5ad4

Simplify env/optional handling

3738559

Merge all spark tests into one spec

629e5d3

We can only have a single Spark Context running per JVM so splitting the tests out over multiple specs is more hassle than it's worth spending time to figure out right now.

Add todos for future work

fb867d9

c-w added the in progress label Jun 23, 2017

Smarker reviewed Jun 23, 2017

View reviewed changes

c-w added 2 commits June 23, 2017 12:25

Implement production cassandra schema

e736898

Avoid hard-coding male/female values

f34f55a

c-w force-pushed the publish-to-cassandra branch from e13eef1 to f34f55a Compare June 23, 2017 19:28

jcjimenez reviewed Jun 23, 2017

View reviewed changes

jcjimenez approved these changes Jun 23, 2017

View reviewed changes

c-w added 5 commits June 23, 2017 12:36

Improve error handling

37c1983

Handle empty rescale

a5ed3f5

Make formatting consistent

32d0701

Fix typo

a46c701

Make enums explicit

8162fe3

jcjimenez merged commit 257576d into master Jun 23, 2017

jcjimenez removed the in progress label Jun 23, 2017

c-w deleted the publish-to-cassandra branch June 23, 2017 20:15

erikschlegel reviewed Jun 23, 2017

View reviewed changes

kevinhartman reviewed Jun 24, 2017

View reviewed changes

		@@ -0,0 +1,6 @@
		package com.microsoft.partnercatalyst.fortis.spark.transforms.gender

		object GenderDetector {

Conversation

c-w commented Jun 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Smarker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

c-w Jun 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcjimenez left a comment

Choose a reason for hiding this comment

Uh oh!

erikschlegel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

c-w commented Jun 23, 2017 •

edited

Loading

c-w Jun 23, 2017 •

edited

Loading