Apply DateToUnitCircleTransformer logic in raw feature transformations. #130

marcovivero · 2018-09-13T00:49:54Z

Related issues
There is currently no related issue open for this, this is an enhancement.

Describe the proposed solution
Add the option to apply circular date time transformer in RawFeatureFilter to any Date, DateTime, DateList, or DateMap feature type. This is the same logic utilized for DateToUnitCircleTransformer:

https://github.com/salesforce/TransmogrifAI/blob/master/core/src/main/scala/com/salesforce/op/stages/impl/feature/DateToUnitCircleTransformer.scala

This is currently turned off by default and should be up to user to turn on in their respective workflows.

Describe alternatives you've considered
We currently use the standard approach to processing Numeric types, i.e. just mapping these to doubles. This may lead to issues when comparing different distributions over time.

Additional context
Add any other context about the changes here.

salesforce-cla · 2018-09-13T00:49:57Z

Thanks for the contribution! Unfortunately we can't verify the commit author(s): marcovivero <m***@s***.com>. One possible solution is to add that email to your GitHub account. Alternatively you can change your commits to another email and force push the change. After getting your commits associated with your GitHub account, refresh the status of this Pull Request.

codecov · 2018-09-13T01:16:09Z

Codecov Report

❗ No coverage uploaded for pull request base (master@45241e6). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master     #130   +/-   ##
=========================================
  Coverage          ?   37.87%           
=========================================
  Files             ?      298           
  Lines             ?     9692           
  Branches          ?      552           
=========================================
  Hits              ?     3671           
  Misses            ?     6021           
  Partials          ?        0

Impacted Files	Coverage Δ
...a/com/salesforce/op/filters/RawFeatureFilter.scala	`0% <0%> (ø)`
.../src/main/scala/com/salesforce/op/OpWorkflow.scala	`31.03% <0%> (ø)`
...a/com/salesforce/op/filters/PreparedFeatures.scala	`0% <0%> (ø)`
...ges/impl/feature/DateToUnitCircleTransformer.scala	`0% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 45241e6...104169e. Read the comment docs.

tovbinm · 2018-09-13T03:22:22Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

@@ -30,9 +30,14 @@

 package com.salesforce.op.filters

+
+import java.time.{Instant, OffsetDateTime, ZoneOffset}


please do not use java.time. instead use joda time

tovbinm · 2018-09-13T03:25:31Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

@@ -183,4 +200,38 @@ private[filters] object PreparedFeatures {
   * @return array of string tokens
   */
  private def tokenize(s: String) = TextTokenizer.Analyzer.analyze(s, Language.Unknown)
+
+  private def prepareDateValue(timestamp: Long, timePeriod: Option[TimePeriod]): Double =


can we please make DateToUnitCircle.convertToRandians reuable instead? this is essentially the same code.

tovbinm · 2018-09-13T03:26:51Z

core/src/main/scala/com/salesforce/op/OpWorkflow.scala

@@ -521,12 +522,15 @@ class OpWorkflow(val uid: String = UID[OpWorkflow]) extends OpWorkflowCore {
    maxCorrelation: Double = 0.95,
    correlationType: CorrelationType = CorrelationType.Pearson,
    protectedFeatures: Array[OPFeature] = Array.empty,
-    textBinsFormula: (Summary, Int) => Int = RawFeatureFilter.textBinsFormula
+    protectedJSFeatures: Array[OPFeature] = Array.empty,


how is protectedJSFeatures different from protectedFeatures?

please add docs for protectedJSFeatures param

This was already enabled in RawFeatureFilter, it protects features from JS divergence check. I didn't see it OpWorkflow.withRawFeatureFilter, just making sure it's here as well.

tovbinm · 2018-09-13T03:27:26Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

+    row: Row,
+    responses: Array[TransientFeature],
+    predictors: Array[TransientFeature],
+    timePeriod: Option[TimePeriod]): PreparedFeatures = {


docs for timePeriod

tovbinm · 2018-09-13T03:27:39Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

+  private def prepareFeature[T <: FeatureType](
+    name: String,
+    value: T,
+    timePeriod: Option[TimePeriod]): Map[FeatureKey, ProcessedSeq] =


docs for timePeriod again

tovbinm · 2018-09-13T03:31:11Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

    value match {
      case v: Text => v.value
        .map(s => Map[FeatureKey, ProcessedSeq]((name, None) -> Left(tokenize(s)))).getOrElse(Map.empty)
+      case v: Date => v.value.map { timestamp =>


I wonder why Text, Date and OPNumeric are handled differently than the other values which require some value to be present case ft@SomeValue(_) below? @leahmcguire @marcovivero

I believe this is because all of the values inside case ft@SomeValue(_) already have ft.value as sequences, we'll end up needing to do the same operation to unwrap the Option[Double].

tovbinm · 2018-09-13T03:31:32Z

core/src/main/scala/com/salesforce/op/filters/RawFeatureFilter.scala

@@ -93,7 +94,8 @@ class RawFeatureFilter[T]
  val correlationType: CorrelationType = CorrelationType.Pearson,
  val jsDivergenceProtectedFeatures: Set[String] = Set.empty,
  val protectedFeatures: Set[String] = Set.empty,
-  val textBinsFormula: (Summary, Int) => Int = RawFeatureFilter.textBinsFormula
+  val textBinsFormula: (Summary, Int) => Int = RawFeatureFilter.textBinsFormula,
+  val timePeriod: Option[TimePeriod] = None


docs for timePeriod

also, please add timePeriod to OpWorkflow.withRawFeatureFilter as well

tovbinm · 2018-09-13T03:33:36Z

core/src/test/scala/com/salesforce/op/filters/PreparedFeaturesTest.scala

@@ -157,6 +162,68 @@ class PreparedFeaturesTest extends FlatSpec with TestSparkContext {
    testCorrMatrix(allResponseKeys2, CorrelationType.Spearman, expected)
  }

+  it should "correctly transform date features when time period is specified" in {


this test has to be split into separate cases or be a for loop not a copy/paste

done, there's a single function for running each check.

…I-1 into mv/datetime-rff

…into mv/datetime-rff

tovbinm · 2018-09-13T20:50:29Z

core/src/test/scala/com/salesforce/op/filters/PreparedFeaturesTest.scala

@@ -157,6 +162,66 @@ class PreparedFeaturesTest extends FlatSpec with TestSparkContext {
    testCorrMatrix(allResponseKeys2, CorrelationType.Spearman, expected)
  }

+  it should "correctly transform date features when time period DayOfMonth is specified" in {


shorten test names to maybe: transform dates with DayOfMonth time period

tovbinm · 2018-09-13T20:50:49Z

core/src/test/scala/com/salesforce/op/filters/PreparedFeaturesTest.scala

+
+  def runDateToUnitCircleTest(
+    period: TimePeriod,
+    expected1: Double,


you can make expected: Double* or expected: Seq[Double]

tovbinm · 2018-09-13T22:07:59Z

core/src/main/scala/com/salesforce/op/stages/impl/feature/DateToUnitCircleTransformer.scala

@@ -100,23 +100,31 @@ class DateToUnitCircleTransformer[T <: Date]
  }
 }

-private[op] object DateToUnitCircle {


why remove private[op]?

Oh woops, I needed to use this with repl at some point

…I-1 into mv/datetime-rff

tovbinm · 2018-09-13T22:28:00Z

core/src/main/scala/com/salesforce/op/filters/PreparedFeatures.scala

      case SomeValue(v: DenseVector) => Map((name, None) -> Right(v.toArray.toSeq))
      case SomeValue(v: SparseVector) => Map((name, None) -> Right(v.indices.map(_.toDouble).toSeq))
      case ft@SomeValue(_) => ft match {
+        case v: Text => Map((name, None) -> Left(v.value.toSeq.flatMap(tokenize)))


thank you! @marcovivero

tovbinm

lgtm!

tovbinm · 2018-09-17T17:19:55Z

I added DO NOT MERGE because we are waiting for @marcovivero to test it.

…s. (#130)

marcovivero added 2 commits September 12, 2018 16:06

Date unit circle binning for RFF

3664d3a

Add to OpWorkflow constructor

f62c38d

marcovivero requested review from tovbinm, leahmcguire and kinfaikan September 13, 2018 00:49

salesforce-cla bot added the cla:missing label Sep 13, 2018

salesforce-cla bot added cla:signed and removed cla:missing labels Sep 13, 2018

tovbinm reviewed Sep 13, 2018

View reviewed changes

marcovivero and others added 6 commits September 13, 2018 11:32

Add to OpWorkflow constructor

c93d083

Merge branch 'mv/datetime-rff' of github.com:marcovivero/TransmogrifA…

6a41399

…I-1 into mv/datetime-rff

Merge branch 'master' of https://github.com/salesforce/TransmogrifAI …

5afa43c

…into mv/datetime-rff

Forgot some docs

37f4c41

Fix tests

84f72ae

Split out time period test cases

9d6ea44

tovbinm reviewed Sep 13, 2018

View reviewed changes

marcovivero and others added 3 commits September 13, 2018 14:22

Add time zone + move out dataframe from function

bc39bfe

mt/Clean up tests

f255f87

Merge branch 'master' into mv/datetime-rff

e1cc959

tovbinm reviewed Sep 13, 2018

View reviewed changes

marcovivero added 3 commits September 13, 2018 15:10

Address last few comments

0ef87da

Merge branch 'mv/datetime-rff' of github.com:marcovivero/TransmogrifA…

3635e24

…I-1 into mv/datetime-rff

Add back private[op]

cb63dd4

tovbinm reviewed Sep 13, 2018

View reviewed changes

tovbinm approved these changes Sep 13, 2018

View reviewed changes

Merge branch 'master' into mv/datetime-rff

724b22e

tovbinm added ready for review needs testing labels Sep 14, 2018

leahmcguire approved these changes Sep 14, 2018

View reviewed changes

tovbinm added the DO NOT MERGE label Sep 14, 2018

kinfaikan approved these changes Sep 17, 2018

View reviewed changes

marcovivero and others added 2 commits September 17, 2018 22:13

Silly bug

9582002

Merge branch 'master' into mv/datetime-rff

2758202

tovbinm removed the DO NOT MERGE label Sep 21, 2018

Merge branch 'master' into mv/datetime-rff

104169e

tovbinm merged commit f9a3718 into salesforce:master Sep 21, 2018

ericwayman pushed a commit that referenced this pull request Feb 8, 2019

Apply DateToUnitCircleTransformer logic in raw feature transformation…

3e84f4d

…s. (#130)

salesforce-cla bot added cla:missing and removed cla:signed labels Aug 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Apply DateToUnitCircleTransformer logic in raw feature transformations. #130

Apply DateToUnitCircleTransformer logic in raw feature transformations. #130

marcovivero commented Sep 13, 2018

salesforce-cla bot commented Sep 13, 2018

codecov bot commented Sep 13, 2018 •

edited

Loading

tovbinm Sep 13, 2018

tovbinm Sep 13, 2018 •

edited

Loading

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018 •

edited

Loading

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

tovbinm Sep 13, 2018 •

edited

Loading

tovbinm Sep 13, 2018

marcovivero Sep 13, 2018

tovbinm Sep 13, 2018

tovbinm left a comment

tovbinm commented Sep 17, 2018

		@@ -30,9 +30,14 @@

		package com.salesforce.op.filters


		import java.time.{Instant, OffsetDateTime, ZoneOffset}

Apply DateToUnitCircleTransformer logic in raw feature transformations. #130

Apply DateToUnitCircleTransformer logic in raw feature transformations. #130

Conversation

marcovivero commented Sep 13, 2018

salesforce-cla bot commented Sep 13, 2018

codecov bot commented Sep 13, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

tovbinm Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcovivero Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tovbinm Sep 13, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tovbinm left a comment

Choose a reason for hiding this comment

tovbinm commented Sep 17, 2018

codecov bot commented Sep 13, 2018 •

edited

Loading

tovbinm Sep 13, 2018 •

edited

Loading

marcovivero Sep 13, 2018 •

edited

Loading

tovbinm Sep 13, 2018 •

edited

Loading