[SPARK-8518] [ML] Log-linear models for survival analysis #8611

yanboliang · 2015-09-05T07:47:03Z

Accelerated Failure Time (AFT) model is the most commonly used and easy to parallel method of survival analysis for censored survival data. It is the log-linear model based on the Weibull distribution of the survival time.
Users can refer to the R function survreg to compare the model and predict to compare the prediction. There are different kinds of model prediction, I have just select the type response which is default used for R.

SparkQA · 2015-09-05T08:34:44Z

Test build #42039 has finished for PR 8611 at commit 7f2a31d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTRegression(override val uid: String)

SparkQA · 2015-09-06T19:15:57Z

Test build #42073 has finished for PR 8611 at commit d0a29bb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTRegression(override val uid: String)

SparkQA · 2015-09-07T11:44:10Z

Test build #42091 has finished for PR 8611 at commit c30a1c2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTRegression(override val uid: String)

SparkQA · 2015-09-07T11:51:24Z

Test build #42092 has finished for PR 8611 at commit 4223793.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTRegression(override val uid: String)

rotationsymmetry · 2015-09-07T16:32:54Z

@yanboliang

As sigma is restricted to be positive, in the L-BLGS algorithm, shall we optimize over log(sigma)? This way, there is no more restriction on the possible value of sigma and usual help with convergence.

yanboliang · 2015-09-08T10:32:40Z

@rotationsymmetry I agree with you, updated PR. Thanks for your comments.

SparkQA · 2015-09-08T11:17:32Z

Test build #42128 has finished for PR 8611 at commit 28ca4c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTRegression(override val uid: String)

Add WeibullGenerator for RandomDataGenerator. #8611 need use WeibullGenerator to generate random data based on Weibull distribution. Author: Yanbo Liang <ybliang8@gmail.com> Closes #8622 from yanboliang/spark-10464.

SparkQA · 2015-09-09T10:49:13Z

Test build #42200 has finished for PR 8611 at commit b7dd012.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2015-09-09T15:09:36Z

@rotationsymmetry Do you have time to make a pass?

rotationsymmetry · 2015-09-09T16:12:37Z

@mengxr Yes, I have done a quick run and added some comments the past weekend. I will do a more thorough pass today/tomorrow.

rotationsymmetry · 2015-09-09T16:32:21Z

mllib/src/main/scala/org/apache/spark/ml/regression/AFTRegression.scala

+
+  def this() = this(Identifiable.randomUID("aftReg"))
+
+  /** @group setParam */


Please provide documentation for public API. Similar for other getters and setters if not inherited.

mengxr · 2015-09-15T17:24:16Z

I made a brief scan over the code. @rotationsymmetry Could you make another pass?

rotationsymmetry · 2015-09-16T17:51:19Z

mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala

+ *   L(\beta,\sigma)=\prod_{i=1}^n[\frac{1}{\sigma}f_{0}
+ *   (\frac{\log{t_{i}}-x^{'}\beta}{\sigma})]^{\delta_{i}}S_{0}
+ *   (\frac{\log{t_{i}}-x^{'}\beta}{\sigma})^{1-\delta_{i}}
+ * }}}


delta in the formula is the event indicator. Since we decide to use the censored indicator, I recommend we clarify this in the formula.

rotationsymmetry · 2015-09-16T21:43:46Z

@yanboliang @mengxr I just made another round. Thanks!

…oint

… constructor.

yanboliang · 2015-09-17T14:23:53Z

@mengxr @rotationsymmetry Thanks for your comments. I have updated the PR, Would you mind to make another pass?

SparkQA · 2015-09-17T14:46:41Z

Test build #42603 has finished for PR 8611 at commit dc1d59f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTSurvivalRegression @Since("1.6.0") (@Since("1.6.0") override val uid: String)

mengxr · 2015-09-17T23:28:43Z

mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala

+  /**
+   * Param for censored column name.
+   * The value of this column could be 0 or 1.
+   * If the value is 1, it means the event has occurred i.e. uncensored; otherwise it censored.


This is confusing. The column name is called censored but value 1 means uncensored. I searched for some survival analysis datasets, where "censor" is commonly used as the column name. See https://www.umass.edu/statdata/statdata/stat-survival.html. However, in those datasets, 1 actually means occurred, i.e., uncensored. So my recommendation would be renaming the param name back to censorCol and keep the current doc. That is 1 for uncensored and 0 for censored. And note that this is to be consistent with existing datasets.

it censored -> censored

mengxr · 2015-09-17T23:39:05Z

@yanboliang Let's rename censoredCol back to cencorCol and fix small issues I mentioned inline. Then this PR should be good to go. We can make another PR to add quantileCol.

mengxr · 2015-09-17T23:42:06Z

mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala

+ * @param features List of features for this data point.
+ * @param label Label for this data point.
+ * @param censored Indicator of the event has occurred or not. If the value is 1, it means
+ *                 the event has occurred i.e. uncensored; otherwise it censored.


remove it

SparkQA · 2015-09-18T04:07:49Z

Test build #42637 has finished for PR 8611 at commit aa37878.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class AFTSurvivalRegression @Since("1.6.0") (@Since("1.6.0") override val uid: String)
- require(censor == 1.0 || censor == 0.0, "censor of class AFTPoint must be 1.0 or 0.0")

mengxr · 2015-09-18T04:38:02Z

LGTM. Merged into master. Thanks! I created https://issues.apache.org/jira/browse/SPARK-10686, SPARK-10687, SPARK-10688, SPARK-10689 for follow-up work.

yanboliang added 2 commits September 5, 2015 15:33

initial implementation of log-linear survival analysis

0edabbb

fix test case issue

7f2a31d

yanboliang changed the title ~~[SPARK-8518] [ML] Log-linear models for survival analysis~~ [WIP] [SPARK-8518] [ML] Log-linear models for survival analysis Sep 5, 2015

add test cases based on generated dataset

d0a29bb

yanboliang added 2 commits September 7, 2015 19:00

fix prediction type, add more test cases

c30a1c2

fix validateAndTransformSchema

4223793

yanboliang mentioned this pull request Sep 7, 2015

[SPARK-10464] [MLlib] Add WeibullGenerator for RandomDataGenerator #8622

Closed

optimize loss function over log(sigma)

28ca4c0

yanboliang added 2 commits September 9, 2015 17:35

merge to master

892ae6b

enable test cases

b7dd012

yanboliang changed the title ~~[WIP] [SPARK-8518] [ML] Log-linear models for survival analysis~~ [SPARK-8518] [ML] Log-linear models for survival analysis Sep 9, 2015

rotationsymmetry reviewed Sep 9, 2015
View reviewed changes

rotationsymmetry reviewed Sep 16, 2015
View reviewed changes

yanboliang added 5 commits September 17, 2015 15:04

rename censoredCol, quantile change to Array[Double], case class AFTP…

5c8d35f

…oint

rename weights to coefficients

f7a6c79

add more R output for unit test

8991563

fix small issues and typos

4da4d5e

add @SInCE version to class, constructor, and public variables in the…

dc1d59f

… constructor.

mengxr reviewed Sep 17, 2015
View reviewed changes

yanboliang added 2 commits September 18, 2015 10:22

rename censoredCol to censorCol, censored to censor

e958b51

address comments

aa37878

asfgit closed this in 98f1ea6 Sep 18, 2015

yanboliang deleted the spark-8518 branch September 18, 2015 07:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8518] [ML] Log-linear models for survival analysis #8611

[SPARK-8518] [ML] Log-linear models for survival analysis #8611

yanboliang commented Sep 5, 2015

SparkQA commented Sep 5, 2015

SparkQA commented Sep 6, 2015

SparkQA commented Sep 7, 2015

SparkQA commented Sep 7, 2015

rotationsymmetry commented Sep 7, 2015

yanboliang commented Sep 8, 2015

SparkQA commented Sep 8, 2015

SparkQA commented Sep 9, 2015

mengxr commented Sep 9, 2015

rotationsymmetry commented Sep 9, 2015

rotationsymmetry Sep 9, 2015

mengxr commented Sep 15, 2015

rotationsymmetry Sep 16, 2015

rotationsymmetry commented Sep 16, 2015

yanboliang commented Sep 17, 2015

SparkQA commented Sep 17, 2015

mengxr Sep 17, 2015

mengxr Sep 17, 2015

mengxr commented Sep 17, 2015

mengxr Sep 17, 2015

SparkQA commented Sep 18, 2015

mengxr commented Sep 18, 2015


		def this() = this(Identifiable.randomUID("aftReg"))

		/** @group setParam */

[SPARK-8518] [ML] Log-linear models for survival analysis #8611

[SPARK-8518] [ML] Log-linear models for survival analysis #8611

Conversation

yanboliang commented Sep 5, 2015

SparkQA commented Sep 5, 2015

SparkQA commented Sep 6, 2015

SparkQA commented Sep 7, 2015

SparkQA commented Sep 7, 2015

rotationsymmetry commented Sep 7, 2015

yanboliang commented Sep 8, 2015

SparkQA commented Sep 8, 2015

SparkQA commented Sep 9, 2015

mengxr commented Sep 9, 2015

rotationsymmetry commented Sep 9, 2015

rotationsymmetry Sep 9, 2015

Choose a reason for hiding this comment

mengxr commented Sep 15, 2015

rotationsymmetry Sep 16, 2015

Choose a reason for hiding this comment

rotationsymmetry commented Sep 16, 2015

yanboliang commented Sep 17, 2015

SparkQA commented Sep 17, 2015

mengxr Sep 17, 2015

Choose a reason for hiding this comment

mengxr Sep 17, 2015

Choose a reason for hiding this comment

mengxr commented Sep 17, 2015

mengxr Sep 17, 2015

Choose a reason for hiding this comment

SparkQA commented Sep 18, 2015

mengxr commented Sep 18, 2015