Skip to content

Commit

Permalink
Refactor core functionality from withSite into pure testable function
Browse files Browse the repository at this point in the history
  • Loading branch information
simonjscott committed Jan 15, 2019
1 parent 75ade2b commit 1ba7fdf
Show file tree
Hide file tree
Showing 2 changed files with 14 additions and 3 deletions.
4 changes: 4 additions & 0 deletions CONTRIBUTING.md
Expand Up @@ -23,3 +23,7 @@ To improve the documentation, please create a [pull request in the kontextfrei r
## Code contributions

If you want to make a code contribution, it would be great to make sure that a GitHub issue for it exists beforehand. Like contributions to documentation, code can be contributed by creating a [pull request in the kontextfrei repository](https://github.com/dwestheide/kontextfrei/pulls). Please use the description of the pull request to explain the what, why, and if necessary, how of your change. Also, please rebase your pull request against the master branch and squash the changes in your pull request into a single commit.

### Developer Notes
* When adding support for new RDD methods, observe the use of withSite loan pattern in the RDDBaseFunction methods to
ensure correct reporting of the call site in the Spark UI.
Expand Up @@ -8,8 +8,16 @@ private[kontextfrei] trait RDDBase {

def withSite[A, R](as: RDD[A])(body: RDD[A] => R): R = {

val (method, site) = callSiteInfo("com.danielwestheide.kontextfrei")

as.sparkContext.setCallSite(s"$method at $site")
body(as)
}

def callSiteInfo(prefix: String): (String, String) = {

def isInKf(ste: StackTraceElement): Boolean = {
Option(ste.getClassName()).exists(_.startsWith("com.danielwestheide.kontextfrei"))
Option(ste.getClassName()).exists(_.startsWith(prefix))
}

val (kf, user) = Thread.currentThread().getStackTrace()
Expand All @@ -19,7 +27,6 @@ private[kontextfrei] trait RDDBase {
val method = kf.lastOption.map(_.getMethodName()).getOrElse("unknown")
val site = user.headOption.map(s => s"${s.getFileName()}:${s.getLineNumber()}").getOrElse("unknown")

as.sparkContext.setCallSite(s"$method at $site")
body(as)
(method, site)
}
}

0 comments on commit 1ba7fdf

Please sign in to comment.