util: Open-source IntelliJ async stack trace capture points

Summary: Problem / Solution Open-source IntelliJ Twitter Futures capture points config. Thanks to our Summer 2017 intern, Haggai Kaunda, for taking on this project! JIRA Issues: CSL-5183 TBR=true Differential Revision: https://phabricator.twitter.biz/D96782
twitter · Nov 1, 2017 · 48fead2 · 48fead2
1 parent e1ee37b
commit 48fead2
Show file tree

Hide file tree

Showing 12 changed files with 523 additions and 19 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -37,6 +37,7 @@ before_script:
 
 script:
     - travis_retry ./sbt ++$TRAVIS_SCALA_VERSION clean coverage test coverageReport
+    - travis_retry ./sbt ++2.11.11 "project util-intellij" coverage test coverageReport
     - ./sbt ++$TRAVIS_SCALA_VERSION coverageAggregate
 
 after_success:

diff --git a/CHANGES b/CHANGES
@@ -17,6 +17,12 @@ Release Version Changes:
   * From now on, release versions will be based on release date in the format of
     YY.MM.x where x is a patch number. ``PHAB_ID=D101244``
 
+New Features:
+
+  * util-intellij: Create util-intellij project and publish IntelliJ capture
+    points plugin for debugging asynchronous stack traces of code using Twitter
+    Futures in Scala 2.11.11. ``PHAB_ID=D96782``
+
 API Changes:
 
   * util-app: c.t.app.Flag.let and letClear are now generic in their return type.

diff --git a/build.sbt b/build.sbt
@@ -22,11 +22,14 @@ val jsr305Lib = "com.google.code.findbugs" % "jsr305" % "2.0.1"
 val scalacheckLib = "org.scalacheck" %% "scalacheck" % "1.13.4" % "test"
 val slf4jLib = "org.slf4j" % "slf4j-api" % slf4jVersion
 
-val sharedSettings = Seq(
+val defaultProjectSettings = Seq(
+  scalaVersion := "2.12.1",
+  crossScalaVersions := Seq("2.11.11", "2.12.1")
+)
+
+val baseSettings = Seq(
   version := releaseVersion,
   organization := "com.twitter",
-  scalaVersion := "2.12.1",
-  crossScalaVersions := Seq("2.11.11", "2.12.1"),
   // Workaround for a scaladoc bug which causes it to choke on empty classpaths.
   unmanagedClasspath in Compile += Attributed.blank(new java.io.File("doesnotexist")),
   libraryDependencies ++= Seq(
@@ -68,22 +71,22 @@ val sharedSettings = Seq(
   pomExtra :=
     <url>https://github.com/twitter/util</url>
       <licenses>
-    <license>
-    <name>Apache License, Version 2.0</name>
-    <url>http://www.apache.org/licenses/LICENSE-2.0</url>
-      </license>
-    </licenses>
-    <scm>
-    <url>git@github.com:twitter/util.git</url>
-    <connection>scm:git:git@github.com:twitter/util.git</connection>
-    </scm>
-    <developers>
-    <developer>
-    <id>twitter</id>
-    <name>Twitter Inc.</name>
-    <url>https://www.twitter.com/</url>
-      </developer>
-    </developers>,
+        <license>
+          <name>Apache License, Version 2.0</name>
+          <url>http://www.apache.org/licenses/LICENSE-2.0</url>
+        </license>
+      </licenses>
+      <scm>
+        <url>git@github.com:twitter/util.git</url>
+        <connection>scm:git:git@github.com:twitter/util.git</connection>
+      </scm>
+      <developers>
+        <developer>
+          <id>twitter</id>
+          <name>Twitter Inc.</name>
+          <url>https://www.twitter.com/</url>
+        </developer>
+      </developers>,
   publishTo := {
     val nexus = "https://oss.sonatype.org/"
     if (version.value.trim.endsWith("SNAPSHOT"))
@@ -93,6 +96,8 @@ val sharedSettings = Seq(
   }
 )
 
+val sharedSettings = defaultProjectSettings ++ baseSettings
+
 lazy val noPublishSettings = Seq(
   publish := {},
   publishLocal := {},
@@ -256,6 +261,16 @@ lazy val utilHashing = Project(
   libraryDependencies += scalacheckLib
 ).dependsOn(utilCore % "test")
 
+lazy val utilIntellij = Project(
+  id = "util-intellij",
+  base = file("util-intellij")
+).settings(
+  baseSettings
+).settings(
+  name := "util-intellij",
+  scalaVersion := "2.11.11"
+).dependsOn(utilCore % "test")
+
 lazy val utilJvm = Project(
   id = "util-jvm",
   base = file("util-jvm")

diff --git a/doc/src/sphinx/index.rst b/doc/src/sphinx/index.rst
@@ -13,6 +13,7 @@ Guides
 
    util-stats/index
    util-cookbook/index
+   util-capturepoints/index
 
 
 Documentation

diff --git a/doc/src/sphinx/util-capturepoints/index.rst b/doc/src/sphinx/util-capturepoints/index.rst
@@ -0,0 +1,107 @@
+Twitter Futures Capture Points
+==============================
+
+Stack traces produced from asynchronous execution have undesirable
+properties, e.g., they are disorganized and often contain frames from
+internal libraries that programmers need not know about. This result is
+inevitable since not only do different parts of the same program get
+executed on different threads, and possibly different cores, but
+execution also jumps between different frames. This means that each
+thread used to execute a program may produce a unique stack trace. Thus,
+an Asynchronous Stack Trace (AST) gives a fragmented window view into
+the execution of a program and thus makes it difficult to understand the
+flow of code because the causality between frames is lost.
+
+In response to this phenomenon, IntelliJ 2017.1 introduced a feature
+called Capture Points which is built on top of the IntelliJ Debugger. A
+capture point is a place in a computer program where the debugger
+collects and saves stack frames to be used later when we reach a
+specific point in the code and we want to see how we got there. IntelliJ
+IDEA does this by substituting part of the call stack with the captured
+frame.
+
+We have written capture points for Twitter Futures in an XML file. Users can
+debug their asynchronous code more easily with these capture points. Note that
+as of October 2017, the capture points work with only Scala 2.11.11.
+
+Setup
+^^^^^
+
+The capture points are defined in
+``util/util-intellij/src/main/resources/TwitterFuturesCapturePoints.xml``.
+To import them into IntelliJ,
+
+1. Open IntelliJ. In the menu bar, click IntelliJ IDEA > Preferences.
+2. Navigate to Build, Execution, Deployment > Debugger > Async
+   Stacktraces.
+3. Click the Import icon on the bottom bar. Find the XML file and click
+   OK.
+
+Use
+^^^
+
+TL;DR: set a breakpoint where you wish to see the stack trace, debug
+your code, and look at the "Frames" tab in the Debugger. Any
+asynchronous calls in the stack trace will appear in logical order. If
+you wish to clean up the stack trace, click the funnel icon in the top
+right, "Hide Frames from Libraries".
+
+We will illustrate how to use the capture points to assist with
+debugging with a small example. The example is located in
+``util/util-intellij/src/test/scala/com/twitter/util/capturepoints/Demo.scala``.
+
+A brief explanation of this test: the test passes a ``Promise[Int]``
+through three methods and then sets the ``Promise``\ ’s value in a
+``futurePool``. The calls are asynchronous, but the logical flow of the
+test is as follows:
+
+1. ``test`` block calls ``someBusinessLogic``
+2. ``someBusinessLogic`` calls ``moreBusinessLogic``
+3. ``moreBusinessLogic`` calls ``lordBusinessLogic``
+4. ``lordBusinessLogic`` waits for the ``Promise``\ ’s value to be set
+5. The ``Promise``\ ’s value is set in the test block (this could happen
+   at any time; it is not necessarily step number 5)
+6. ``lordBusinessLogic`` returns
+7. ``test``\ ’s ``result`` variable is ``4``, and the test passes
+
+Suppose we wish to inspect the stack trace from inside
+``lordBusinessLogic``. If we set a breakpoint at line 47 and then debug
+the test with pants in IntelliJ with the capture points disabled, we see
+the following stack trace:
+
+::
+
+    apply$mcVI$sp:47, Demo$$anonfun$lordBusinessLogic$1
+    apply:1820, Future$$anonfun$onSuccess$1
+    apply:205, Promise$Monitored
+    run:532, Promise$$anon$7
+    run:198, LocalScheduler$Activation
+    submit:157, LocalScheduler$Activation
+    submit:274, LocalScheduler
+    submit:109, Scheduler$
+    runq:522, Promise
+    updatelfEmpty:880, Promise
+    update:859, Promise
+    setValue:835, Promise
+    apply$mcV$sp:20, Demo$$anonfun$3$$anonfun$apply$1
+    apply:15, Try$
+    run:140, ExecutorServiceFuturePool$$anon$4
+
+The library calls in the stack trace do not help us understand our code.
+If we enable the capture points and again debug the test, we see the
+following stack trace:
+
+::
+
+    apply$mcVI$sp:47, Demo$$anonfun$lordBusinessLogic$1
+    apply:1820, Future$$anonfun$onSuccess$1
+    onSuccess:1819, Future
+    lordBusinessLogic:42, Demo
+    moreBusinessLogic:38, Demo
+    someBusinessLogic:31, Demo
+    apply:17, Demo$$anonfun$3
+
+This is much cleaner and it includes only calls to functions we wrote.
+It lets us clearly see that the test block called ``someBusinessLogic``,
+that ``someBusinessLogic`` called ``moreBusinessLogic``, and that
+``moreBusinessLogic`` called ``lordBusinessLogic``.
diff --git a/util-intellij/OWNERS b/util-intellij/OWNERS
@@ -0,0 +1,6 @@
+banderson
+jillianc
+koliver
+mnakamura
+roanta
+slance
diff --git a/util-intellij/README.md b/util-intellij/README.md
@@ -0,0 +1,103 @@
+# util-intellij
+
+`util-intellij` is a project containing IntelliJ plugins and other utilities.
+
+## Contents
+
+### Twitter Futures Capture Points
+
+Stack traces produced from asynchronous execution have undesirable properties,
+e.g., they are disorganized and often contain frames from internal libraries
+that programmers need not know about. This result is inevitable since not only
+do different parts of the same program get executed on different threads, and
+possibly different cores, but execution also jumps between different frames.
+This means that each thread used to execute a program may produce a unique stack
+trace. Thus, an Asynchronous Stack Trace (AST) gives a fragmented window view
+into the execution of a program and thus makes it difficult to understand the
+flow of code because the causality between frames is lost.
+
+In response to this phenomenon, IntelliJ 2017.1 introduced a feature called
+Capture Points which is built on top of the IntelliJ Debugger. A capture point
+is a place in a computer program where the debugger collects and saves stack
+frames to be used later when we reach a specific point in the code and we want
+to see how we got there. IntelliJ IDEA does this by substituting part of the
+call stack with the captured frame.
+
+We have written capture points for Twitter Futures in an XML file. Users can
+debug their asynchronous code more easily with these capture points. Note that
+as of October 2017, the capture points work with only Scala 2.11.11.
+
+#### Setup
+
+The capture points are defined in
+`util/util-intellij/src/main/resources/TwitterFuturesCapturePoints.xml`.
+To import them into IntelliJ,
+
+1. Open IntelliJ. In the menu bar, click IntelliJ IDEA > Preferences.
+2. Navigate to Build, Execution, Deployment > Debugger > Async Stacktraces.
+3. Click the Import icon on the bottom bar. Find the XML file and click OK.
+
+#### Use
+
+TL;DR: set a breakpoint where you wish to see the stack trace, debug your code,
+and look at the "Frames" tab in the Debugger. Any asynchronous calls in the 
+stack trace will appear in logical order. If you wish to clean up the stack
+trace, click the funnel icon in the top right, "Hide Frames from Libraries".
+
+We will illustrate how to use the capture points to assist with debugging with a
+small example. The example is located in
+`util/util-intellij/src/test/scala/com/twitter/util/capturepoints/Demo.scala`.
+
+A brief explanation of this test: the test passes a `Promise[Int]` through three
+methods and then sets the `Promise`’s value in a `futurePool`. The calls are
+asynchronous, but the logical flow of the test is as follows:
+
+1. `test` block calls `someBusinessLogic`
+2. `someBusinessLogic` calls `moreBusinessLogic`
+3. `moreBusinessLogic` calls `lordBusinessLogic`
+4. `lordBusinessLogic` waits for the `Promise`’s value to be set
+5. The `Promise`’s value is set in the test block (this could happen at any
+   time; it is not necessarily step number 5)
+6. `lordBusinessLogic` returns
+7. `test`’s `result` variable is `4`, and the test passes
+
+Suppose we wish to inspect the stack trace from inside `lordBusinessLogic`. If 
+we set a breakpoint at line 47 and then debug the test with pants in IntelliJ 
+with the capture points disabled, we see the following stack trace:
+
+```
+apply$mcVI$sp:47, Demo$$anonfun$lordBusinessLogic$1
+apply:1820, Future$$anonfun$onSuccess$1
+apply:205, Promise$Monitored
+run:532, Promise$$anon$7
+run:198, LocalScheduler$Activation
+submit:157, LocalScheduler$Activation
+submit:274, LocalScheduler
+submit:109, Scheduler$
+runq:522, Promise
+updatelfEmpty:880, Promise
+update:859, Promise
+setValue:835, Promise
+apply$mcV$sp:20, Demo$$anonfun$3$$anonfun$apply$1
+apply:15, Try$
+run:140, ExecutorServiceFuturePool$$anon$4
+```
+
+The library calls in the stack trace do not help us understand our code. If we
+enable the capture points and again debug the test, we see the following stack
+trace:
+
+```
+apply$mcVI$sp:47, Demo$$anonfun$lordBusinessLogic$1
+apply:1820, Future$$anonfun$onSuccess$1
+onSuccess:1819, Future
+lordBusinessLogic:42, Demo
+moreBusinessLogic:38, Demo
+someBusinessLogic:31, Demo
+apply:17, Demo$$anonfun$3
+```
+
+This is much cleaner and it includes only calls to functions we wrote. It lets
+us clearly see that the test block called `someBusinessLogic`, that
+`someBusinessLogic` called `moreBusinessLogic`, and that `moreBusinessLogic`
+called `lordBusinessLogic`.
diff --git a/util-intellij/src/main/resources/BUILD b/util-intellij/src/main/resources/BUILD
@@ -0,0 +1,3 @@
+resources(
+  sources=rglobs('*')
+)