Add remote debug guidance #259
Conversation
| @@ -1,4 +1,5 @@ | |||
| .idea/ | |||
| target/ | |||
| .DS_Store | |||
| .jekyll-cache/ | |||
There was a problem hiding this comment.
It's generated in jekyll 4.0.0.
developer-tools.md
Outdated
| ``` | ||
| sbt > set javaOptions in Test += "-agentlib:jdwp=transport=dt_socket,server=n,suspend=n,address=localhost:5005" | ||
| ``` | ||
| Set breakpoints and run the test, e.g.: |
There was a problem hiding this comment.
Set breakpoints with IntelliJ and run the test with SBT
|
thanks for adding! cc @srowen @dongjoon-hyun as well |
developer-tools.md
Outdated
| @@ -435,6 +435,44 @@ Error:(147, 9) value q is not a member of StringContext | |||
| q""" | |||
| ^ | |||
| ``` | |||
| <h4>Debug Spark with SBT and IntelliJ</h4> | |||
There was a problem hiding this comment.
It seems that the content below is just about Remote debug unit tests on SBT with IntelliJ.
And of course, we can provide an example with some application here.
runMain exampleFoo
Otherwise, the title is too big here.
|
Thank you for pinging me, @cloud-fan . |
developer-tools.md
Outdated
| @@ -435,6 +435,44 @@ Error:(147, 9) value q is not a member of StringContext | |||
| q""" | |||
| ^ | |||
| ``` | |||
| <h4>Debug Spark with SBT and IntelliJ</h4> | |||
| Debug Spark directly inside IntelliJ is somehow difficult. But with remote debug functionality which provided by IntelliJ, | |||
There was a problem hiding this comment.
Let's remove this, Debug Spark directly inside IntelliJ is somehow difficult. because adding a guide is enough.
|
Kindly ping @cloud-fan @dongjoon-hyun @gengliangwang @srowen Hi all, I've updated according to your comments and update the screenshots. PTAL, thanks! |
developer-tools.md
Outdated
| @@ -435,6 +435,46 @@ Error:(147, 9) value q is not a member of StringContext | |||
| q""" | |||
| ^ | |||
| ``` | |||
| <h4>Remote Debug Unit Test</h4> | |||
There was a problem hiding this comment.
I think we should make it more general: "Debug Spark Remotely".
Running SBT unit test is just one way to trigger the IntelliJ debugging, users can go other ways like running pyspark tests, as long as they set the java options correctly.
There was a problem hiding this comment.
+1 for @cloud-fan 's point.
developer-tools.md
Outdated
| @@ -435,6 +435,46 @@ Error:(147, 9) value q is not a member of StringContext | |||
| q""" | |||
| ^ | |||
| ``` | |||
| <h4>Remote Debug Unit Test</h4> | |||
| This part will show you how to debug unit test in SBT with the help of remote debug functionality which provided by IntelliJ. | |||
There was a problem hiding this comment.
This part will show you how to debug Spark remotely with IntelliJ.
developer-tools.md
Outdated
|
|
||
| <img src="/images/intellij_start_remote_debug.png" style="width: 75%; max-width: 660px;" /> | ||
|
|
||
| <h5>Debug Test in SBT</h5> |
There was a problem hiding this comment.
Trigger the remote debugging
developer-tools.md
Outdated
| <img src="/images/intellij_remote_debug_configuration.png" style="width: 75%; max-width: 660px;" /> | ||
|
|
||
| Normally, the default values should be good enough to use. Make sure that you choose <b>Listen to remote JVM</b> | ||
| as <i>Debugger mode</i> and select the right <i>Command line arguments for remote JVM</i> as per your JDK version. |
There was a problem hiding this comment.
select the right JDK version to generate proper <i>Command line arguments for remote JVM</i>
| <img src="/images/intellij_start_remote_debug.png" style="width: 75%; max-width: 660px;" /> | ||
|
|
||
| <h5>Debug Test in SBT</h5> | ||
|
|
There was a problem hiding this comment.
In general, there are 2 steps:
1. Set JVM options using the <i>Command line arguments for remote JVM</i> generated in the last step.
2. Start the Spark execution (SBT test, pyspark test, spark-shell, etc.)
The following is an example of how to trigger the remote debugging using SBT unit tests.
There was a problem hiding this comment.
Maybe, we could move this overall instruction to under the first level tile Debug Spark Remotely?
developer-tools.md
Outdated
| ``` | ||
| sbt > project core | ||
| ``` | ||
| Set javaOptions by copy pasting the following: |
There was a problem hiding this comment.
copy pasting the <i>Command line arguments for remote JVM</i>
developer-tools.md
Outdated
| debug in IntelliJ as usual. | ||
|
|
||
| To exit remote debug mode (so that you don't have to keep starting the remote debugger), | ||
| type "session clear" in SBT console - while you're in a project. |
|
thanks, merged! |
|
thanks all!! |
Debug in IntelliJ directly is somehow difficult. For example, it will easily fail on compile if user don't generate source files from
SqlBase.g4manually, which can be confused for beginning user. So, this PR suggests to use remote debug for better experience.Note I added the guidance after the IDE Setup > IntelliJ section: