This repository is created to reproduce the issue described here.
- Download Spark:
https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz - Download Hadoop compiled for Windows (
winutils.exe
):
https://github.com/cdarlint/winutils/raw/master/hadoop-3.3.5/bin/winutils.exe - Navigate to the folder where Spark and Hadoop will be placed,
for example:C:\Users\progxaker\Downloads
. - Create a new folder named Spark
(e.g.,C:\Users\progxaker\Downloads\Spark
). - Unpack the Spark archive to the created folder
(e.g.,C:\Users\progxaker\Downloads\Spark\spark-3.5.0-bin-hadoop3
). - Create a new folder named Hadoop
(e.g.,C:\Users\progxaker\Downloads\Hadoop
). - Create a new folder named bin in the created folder
(e.g.,C:\Users\progxaker\Downloads\Hadoop\bin
). - Move the Hadoop binary to the created folder
(e.g.,C:\Users\progxaker\Downloads\Hadoop\bin\winutils.exe
).
- Open "Command Prompt".
- Navigate to the folder with the cloned repository.
- Build the application:
mvn package
.
- Go to the folder with the Application Insights JVM agent.
- Open "Command Prompt".
- Run the following:
Please note that no quotes are required.
SET SPARK_HOME=<the-folder-from-step-5> SET HADOOP_HOME=<the-folder-from-step-6> %SPARK_HOME%\bin\spark-submit --class SimpleApp --master local[4] <absolute-path-to-the-clonned-repository>\target\simple-project-1.0.jar
- Run with the Application Insights JVM agent:
SET SPARK_HOME=<folder-from-step-5> SET HADOOP_HOME=<folder-from-step-6> SET APPLICATIONINSIGHTS_CONNECTION_STRING=<connection-string> SET APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL=TRACE %SPARK_HOME%\bin\spark-submit --conf "spark.driver.extraJavaOptions='-javaagent:applicationinsights-agent-3.5.1.jar'" --class SimpleApp --master local[4] <absolute-path-to-the-clonned-repository>\target\simple-project-1.0.jar
- Go to the folder with the Application Insights JVM agent.
- Open "Command Prompt".
- Run the following:
Please note that no quotes are required.
SET SPARK_HOME=<the-folder-from-step-5> SET HADOOP_HOME=<the-folder-from-step-6> SET JAVA_TOOL_OPTIONS=-javaagent:application-insights-3.5.1.jar SET APPLICATIONINSIGHTS_CONNECTION_STRING=<connection-string> SET APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL=TRACE %SPARK_HOME%\bin\spark-submit --class SimpleApp --master local[4] <absolute-path-to-the-clonned-repository>\target\simple-project-1.0.jar
docker build -t reproduce-trace-issues:v1 .
docker run --rm -ti --env APPLICATIONINSIGHTS_CONNECTION_STRING="<connection-string>" \
--env APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL="TRACE" \
reproduce-trace-issues:v1 \
/tmp/spark/bin/spark-submit --conf "spark.driver.extraJavaOptions='-javaagent:/tmp/applicationinsights-agent.jar'" \
--class SimpleApp --master local[4] /tmp/simple-project-1.0.jar
docker run --rm -ti --env APPLICATIONINSIGHTS_CONNECTION_STRING="<connection-string>" \
--env APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL="TRACE" \
--env JAVA_TOOL_OPTIONS="-javaagent:/tmp/applicationinsights-agent.jar" \
reproduce-trace-issues:v1