Documentation issue: spark, heuristics and hdfs readers

[kockockockoc]

hi!

Could you give some tips/hints/examples how install/config/use:

• garmadon-frameworks-spark
• garmadon-readers-hdfs
• garmadon-readers-heuristics

And how to use (what is it for?):

• garmadon-schema.jar
• garmadon-jvm-statistics.jar (garmadon-jvm-statistics-*)

thanks in advance for your help

[ashangit]

Hi @kockockockoc ,

Only garmadon-agent, garmadon-forwarder and garmadon-readers-* are usefull to deploy garmadon
Other jars are dependencies to agent/forwarder/readers jars to share protobuf schema, some mechanism to trace metrics...

garmadon-frameworks-spark is loaded by the garmadon-agent to retreive metrics from spark application using Spark Listener mechanism.
The listener is automatically attached to spark driver if you use StandaloneModule (-javaagent:path-to-garmadon-agent-jar=com.criteo.hadoop.garmadon.agent.modules.StandaloneModule) or ContainerModule (-javaagent:path-to-garmadon-agent-jar=com.criteo.hadoop.garmadon.agent.modules.ContainerModule)

executorService.submit(() -> {
    SparkListenerTracer.setup(ContainerHeader.getInstance().getHeader(),
            (timestamp, header, event) -> eventProcessor.offer(timestamp, header, event));
});

Properties props = System.getProperties();
        props.setProperty("spark.extraListeners", "com.criteo.hadoop.garmadon.spark.listener.GarmadonSparkListener");

So you don't have any specific things to do, only need to bind the agent to all spark JVMs (client and containers).
Will see to improve the documentation around garmadon and spark

For readers:

• garmadon-readers-hdfs, will need to add some documentation about it, @gray-matter ?
• garmadon-readers-heuristics, it is still bind to some specificities from criteo so not really available for now.

[kockockockoc]

Thank you for explaining. I was a bit confused by the fact that there were separate jars after "mvn clean package".