SPARK-2127: Use application specific folders to dump metrics via CsvSink #1067
Conversation
|
Can one of the admins verify this patch? |
There was a problem hiding this comment.
Hi @rahulsinghaliitd , do we really need to use SparkConf to get this unique app name? I think metrics system is driven by its own configuration system, here you involved SparkConf as another configuration system just only for CsvSink, this will give people the chance to bypass the original way and lose its controllability. I think it would be nice to modify the way to get appUniqueName (eg. get it from SparkEnv).
There was a problem hiding this comment.
@jerryshao Thanks for the feedback! I was not aware of the fact the SparkEnv provides access to SparkConf. Will follow up with the suggested modification.
There was a problem hiding this comment.
Forgot to mention that several sinks like CsvSink will be started in master and worker for standalone mode if configured, in this situation you may not get unique app name or SparkEnv (something created only when app started). So maybe you should consider this situation when there's no app name available.
There was a problem hiding this comment.
Hi @jerryshao ,
- At the moment the Sinks are being created, SparkEnv has not been created. I may be able to modify the Properties being passed to this Sink or even get the SparkConf from SecurityManager. But neither of those approaches seems generic to me. For e.g. we will need hadoopConf if we wanted the csv directory to be on HDFS.
- Thanks for pointing out the problem with Master and Worker. I have for now added app names to these classes. Please let me know if you think adding null checks in CsvSink would also be useful.
1. Generate an unique app name which is used to create events and metric folders. 2. Add app name & unique app name to Standalone mode's Master and Worker. 3. Log the unique app directory that will be used by CsvSink.
|
Can one of the admins verify this patch? |
…n for Hadoop platform (#1067) * [CARMEL-6200] Add unified command to extract table lineage information for Hadoop platform * Add tests
Generate an unique app name which is used to create events and metric folders.