[SPARK-29032][CORE] Simplify Prometheus support by adding PrometheusServlet/Resource

[dongjoon-hyun]

What changes were proposed in this pull request?

Prometheus.io is a CNCF project used widely with K8s.

• https://github.com/prometheus/prometheus

This PR aims to simplify Prometheus support by adding PrometheusServlet and PrometheusResource. The main use cases are K8s and Spark Standalone cluster environments.

Note that this PR focus Spark-generated metrics. Also, we need to update the document later in a separate PR.

Why are the changes needed?

There exists a few ways to support Prometheus. However, they requires extra configurations and new resources (port numbers to pull or gateways to push). And, the endpoints are widely spreader over Master/Slave/Driver/CoarseGrainedExecutorBackend. This PR aims to export natively Spark's metrics which starts with metrics_ prefix from the existing (1) method at Master/Worker/Driver. For task metrics, instead of using CoarseGrainedExecutorBackend , we will reuse the collected information used for the Apache Spark REST API, api/v1.

1. Spark JMX Sink + Prometheus JMX Converter
2. Custom Sink + Prometheus Pushgateway

Does this PR introduce any user-facing change?

Yes. New web interfaces are added along with the existing JSON API.

	JSON End Point	Prometheus End Point
Master	/metrics/master/json/	/metrics/master/prometheus/
Master	/metrics/applications/json/	/metrics/applications/prometheus/
Worker	/metrics/json/	/metrics/prometheus/
Driver	/metrics/json/	/metrics/prometheus/
Driver	/api/v1/applications/{id}/(executors|stages)	/metrics/detail/prometheus/

$ bin/spark-shell
...
Spark context Web UI available at http://localhost:4040
...

$ curl --silent http://localhost:4040/metrics/prometheus/ | head -n5
metrics_local_1568101220707_driver_BlockManager_disk_diskSpaceUsed_MB_Value 0
metrics_local_1568101220707_driver_BlockManager_memory_maxMem_MB_Value 366
metrics_local_1568101220707_driver_BlockManager_memory_maxOffHeapMem_MB_Value 0
metrics_local_1568101220707_driver_BlockManager_memory_maxOnHeapMem_MB_Value 366
metrics_local_1568101220707_driver_BlockManager_memory_memUsed_MB_Value 0

How was this patch tested?

Manually connect the new end-points with curl.

Or, run prometheus --config.file=config.yaml with the following configuration and see through the Prometheus UI.

config.yaml

global:                                                                                             
  scrape_interval:     5s                                                                           
  evaluation_interval: 15s                                                                          
  external_labels:                                                                                  
      monitor: 'codelab-monitor'                                                                    
rule_files:                                                                                         
scrape_configs:                                                                                     
  - job_name: 'spark-master'                                                                        
    metrics_path: '/metrics/master/prometheus/'                                                     
    static_configs:                                                                                 
      - targets: ['localhost:8080']                                                                 
  - job_name: 'spark-applications'                                                                  
    metrics_path: '/metrics/applications/prometheus/'                                               
    static_configs:                                                                                 
      - targets: ['localhost:8080']                                                                 
  - job_name: 'spark-worker'                                                                        
    metrics_path: '/metrics/prometheus/'                                                            
    static_configs:                                                                                 
      - targets: ['localhost:8081']                                                                 
  - job_name: 'spark-driver'                                                                        
    metrics_path: '/metrics/prometheus/'                                                            
    static_configs:                                                                                 
      - targets: ['localhost:4040']                                                                 
  - job_name: 'active-executors-and-stages'                                                         
    metrics_path: '/metrics/detail/prometheus/'                                                     
    static_configs:                                                                                 
      - targets: ['localhost:4040']

[dongjoon-hyun]

Hi, @vanzin and @srowen . Could you review this please?

[dongjoon-hyun]

Oops. I missed the task metrics. I'll update this PR later.

[srowen]

What is prometheus BTW? is it worth adding special code for?

[srowen]

If you want you can make kv into case (key, value) here for clarity, but not necessary

[dongjoon-hyun]

Thank you for review, @srowen . Prometheus.io is a CNCF project which is used widely with K8s.

• https://github.com/prometheus/prometheus

[SparkQA]

Test build #110491 has finished for PR 25741 at commit 8f063cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Although this is not a long PR, but I decided to split this into two PRs because PrometheusResource is a new feature aligned with SPARK-23429 (which is added at 3.0.0). That will make the review easier.

[dongjoon-hyun]

Sorry for this change~