Skip to content

Files

Latest commit

 

History

History
 
 

mapreduce

README.md

Mapreduce Integration

MapReduce Dashboard

Overview

Get metrics from mapreduce service in real time to:

  • Visualize and monitor mapreduce states
  • Be notified about mapreduce failovers and events.

Setup

Installation

The Mapreduce check is included in the Datadog Agent package, so you don't need to install anything else on your servers.

Configuration

Host

To configure this check for an Agent running on a host:

  1. Edit the mapreduce.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration directory to point to your server and port, set the masters to monitor. See the sample mapreduce.d/conf.yaml for all available configuration options.

  2. Restart the Agent.

Log collection

  1. Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

    logs_enabled: true

  2. Uncomment and edit the logs configuration block in your mapreduce.d/conf.yaml file. Change the type, path, and service parameter values based on your environment. See the sample mapreduce.d/conf.yaml for all available configuration options.

    logs:
  - type: file
    path: <LOG_FILE_PATH>
    source: mapreduce
    service: <SERVICE_NAME>
    # To handle multi line that starts with yyyy-mm-dd use the following pattern
    # log_processing_rules:
    #   - type: multi_line
    #     pattern: \d{4}\-\d{2}\-\d{2} \d{2}:\d{2}:\d{2},\d{3}
    #     name: new_log_start_with_date

  3. Restart the Agent.

Containerized

For containerized environments, see the Autodiscovery Integration Templates for guidance on applying the parameters below.

Parameter Value
<INTEGRATION_NAME> mapreduce
<INIT_CONFIG> blank or {}
<INSTANCE_CONFIG> {"resourcemanager_uri": "https://%%host%%:8088", "cluster_name":"<MAPREDUCE_CLUSTER_NAME>"}
Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see the Docker Log Collection.

Then, set log integrations as Docker labels:

LABEL "com.datadoghq.ad.logs"='[{"source": "mapreduce", "service": "<SERVICE_NAME>"}]'

Validation

Run the Agent's status subcommand and look for mapreduce under the Checks section.

Data Collected

Metrics

See metadata.csv for a list of metrics provided by this integration.

Events

The Mapreduce check does not include any events.

Service Checks

See service_checks.json for a list of service checks provided by this integration.

Troubleshooting

Need help? Contact Datadog support.

Further Reading