Add Cloudera integration

.azure-pipelines/templates/test-all-checks.yml

@@ -120,6 +120,9 @@ jobs:
     - checkName: cloud_foundry_api
       displayName: Cloud Foundry API
       os: linux
+    - checkName: cloudera
+      displayName: Cloudera
+      os: linux
     - checkName: cockroachdb
       displayName: CockroachDB
       os: linux

.codecov.yml

@@ -114,6 +114,10 @@ coverage:
         target: 75
         flags:
         - cloud_foundry_api
+      Cloudera:
+        target: 75
+        flags:
+        - cloudera
       CockroachDB:
         target: 75
         flags:
@@ -736,6 +740,11 @@ flags:
     paths:
     - cloud_foundry_api/datadog_checks/cloud_foundry_api
     - cloud_foundry_api/tests
+  cloudera:
+    carryforward: true
+    paths:
+    - cloudera/datadog_checks/cloudera
+    - cloudera/tests
   cockroachdb:
     carryforward: true
     paths:

.github/workflows/config/labeler.yml

@@ -94,6 +94,8 @@ integration/clickhouse:
 - clickhouse/**/*
 integration/cloud_foundry_api:
 - cloud_foundry_api/**/*
+integration/cloudera:
+- cloudera/**/*
 integration/cockroachdb:
 - cockroachdb/**/*
 integration/confluent_platform:

LICENSE-3rdparty.csv

@@ -19,6 +19,7 @@ cachetools,PyPI,MIT,Thomas Kemmer
 check-postgres,"https://github.com/bucardo/",BSD-2-Clause,Greg Sabino Mullane
 clickhouse-cityhash,PyPI,MIT,Alexander [Amper] Marshalov
 clickhouse-driver,PyPI,MIT,Konstantin Lebedev
+cm-client,PyPI,Apache-2.0,
 contextlib2,PyPI,PSF,Nick Coghlan
 cryptography,PyPI,Apache-2.0,The Python Cryptographic Authority and individual contributors | The cryptography developers
 cryptography,PyPI,BSD-3-Clause,The Python Cryptographic Authority and individual contributors | The cryptography developers

cloudera/CHANGELOG.md

@@ -0,0 +1,2 @@
+# CHANGELOG - Cloudera
+

cloudera/README.md

@@ -0,0 +1,150 @@
+# Agent Check: Cloudera
+
+## Overview
+
+This integration monitors your [Cloudera Data Platform][1] through the Datadog Agent, allowing you to submit metrics and service checks on the health of your Cloudera Data Hub clusters, hosts, and roles.  
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates][3] for guidance on applying these instructions.
+
+### Installation
+
+The Cloudera check is included in the [Datadog Agent][2] package.
+No additional installation is needed on your server.
+
+### Configuration
+
+#### Prepare Cloudera Manager
+1. In Cloudera Data Platform, navigate to the Management Console and click on the **User Management** tab.
+![User Management][10]
+
+2. Click on **Actions**, then **Create Machine User** to create the machine user that queries the Cloudera Manager through the Datadog Agent.
+![Create Machine User][11]
+
+3. If the workload password hasn't been set, click on **Set Workload Password** after the user is created.
+![Set Workload Password][12]
+
+<!-- xxx tabs xxx -->
+<!-- xxx tab "Host" xxx -->
+
+#### Host
+1. Edit the `cloudera.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting your Cloudera cluster and host data. See the [sample cloudera.d/conf.yaml][4] for all available configuration options. Note that the `api_url` should contain the API version at the end.
+
+   ```yaml
+   init_config:
+
+      ## @param workload_username - string - required
+      ## The Workload username. This value can be found in the `User Management` tab of the Management 
+      ## Console in the `Workload User Name`.
+      #
+      workload_username: <WORKLOAD_USERNAME>
+
+      ## @param workload_password - string - required
+      ## The Workload password. This value can be found in the `User Management` tab of the Management 
+      ## Console in the `Workload Password`.
+      #
+      workload_password: <WORKLOAD_PASSWORD>
+
+   ## Every instance is scheduled independently of the others.
+   #
+   instances:
+
+      ## @param api_url - string - required
+      ## The URL endpoint for the Cloudera Manager API. This can be found under the Endpoints tab for 
+      ## your Data Hub to monitor. 
+      ##
+      ## Note: The version of the Cloudera Manager API needs to be appended at the end of the URL. 
+      ## For example, using v48 of the API for Data Hub `cluster_1` should result with a URL similar 
+      ## to the following:
+      ## `https://cluster1.cloudera.site/cluster_1/cdp-proxy-api/cm-api/v48`
+      #
+      - api_url: <API_URL>
+   ```
+
+2. [Restart the Agent][5] to start collecting and sending Cloudera Data Hub cluster data to Datadog.
+
+<!-- xxz tab xxx -->
+<!-- xxx tab "Containerized" xxx -->
+
+#### Containerized
+
+For containerized environments, see the [Autodiscovery Integration Templates][9] for guidance on applying the parameters below.
+
+| Parameter            | Value                                                                                                            |
+| -------------------- | ---------------------------------------------------------------------------------------------------------------- |
+| `<INTEGRATION_NAME>` | `cloudera`                                                                                                       |
+| `<INIT_CONFIG>`      | `{"workload_username": "<WORKLOAD_USERNAME>", 'workload_password": "<WORKLOAD_PASSWORD>"}`                       |
+| `<INSTANCE_CONFIG>`  | `{"api_url": <API_URL>"}`                                                                                        |
+
+<!-- xxz tab xxx -->
+<!-- xxz tabs xxx -->
+
+### Validation
+
+[Run the Agent's status subcommand][6] and look for `cloudera` under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+See [metadata.csv][7] for a list of metrics provided by this integration.
+
+### Events
+
+The Cloudera integration does not include any events.
+
+### Service Checks
+
+See [service_checks.json][8] for a list of service checks provided by this integration.
+
+## Troubleshooting
+
+### Collecting metrics of Datadog integrations on Cloudera hosts
+To install the Datadog Agent on a Cloudera host, make sure that the security group associated with the host allows SSH access. 
+Then, you need to use the [root user `cloudbreak`][13] when accessing the host with the SSH key generated during the environment creation:
+
+```
+sudo ssh -i "/path/to/key.pem" cloudbreak@<HOST_IP_ADDRESS>
+```
+
+The workload username and password can be used to access Cloudera hosts via SSH, although only the `cloudbreak` user can install the Datadog Agent. 
+Trying to use any user that is not `cloudbreak` may result in the following error:
+```
+<NON_CLOUDBREAK_USER> is not allowed to run sudo on <CLOUDERA_HOSTNAME>.  This incident will be reported.
+```
+
+### Config errors when collecting Datadog metrics
+If you something similar to the following in the Agent status when collecting metrics from your Cloudera host:
+
+```
+  Config Errors
+  ==============
+    zk
+    --
+      open /etc/datadog-agent/conf.d/zk.d/conf.yaml: permission denied
+```
+
+you need to change the ownership of the `conf.yaml` to `dd-agent`:
+
+```
+[cloudbreak@<CLOUDERA_HOSTNAME> ~]$ sudo chown -R dd-agent:dd-agent /etc/datadog-agent/conf.d/zk.d/conf.yaml
+```
+
+
+Need help? Contact [Datadog support][9].
+
+
+[1]: https://www.cloudera.com/products/cloudera-data-platform.html
+[2]: https://app.datadoghq.com/account/settings#agent
+[3]: https://docs.datadoghq.com/agent/kubernetes/integrations/
+[4]: https://github.com/DataDog/integrations-core/blob/master/cloudera/datadog_checks/cloudera/data/conf.yaml.example
+[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
+[6]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
+[7]: https://github.com/DataDog/integrations-core/blob/master/cloudera/metadata.csv
+[8]: https://github.com/DataDog/integrations-core/blob/master/cloudera/assets/service_checks.json
+[9]: https://docs.datadoghq.com/help/
+[10]: https://raw.githubusercontent.com/DataDog/integrations-core/master/cloudera/images/user_management.png
+[11]: https://raw.githubusercontent.com/DataDog/integrations-core/master/cloudera/images/create_machine_user.png
+[12]: https://raw.githubusercontent.com/DataDog/integrations-core/master/cloudera/images/set_workload_password.png
+[13]: https://docs.cloudera.com/data-hub/cloud/access-clusters/topics/mc-accessing-cluster-via-ssh.html

cloudera/assets/configuration/spec.yaml

@@ -0,0 +1,44 @@
+name: Cloudera
+files:
+- name: cloudera.yaml
+  options:
+  - template: init_config
+    options:
+    - name: workload_username
+      description: |
+        The Workload username. This value can be found in the `User Management` tab of the Management 
+        Console in the `Workload User Name`.
+      required: true
+      value:
+        type: string
+    - name: workload_password
+      description: |
+        The Workload password. This value can be found in the `User Management` tab of the Management 
+        Console in the `Workload Password`.
+      required: true
+      value:
+        type: string
+    - template: init_config/default
+  - template: instances
+    options:
+    - name: api_url
+      required: true
+      description: |
+        The URL endpoint for the Cloudera Manager API. This can be found under the Endpoints tab for 
+        your Data Hub to monitor. 
+
+        Note: The version of the Cloudera Manager API needs to be appended at the end of the URL. 
+        For example, using v48 of the API for Data Hub `cluster_1` should result with a URL similar 
+        to the following:
+        `https://cluster1.cloudera.site/cluster_1/cdp-proxy-api/cm-api/v48`
+
+      value:
+        type: string
+    - name: max_parallel_requests
+      description: | 
+        The maximum number of requests to Cloudera Manager that are allowed in parallel.
+      hidden: true
+      value:
+        type: integer
+        example: 100
+    - template: instances/default

cloudera/assets/service_checks.json

@@ -0,0 +1,50 @@
+[
+  {
+    "agent_version": "7.43.0",
+    "integration": "Cloudera",
+    "check": "cloudera.can_connect",
+    "statuses": [
+      "ok",
+      "critical"
+    ],
+    "groups": [
+      "api_url"
+    ],
+    "name": "Cloudera Manager Can Connect",
+    "description": "Returns `OK` if the check is able to connect to the Cloudera Manager API and collect metrics, `CRITICAL` otherwise."
+  },
+  {
+    "agent_version": "7.43.0",
+    "integration": "Cloudera",
+    "check": "cloudera.cluster.health",
+    "statuses": [
+      "ok",
+      "critical",
+      "warning",
+      "unknown"
+    ],
+    "groups": [
+      "cloudera_cluster"
+    ],
+    "name": "Cloudera Cluster Health",
+    "description": "Returns `OK` if the cluster is in good health or is starting, `WARNING` if the cluster is stopping or the health is concerning, `CRITICAL` if the cluster is down or in bad health, and `UNKNOWN` otherwise."
+  },
+  {
+    "agent_version": "7.43.0",
+    "integration": "Cloudera",
+    "check": "cloudera.host.health",
+    "statuses": [
+      "ok",
+      "critical",
+      "warning",
+      "unknown"
+    ],
+    "groups": [
+      "cloudera_cluster",
+      "cloudera_rack_id",
+      "cloudera_hostname"
+    ],
+    "name": "Cloudera Host Health",
+    "description": "Returns `OK` if the host is in good health or is starting, `WARNING` if the host is stopping or the health is concerning, `CRITICAL` if the host is down or in bad health, and `UNKNOWN` otherwise."
+  }
+]

cloudera/datadog_checks/__init__.py

@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2022-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

cloudera/datadog_checks/cloudera/__about__.py

@@ -0,0 +1,4 @@
+# (C) Datadog, Inc. 2022-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+__version__ = '0.0.1'

cloudera/datadog_checks/cloudera/__init__.py

@@ -0,0 +1,7 @@
+# (C) Datadog, Inc. 2022-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+from .__about__ import __version__
+from .check import ClouderaCheck
+
+__all__ = ['__version__', 'ClouderaCheck']

cloudera/datadog_checks/cloudera/api_client.py

@@ -0,0 +1,13 @@
+from abc import ABC, abstractmethod
+
+
+class ApiClient(ABC):
+    def __init__(self, check, api_client):
+        self._check = check
+        self._log = check.log
+        self._api_client = api_client
+
+    @abstractmethod
+    def collect_data(self):
+        """Collect metrics and service checks via the Cloudera API Client"""
+        pass

cloudera/datadog_checks/cloudera/api_client_factory.py

@@ -0,0 +1,40 @@
+import cm_client
+from cm_client.rest import RESTClientObject
+from packaging.version import parse
+
+from datadog_checks.base import ConfigurationError
+from datadog_checks.cloudera.api_client_v7 import ApiClientV7
+
+
+def make_api_client(check, config, shared_config):
+    cm_client.configuration.username = shared_config.workload_username
+    cm_client.configuration.password = shared_config.workload_password
+    api_client = cm_client.ApiClient(config.api_url)
+    api_client.rest_client = RESTClientObject(maxsize=(config.max_parallel_requests))
+    check.log.debug('Getting version from cloudera API URL: %s', config.api_url)
+    cloudera_manager_resource_api = cm_client.ClouderaManagerResourceApi(api_client)
+    try:
+        get_version_response = cloudera_manager_resource_api.get_version()
+    except Exception:
+        check.log.warning(
+            "Unable to get the version of Cloudera Manager, please check that the URL is valid and API version \
+                is appended at the end"
+        )
+        raise
+    check.log.debug('get_version_response: %s', get_version_response)
+    response_version = get_version_response.version
+    if response_version:
+        cloudera_version = parse(response_version)
+        check.log.debug('Cloudera Manager Version: %s', cloudera_version)
+        if cloudera_version.major == 7:
+            version_raw = str(cloudera_version)
+            version_parts = {
+                'major': str(cloudera_version.major),
+                'minor': str(cloudera_version.minor),
+                'patch': str(cloudera_version.micro),
+            }
+            check.set_metadata('version', version_raw, scheme='parts', part_map=version_parts)
+
+            return ApiClientV7(check, api_client)
+
+    raise ConfigurationError(f'Cloudera Manager Version is unsupported or unknown: {response_version}')