[Celerdata] integration with CelerData

.codecov.yml

@@ -231,6 +231,10 @@ coverage:
         target: 75
         flags:
         - zenoh_router
+      celerdata:
+        target: 75
+        flags:
+        - celerdata
       cfssl:
         target: 75
         flags:
@@ -307,6 +311,11 @@ flags:
     paths:
     - bind9/datadog_checks/bind9
     - bind9/tests
+  celerdata:
+    carryforward: true
+    paths:
+    - celerdata/datadog_checks/celerdata
+    - celerdata/tests
   cfssl:
     carryforward: true
     paths:

.github/CODEOWNERS

@@ -236,6 +236,7 @@
 /notion/                                 @makenotion/public-api @DataDog/ecosystems-review
 /singlestoredb_cloud/                    @tdasarathan @DataDog/ecosystems-review
 /emqx/                                   @zhongwencool @zmstone @DataDog/ecosystems-review
+/celerdata/                              @yandongxiao yandongxiao@starrocks.com
 
 # Community Partners + Documentation
 /1e/*metadata.csv                        support@1e.com @DataDog/documentation @DataDog/ecosystems-review
@@ -1236,3 +1237,10 @@
 /emqx/assets/monitors                      @zhongwencool @zmstone @DataDog/documentation @DataDog/monitor-app @DataDog/agent-integrations
 /emqx/assets/logs/                         @zhongwencool @zmstone @DataDog/agent-integrations @DataDog/logs-backend
 /emqx/README.md                        @zhongwencool @zmstone @DataDog/documentation @DataDog/ecosystems-review
+
+/celerdata/*metadata.csv                     @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/manifest.json                     @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/README.md                         @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/assets/dashboards                    @yandongxiao yandongxiao@starrocks.com @DataDog/documentation @DataDog/dashboards-backend @DataDog/agent-integrations
+/celerdata/assets/monitors                      @yandongxiao yandongxiao@starrocks.com @DataDog/documentation @DataDog/monitor-app @DataDog/agent-integrations
+/celerdata/assets/logs/                         @yandongxiao yandongxiao@starrocks.com @DataDog/agent-integrations @DataDog/logs-backend

.github/workflows/test-all.yml

@@ -106,6 +106,25 @@ jobs:
       test-py3: ${{ inputs.test-py3 }}
       setup-env-vars: "${{ inputs.setup-env-vars }}"
     secrets: inherit
+  j9eb5f31:
+    uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
+    with:
+      job-name: celerdata
+      target: celerdata
+      platform: linux
+      runner: '["ubuntu-22.04"]'
+      repo: "${{ inputs.repo }}"
+      python-version: "${{ inputs.python-version }}"
+      standard: ${{ inputs.standard }}
+      latest: ${{ inputs.latest }}
+      agent-image: "${{ inputs.agent-image }}"
+      agent-image-py2: "${{ inputs.agent-image-py2 }}"
+      agent-image-windows: "${{ inputs.agent-image-windows }}"
+      agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
+      test-py2: ${{ inputs.test-py2 }}
+      test-py3: ${{ inputs.test-py3 }}
+      setup-env-vars: "${{ inputs.setup-env-vars }}"
+    secrets: inherit
   jcb3c31b:
     uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
     with:

celerdata/CHANGELOG.md

@@ -0,0 +1,8 @@
+# CHANGELOG - celerdata
+
+## 1.0.0 / 2024-03-11
+
+***Added***:
+
+* Initial Release
+* StarRocks provides a Prometheus-compatible information collection interface, and this integration collects metrics and logs from StarRocks.

celerdata/README.md

@@ -0,0 +1,64 @@
+# Agent Check: celerdata
+
+## Overview
+
+[CelerData][1], based on the open source project StarRocks, is a commercial SQL engine designed for high-speed data analytics directly on your data lake house. It offers a cloud-managed solution that enhances the performance and scalability of running complex queries and workloads.
+
+This integration allows users to collect Prometheus metrics and logs. By leveraging this integration, customers can gain insights into query performance, system health, and resource utilization, enabling them to optimize their data analytics operations effectively.
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates][2] for guidance on applying these instructions.
+
+### Installation
+
+To collect StarRocks [metrics][9] and logs:
+
+1. Download and install the [Datadog Agent][8].
+2. Install the CelerData check on your host with the following command:
+
+   ```shell
+   datadog-agent integration install -t datadog-celerdata==1.0.0
+   ```
+
+### Configuration
+
+1. Edit the `celerdata.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory, to start collecting metrics and logs. See the [celerdata.d/conf.yaml.example][3] for all available configuration options.
+
+2. Datadog listens on port 5000 for the `dogstatsd_stats_port` and `expvar_port`. In your `celerdata.conf` file, you will need to change the `server.discovery.listen_address` and the `server.discovery.advertised_address` to use a port other than 5000.
+
+3. [Restart the Agent][4].
+
+### Validation
+
+[Run the Agent's status subcommand][5] and look for `celerdata` under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+
+See [metadata.csv][6] for a list of metrics provided by this integration.
+
+### Service Checks
+
+See [service_checks.json][10] for a list of service checks provided by this integration.
+
+### Events
+
+The CelerData integration does not include any events.
+
+## Troubleshooting
+
+Need help? Contact [CelerData support][7].
+
+[1]: https://celerdata.com/
+[2]: https://docs.datadoghq.com/agent/autodiscovery/integrations
+[3]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/datadog_checks/celerdata/data/conf.yaml.example
+[4]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
+[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
+[6]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/metadata.csv
+[7]: support@celerdata.com
+[8]: https://app.datadoghq.com/account/settings/agent/latest
+[9]: https://docs.starrocks.io/docs/administration/metrics/ 
+[10]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/service_checks.json

celerdata/assets/configuration/spec.yaml

@@ -0,0 +1,73 @@
+name: celerdata
+
+files:
+
+- name: celerdata.yaml
+  options:
+  - template: instances
+    options:
+    - template: instances/openmetrics
+      overrides:
+        openmetrics_endpoint.display_priority: 5
+        openmetrics_endpoint.value.example: http://127.0.0.1:8030/metrics
+        openmetrics_endpoint.description: |
+          The URL exposing metrics in the OpenMetrics format.
+          You should configure this based on the component you're monitoring.
+          For FE component, use something like http://127.0.0.1:8030/metrics
+          For BE component, use something like http://127.0.0.1:8040/metrics
+  - template: logs
+    description: |
+      Log Section
+
+      type - required - Type of log input source (tcp / udp / file / windows_event).
+      port / path / channel_path - required - Set port if type is tcp or udp.
+                                              Set path if type is file.
+                                              Set channel_path if type is windows_event.
+      source  - required - Attribute that defines which integration sent the logs.
+      encoding - optional - For file specifies the file encoding. Default is utf-8. Other
+                            possible values are utf-16-le and utf-16-be.
+      service - optional - The name of the service that generates the log.
+                           Overrides any `service` defined in the `init_config` section.
+      tags - optional - Add tags to the collected logs.
+
+      Discover Datadog log collection: https://docs.datadoghq.com/logs/log_collection/
+      You should configure this based on the component you're monitoring.
+      For FE component, use items which has service: fe.
+      For BE component, use items which has service: be.
+    example:
+      - type: file
+        path: /path/to/fe.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.out
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.warn.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.audit.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.big_query.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.dump.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/be.INFO
+        service: starrocks_be
+        source: celerdata
+      - type: file
+        path: /path/to/be.out
+        service: be
+        source: celerdata
+      - type: file
+        path: /path/to/be.WARNING
+        service: be
+        source: celerdata

celerdata/assets/service_checks.json

@@ -0,0 +1,31 @@
+[
+    {
+        "agent_version": "7.35.0",
+        "integration": "CelerData",
+        "check": "celerdata_fe.prometheus.health",
+        "statuses": [
+            "ok",
+            "critical"
+        ],
+        "groups": [
+            "host"
+        ],
+        "name": "Can Connect",
+        "description": "Returns CRITICAL if the Agent cannot connect to CelerData to collect metrics. Returns OK otherwise."
+    },
+    {
+        "agent_version": "7.35.0",
+        "integration": "CelerData",
+        "check": "celerdata_be.prometheus.health",
+        "statuses": [
+            "ok",
+            "critical"
+        ],
+        "groups": [
+            "host"
+        ],
+        "name": "Can Connect",
+        "description": "Returns CRITICAL if the Agent cannot connect to CelerData to collect metrics. Returns OK otherwise."
+    }
+]
+

celerdata/datadog_checks/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

celerdata/datadog_checks/celerdata/__about__.py

@@ -0,0 +1 @@
+__version__ = '1.0.0'

celerdata/datadog_checks/celerdata/__init__.py

@@ -0,0 +1,4 @@
+from .__about__ import __version__
+from .celerdata import CelerdataCheck
+
+__all__ = ['__version__', 'CelerdataCheck']

celerdata/datadog_checks/celerdata/celerdata.py

@@ -0,0 +1,17 @@
+from datadog_checks.base import OpenMetricsBaseCheckV2
+
+
+class CelerdataCheck(OpenMetricsBaseCheckV2):
+    __NAMESPACE__ = 'celerdata'
+
+    def __init__(self, name, init_config, instances):
+        super(CelerdataCheck, self).__init__(name, init_config, instances)
+
+    def get_default_config(self):
+        """
+        Returns the default OpenMetrics configuration.
+        """
+        return {
+            'metrics': ['.*'],
+            'exclude_metrics': [r'.*8060.*'],
+        }

celerdata/datadog_checks/celerdata/config_models/__init__.py

@@ -0,0 +1,14 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+from .instance import InstanceConfig
+
+
+class ConfigMixin:
+    _config_model_instance: InstanceConfig
+
+    @property
+    def config(self) -> InstanceConfig:
+        return self._config_model_instance