[Celerdata] integration with CelerData

.codecov.yml

@@ -231,6 +231,10 @@ coverage:
         target: 75
         flags:
         - zenoh_router
+      celerdata:
+        target: 75
+        flags:
+        - celerdata
       cfssl:
         target: 75
         flags:
@@ -307,6 +311,11 @@ flags:
     paths:
     - bind9/datadog_checks/bind9
     - bind9/tests
+  celerdata:
+    carryforward: true
+    paths:
+    - celerdata/datadog_checks/celerdata
+    - celerdata/tests
   cfssl:
     carryforward: true
     paths:

.github/CODEOWNERS

@@ -236,6 +236,7 @@
 /notion/                                 @makenotion/public-api @DataDog/ecosystems-review
 /singlestoredb_cloud/                    @tdasarathan @DataDog/ecosystems-review
 /emqx/                                   @zhongwencool @zmstone @DataDog/ecosystems-review
+/celerdata/                              @yandongxiao yandongxiao@starrocks.com
 
 # Community Partners + Documentation
 /1e/*metadata.csv                        support@1e.com @DataDog/documentation @DataDog/ecosystems-review
@@ -1236,3 +1237,10 @@
 /emqx/assets/monitors                      @zhongwencool @zmstone @DataDog/documentation @DataDog/monitor-app @DataDog/agent-integrations
 /emqx/assets/logs/                         @zhongwencool @zmstone @DataDog/agent-integrations @DataDog/logs-backend
 /emqx/README.md                        @zhongwencool @zmstone @DataDog/documentation @DataDog/ecosystems-review
+
+/celerdata/*metadata.csv                     @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/manifest.json                     @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/README.md                         @yandongxiao yandongxiao@starrocks.com @DataDog/documentation
+/celerdata/assets/dashboards                    @yandongxiao yandongxiao@starrocks.com @DataDog/documentation @DataDog/dashboards-backend @DataDog/agent-integrations
+/celerdata/assets/monitors                      @yandongxiao yandongxiao@starrocks.com @DataDog/documentation @DataDog/monitor-app @DataDog/agent-integrations
+/celerdata/assets/logs/                         @yandongxiao yandongxiao@starrocks.com @DataDog/agent-integrations @DataDog/logs-backend

.github/workflows/test-all.yml

@@ -106,6 +106,25 @@ jobs:
       test-py3: ${{ inputs.test-py3 }}
       setup-env-vars: "${{ inputs.setup-env-vars }}"
     secrets: inherit
+  j9eb5f31:
+    uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
+    with:
+      job-name: celerdata
+      target: celerdata
+      platform: linux
+      runner: '["ubuntu-22.04"]'
+      repo: "${{ inputs.repo }}"
+      python-version: "${{ inputs.python-version }}"
+      standard: ${{ inputs.standard }}
+      latest: ${{ inputs.latest }}
+      agent-image: "${{ inputs.agent-image }}"
+      agent-image-py2: "${{ inputs.agent-image-py2 }}"
+      agent-image-windows: "${{ inputs.agent-image-windows }}"
+      agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
+      test-py2: ${{ inputs.test-py2 }}
+      test-py3: ${{ inputs.test-py3 }}
+      setup-env-vars: "${{ inputs.setup-env-vars }}"
+    secrets: inherit
   jcb3c31b:
     uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
     with:

celerdata/CHANGELOG.md

@@ -0,0 +1,8 @@
+# CHANGELOG - celerdata
+
+## 1.0.0 / 2024-03-11
+
+***Added***:
+
+* Initial Release
+* StarRocks provides a Prometheus-compatible information collection interface, and this integration collects metrics and logs from StarRocks.

celerdata/README.md

@@ -0,0 +1,67 @@
+# Agent Check: celerdata
+
+## Overview
+
+[CelerData][1] is the only SQL engine that is fast enough to run the most demanding workloads directly on your data lakehouse, all powered by StarRocks and cloud-managed by CelerData.
+
+StarRocks metrics enable database administrators to monitor their deployments. StarRocks provides a Prometheus-compatible information collection interface. 
+
+With this integration, we collect metrics and logs from StarRocks.
+
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates][2] for guidance on applying these instructions.
+
+### Installation
+
+To install the CelerData check on your host:
+
+1. Download and install the [Datadog Agent][8].
+2. To install the celerdata check on your host:
+
+   ```shell
+   datadog-agent integration install -t datadog-celerdata==<INTEGRATION_VERSION>
+   ```
+
+### Configuration
+
+1. Edit the `celerdata.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting metrics and logs. See the [celerdata.d/conf.yaml.example][3] for all available configuration options.
+
+2. Datadog listens on port 5000 for the dogstatsd_stats_port and expvar_port. In your celerdata.conf file, you will need to change the server.discovery.listen_address and the server.discovery.advertised_address to use a port other than 5000.
+
+3. [Restart the Agent][4].
+
+### Validation
+
+[Run the Agent's status subcommand][5] and look for `celerdata` under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+StarRocks metrics are collected as documented [here][9].
+
+See [metadata.csv][6] for the full list of metrics provided by this check.
+
+### Service Checks
+
+In the base check, Service check `celerdata_fe.prometheus.health` is submitted for FE component and `celerdata_be.prometheus.health` is submitted for BE component 
+
+### Events
+
+CelerData does not include any events.
+
+## Troubleshooting
+
+Need help? Contact [CelerData support][7].
+
+[1]: https://celerdata.com/
+[2]: https://docs.datadoghq.com/agent/autodiscovery/integrations
+[3]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/datadog_checks/celerdata/data/conf.yaml.example
+[4]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
+[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
+[6]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/metadata.csv
+[7]: todo@starrocks.com
+[8]: https://app.datadoghq.com/account/settings/agent/latest
+[9]: https://docs.starrocks.io/docs/administration/metrics/ 

celerdata/assets/configuration/spec.yaml

@@ -0,0 +1,73 @@
+name: celerdata
+
+files:
+
+- name: celerdata.yaml
+  options:
+  - template: instances
+    options:
+    - template: instances/openmetrics
+      overrides:
+        openmetrics_endpoint.display_priority: 5
+        openmetrics_endpoint.value.example: http://127.0.0.1:8030/metrics
+        openmetrics_endpoint.description: |
+          The URL exposing metrics in the OpenMetrics format.
+          You should configure this based on the component you're monitoring.
+          For FE component, use something like http://127.0.0.1:8030/metrics
+          For BE component, use something like http://127.0.0.1:8040/metrics
+  - template: logs
+    description: |
+      Log Section
+
+      type - required - Type of log input source (tcp / udp / file / windows_event).
+      port / path / channel_path - required - Set port if type is tcp or udp.
+                                              Set path if type is file.
+                                              Set channel_path if type is windows_event.
+      source  - required - Attribute that defines which integration sent the logs.
+      encoding - optional - For file specifies the file encoding. Default is utf-8. Other
+                            possible values are utf-16-le and utf-16-be.
+      service - optional - The name of the service that generates the log.
+                           Overrides any `service` defined in the `init_config` section.
+      tags - optional - Add tags to the collected logs.
+
+      Discover Datadog log collection: https://docs.datadoghq.com/logs/log_collection/
+      You should configure this based on the component you're monitoring.
+      For FE component, use items which has service: fe.
+      For BE component, use items which has service: be.
+    example:
+      - type: file
+        path: /path/to/fe.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.out
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.warn.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.audit.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.big_query.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/fe.dump.log
+        service: fe
+        source: celerdata
+      - type: file
+        path: /path/to/be.INFO
+        service: starrocks_be
+        source: celerdata
+      - type: file
+        path: /path/to/be.out
+        service: be
+        source: celerdata
+      - type: file
+        path: /path/to/be.WARNING
+        service: be
+        source: celerdata

celerdata/assets/service_checks.json

@@ -0,0 +1 @@
+[]

celerdata/datadog_checks/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

celerdata/datadog_checks/celerdata/__about__.py

@@ -0,0 +1 @@
+__version__ = '1.0.0'

celerdata/datadog_checks/celerdata/__init__.py

@@ -0,0 +1,4 @@
+from .__about__ import __version__
+from .celerdata import CelerdataCheck
+
+__all__ = ['__version__', 'CelerdataCheck']

celerdata/datadog_checks/celerdata/celerdata.py

@@ -0,0 +1,17 @@
+from datadog_checks.base import OpenMetricsBaseCheckV2
+
+
+class CelerdataCheck(OpenMetricsBaseCheckV2):
+    __NAMESPACE__ = 'celerdata'
+
+    def __init__(self, name, init_config, instances):
+        super(CelerdataCheck, self).__init__(name, init_config, instances)
+
+    def get_default_config(self):
+        """
+        Returns the default OpenMetrics configuration.
+        """
+        return {
+            'metrics': ['.*'],
+            'exclude_metrics': [r'.*8060.*'],
+        }

celerdata/datadog_checks/celerdata/config_models/__init__.py

@@ -0,0 +1,14 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+from .instance import InstanceConfig
+
+
+class ConfigMixin:
+    _config_model_instance: InstanceConfig
+
+    @property
+    def config(self) -> InstanceConfig:
+        return self._config_model_instance

celerdata/datadog_checks/celerdata/config_models/defaults.py

@@ -0,0 +1,120 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+
+def instance_allow_redirects():
+    return True
+
+
+def instance_auth_type():
+    return 'basic'
+
+
+def instance_cache_metric_wildcards():
+    return True
+
+
+def instance_cache_shared_labels():
+    return True
+
+
+def instance_collect_counters_with_distributions():
+    return False
+
+
+def instance_collect_histogram_buckets():
+    return True
+
+
+def instance_disable_generic_tags():
+    return False
+
+
+def instance_empty_default_hostname():
+    return False
+
+
+def instance_enable_health_service_check():
+    return True
+
+
+def instance_histogram_buckets_as_distributions():
+    return False
+
+
+def instance_ignore_connection_errors():
+    return False
+
+
+def instance_kerberos_auth():
+    return 'disabled'
+
+
+def instance_kerberos_delegate():
+    return False
+
+
+def instance_kerberos_force_initiate():
+    return False
+
+
+def instance_log_requests():
+    return False
+
+
+def instance_min_collection_interval():
+    return 15
+
+
+def instance_non_cumulative_histogram_buckets():
+    return False
+
+
+def instance_persist_connections():
+    return False
+
+
+def instance_request_size():
+    return 16
+
+
+def instance_skip_proxy():
+    return False
+
+
+def instance_tag_by_endpoint():
+    return True
+
+
+def instance_telemetry():
+    return False
+
+
+def instance_timeout():
+    return 10
+
+
+def instance_tls_ignore_warning():
+    return False
+
+
+def instance_tls_use_host_header():
+    return False
+
+
+def instance_tls_verify():
+    return True
+
+
+def instance_use_latest_spec():
+    return False
+
+
+def instance_use_legacy_auth_encoding():
+    return True
+
+
+def instance_use_process_start_time():
+    return False