[Celerdata] integration with CelerData

Signed-off-by: yandongxiao <yandongxiao@starrocks.com>

.codecov.yml

@@ -231,6 +231,10 @@ coverage:
         target: 75
         flags:
         - zenoh_router
+      celerdata:
+        target: 75
+        flags:
+        - celerdata
       cfssl:
         target: 75
         flags:
@@ -307,6 +311,11 @@ flags:
     paths:
     - bind9/datadog_checks/bind9
     - bind9/tests
+  celerdata:
+    carryforward: true
+    paths:
+    - celerdata/datadog_checks/celerdata
+    - celerdata/tests
   cfssl:
     carryforward: true
     paths:

.github/workflows/test-all.yml

@@ -106,6 +106,25 @@ jobs:
       test-py3: ${{ inputs.test-py3 }}
       setup-env-vars: "${{ inputs.setup-env-vars }}"
     secrets: inherit
+  j9eb5f31:
+    uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
+    with:
+      job-name: celerdata
+      target: celerdata
+      platform: linux
+      runner: '["ubuntu-22.04"]'
+      repo: "${{ inputs.repo }}"
+      python-version: "${{ inputs.python-version }}"
+      standard: ${{ inputs.standard }}
+      latest: ${{ inputs.latest }}
+      agent-image: "${{ inputs.agent-image }}"
+      agent-image-py2: "${{ inputs.agent-image-py2 }}"
+      agent-image-windows: "${{ inputs.agent-image-windows }}"
+      agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
+      test-py2: ${{ inputs.test-py2 }}
+      test-py3: ${{ inputs.test-py3 }}
+      setup-env-vars: "${{ inputs.setup-env-vars }}"
+    secrets: inherit
   jcb3c31b:
     uses: DataDog/integrations-core/.github/workflows/test-target.yml@master
     with:

celerdata/CHANGELOG.md

@@ -0,0 +1,8 @@
+# CHANGELOG - celerdata
+
+## 1.0.0 / 2024-03-11
+
+***Added***:
+
+* Initial Release
+* StarRocks provides a Prometheus-compatible information collection interface, and this integration collects metrics and logs from StarRocks.

celerdata/README.md

@@ -0,0 +1,67 @@
+# Agent Check: celerdata
+
+## Overview
+
+[CelerData][1] is the only SQL engine that is fast enough to run the most demanding workloads directly on your data lakehouse, all powered by StarRocks and cloud-managed by CelerData.
+
+StarRocks metrics enable database administrators to monitor their deployments. StarRocks provides a Prometheus-compatible information collection interface. 
+
+With this integration, we collect metrics and logs from StarRocks.
+
+
+## Setup
+
+Follow the instructions below to install and configure this check for an Agent running on a host. For containerized environments, see the [Autodiscovery Integration Templates][2] for guidance on applying these instructions.
+
+### Installation
+
+To install the CelerData check on your host:
+
+1. Download and install the [Datadog Agent][8].
+2. To install the celerdata check on your host:
+
+   ```shell
+   datadog-agent integration install -t datadog-celerdata==<INTEGRATION_VERSION>
+   ```
+
+### Configuration
+
+1. Edit the `celerdata.d/conf.yaml` file, in the `conf.d/` folder at the root of your Agent's configuration directory to start collecting metrics and logs. See the [celerdata.d/conf.yaml.example][3] for all available configuration options.
+
+2. Datadog listens on port 5000 for the dogstatsd_stats_port and expvar_port. In your celerdata.conf file, you will need to change the server.discovery.listen_address and the server.discovery.advertised_address to use a port other than 5000.
+
+3. [Restart the Agent][4].
+
+### Validation
+
+[Run the Agent's status subcommand][5] and look for `celerdata` under the Checks section.
+
+## Data Collected
+
+### Metrics
+
+StarRocks metrics are collected as documented [here][9].
+
+See [metadata.csv][6] for the full list of metrics provided by this check.
+
+### Service Checks
+
+In the base check, Service check `celerdata_fe.prometheus.health` is submitted for FE component and `celerdata_be.prometheus.health` is submitted for BE component 
+
+### Events
+
+CelerData does not include any events.
+
+## Troubleshooting
+
+Need help? Contact [CelerData support][7].
+
+[1]: https://celerdata.com/
+[2]: https://docs.datadoghq.com/agent/autodiscovery/integrations
+[3]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/datadog_checks/celerdata/data/conf.yaml.example
+[4]: https://docs.datadoghq.com/agent/guide/agent-commands/#start-stop-and-restart-the-agent
+[5]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
+[6]: https://github.com/DataDog/integrations-extras/blob/master/celerdata/metadata.csv
+[7]: todo@starrocks.com
+[8]: https://app.datadoghq.com/account/settings/agent/latest
+[9]: https://docs.starrocks.io/docs/administration/metrics/ 

celerdata/assets/configuration/spec.yaml

@@ -0,0 +1,73 @@
+name: celerdata
+
+files:
+
+- name: celerdata.yaml
+  options:
+  - template: instances
+    options:
+    - template: instances/openmetrics
+      overrides:
+        openmetrics_endpoint.display_priority: 5
+        openmetrics_endpoint.value.example: http://127.0.0.1:8030/metrics
+        openmetrics_endpoint.description: |
+          The URL exposing metrics in the OpenMetrics format.
+          You should configure this based on the component you're monitoring.
+          For FE component, use something like http://127.0.0.1:8030/metrics
+          For BE component, use something like http://127.0.0.1:8040/metrics
+    - template: logs
+      description: |
+        Log Section
+        
+        type - required - Type of log input source (tcp / udp / file / windows_event).
+        port / path / channel_path - required - Set port if type is tcp or udp.
+                                                Set path if type is file.
+                                                Set channel_path if type is windows_event.
+        source  - required - Attribute that defines which integration sent the logs.
+        encoding - optional - For file specifies the file encoding. Default is utf-8. Other
+                              possible values are utf-16-le and utf-16-be.
+        service - optional - The name of the service that generates the log.
+                             Overrides any `service` defined in the `init_config` section.
+        tags - optional - Add tags to the collected logs.
+        
+        Discover Datadog log collection: https://docs.datadoghq.com/logs/log_collection/
+        You should configure this based on the component you're monitoring.
+        For FE component, use items which has service: fe.
+        For BE component, use items which has service: be.
+      example:
+        - type: file
+          path: /path/to/fe.log
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/fe.out
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/fe.warn.log
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/fe.audit.log
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/fe.big_query.log
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/fe.dump.log
+          service: fe
+          source: celerdata
+        - type: file
+          path: /path/to/be.INFO
+          service: starrocks_be
+          source: celerdata
+        - type: file
+          path: /path/to/be.out
+          service: be
+          source: celerdata
+        - type: file
+          path: /path/to/be.WARNING
+          service: be
+          source: celerdata

celerdata/assets/service_checks.json

@@ -0,0 +1 @@
+[]

celerdata/datadog_checks/__init__.py

@@ -0,0 +1 @@
+__path__ = __import__('pkgutil').extend_path(__path__, __name__)  # type: ignore

celerdata/datadog_checks/celerdata/__about__.py

@@ -0,0 +1 @@
+__version__ = '1.0.0'

celerdata/datadog_checks/celerdata/__init__.py

@@ -0,0 +1,4 @@
+from .__about__ import __version__
+from .celerdata import CelerdataCheck
+
+__all__ = ['__version__', 'CelerdataCheck']

celerdata/datadog_checks/celerdata/celerdata.py

@@ -0,0 +1,17 @@
+from datadog_checks.base import OpenMetricsBaseCheckV2
+
+
+class CelerdataCheck(OpenMetricsBaseCheckV2):
+    __NAMESPACE__ = 'celerdata'
+
+    def __init__(self, name, init_config, instances):
+        super(CelerdataCheck, self).__init__(name, init_config, instances)
+
+    def get_default_config(self):
+        """
+        Returns the default OpenMetrics configuration.
+        """
+        return {
+            'metrics': ['.*'],
+            'exclude_metrics': [r'.*8060.*'],
+        }

celerdata/datadog_checks/celerdata/config_models/__init__.py

@@ -0,0 +1,21 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+
+from .instance import InstanceConfig
+from .shared import SharedConfig
+
+
+class ConfigMixin:
+    _config_model_instance: InstanceConfig
+    _config_model_shared: SharedConfig
+
+    @property
+    def config(self) -> InstanceConfig:
+        return self._config_model_instance
+
+    @property
+    def shared_config(self) -> SharedConfig:
+        return self._config_model_shared

celerdata/datadog_checks/celerdata/config_models/defaults.py

@@ -0,0 +1,12 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+
+def instance_empty_default_hostname():
+    return False
+
+
+def instance_min_collection_interval():
+    return 15

celerdata/datadog_checks/celerdata/config_models/instance.py

@@ -0,0 +1,47 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel, ConfigDict, field_validator, model_validator
+
+from datadog_checks.base.utils.functions import identity
+from datadog_checks.base.utils.models import validation
+
+from . import defaults, validators
+
+
+class InstanceConfig(BaseModel):
+    model_config = ConfigDict(
+        validate_default=True,
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    empty_default_hostname: Optional[bool] = None
+    min_collection_interval: Optional[float] = None
+    service: Optional[str] = None
+    tags: Optional[tuple[str, ...]] = None
+
+    @model_validator(mode='before')
+    def _initial_validation(cls, values):
+        return validation.core.initialize_config(getattr(validators, 'initialize_instance', identity)(values))
+
+    @field_validator('*', mode='before')
+    def _validate(cls, value, info):
+        field = cls.model_fields[info.field_name]
+        field_name = field.alias or info.field_name
+        if field_name in info.context['configured_fields']:
+            value = getattr(validators, f'instance_{info.field_name}', identity)(value, field=field)
+        else:
+            value = getattr(defaults, f'instance_{info.field_name}', lambda: value)()
+
+        return validation.utils.make_immutable(value)
+
+    @model_validator(mode='after')
+    def _final_validation(cls, model):
+        return validation.core.check_model(getattr(validators, 'check_instance', identity)(model))

celerdata/datadog_checks/celerdata/config_models/shared.py

@@ -0,0 +1,44 @@
+# This file is autogenerated.
+# To change this file you should edit assets/configuration/spec.yaml and then run the following commands:
+#     ddev -x validate config -s <INTEGRATION_NAME>
+#     ddev -x validate models -s <INTEGRATION_NAME>
+
+
+from __future__ import annotations
+
+from typing import Optional
+
+from pydantic import BaseModel, ConfigDict, field_validator, model_validator
+
+from datadog_checks.base.utils.functions import identity
+from datadog_checks.base.utils.models import validation
+
+from . import defaults, validators
+
+
+class SharedConfig(BaseModel):
+    model_config = ConfigDict(
+        validate_default=True,
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    service: Optional[str] = None
+
+    @model_validator(mode='before')
+    def _initial_validation(cls, values):
+        return validation.core.initialize_config(getattr(validators, 'initialize_shared', identity)(values))
+
+    @field_validator('*', mode='before')
+    def _validate(cls, value, info):
+        field = cls.model_fields[info.field_name]
+        field_name = field.alias or info.field_name
+        if field_name in info.context['configured_fields']:
+            value = getattr(validators, f'shared_{info.field_name}', identity)(value, field=field)
+        else:
+            value = getattr(defaults, f'shared_{info.field_name}', lambda: value)()
+
+        return validation.utils.make_immutable(value)
+
+    @model_validator(mode='after')
+    def _final_validation(cls, model):
+        return validation.core.check_model(getattr(validators, 'check_shared', identity)(model))

celerdata/datadog_checks/celerdata/config_models/validators.py

@@ -0,0 +1,9 @@
+# Here you can include additional config validators or transformers
+#
+# def initialize_instance(values, **kwargs):
+#     if 'my_option' not in values and 'my_legacy_option' in values:
+#         values['my_option'] = values['my_legacy_option']
+#     if values.get('my_number') > 10:
+#         raise ValueError('my_number max value is 10, got %s' % str(values.get('my_number')))
+#
+#     return values