From a64de9472b661ba6768cd8e6181b7e0a6967d4bb Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Tue, 2 Dec 2025 17:07:12 +0200 Subject: [PATCH 1/6] Add observability alerts for chargeback integration - Add two ES|QL alerting rules: detect new chargeback groups and detect deployments missing usage data - Add comprehensive documentation for alert setup and configuration - Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support - Add transform startup and monitoring instructions to README --- packages/chargeback/_dev/build/docs/README.md | 176 +++++++++++++++++- packages/chargeback/changelog.yml | 25 ++- packages/chargeback/docs/README.md | 176 +++++++++++++++++- .../billing_cluster_cost/transform.yml | 2 +- .../transform.yml | 2 +- .../transform.yml | 2 +- .../transform.yml | 2 +- .../cluster_tier_contribution/transform.yml | 2 +- packages/chargeback/manifest.yml | 2 +- 9 files changed, 353 insertions(+), 36 deletions(-) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index d4ee6fda934..903984ca406 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -2,7 +2,9 @@ _Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._ -The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier accross the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. +The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier across the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. + +The integration creates several transforms that aggregate billing and usage data into lookup indices optimized for cost analysis and chargeback reporting. ## What is FinOps? @@ -28,9 +30,11 @@ Currently, Chargeback calculations consider only Elasticsearch data nodes. Contr This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment. -Chargeback is also present based on a configured rate and unit. These are used to display cost in the local currency, for instance `EUR`, with a rate of `0.85`. +Chargeback costs are presented based on a configured rate and unit. These are used to display cost in your local currency, for instance `EUR`, with a rate of `0.85` per ECU. + +## Configuration -All configuration values can be updated, as follows: +All configuration values can be updated via the configuration lookup index: ``` POST chargeback_conf_lookup/_update/config @@ -40,22 +44,174 @@ POST chargeback_conf_lookup/_update/config "conf_ecu_rate_unit": "EUR", "conf_indexing_weight": 50, "conf_query_weight": 20, - "conf_storage_weight": 40 + "conf_storage_weight": 40, + "conf_start_date": "2024-01-01T00:00:00.000Z", + "conf_end_date": "2024-12-31T23:59:59.999Z" } } ``` -Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard. +**Configuration Options:** +- `conf_ecu_rate`: The monetary value per ECU (e.g., 0.85) +- `conf_ecu_rate_unit`: The currency code (e.g., "EUR", "USD", "GBP") +- `conf_indexing_weight`: Weight for indexing operations (default: 20, only applies to hot tier) +- `conf_query_weight`: Weight for query operations (default: 20) +- `conf_storage_weight`: Weight for storage (default: 40) +- `conf_start_date`: Start date/time for the configuration period (ISO 8601 format) +- `conf_end_date`: End date/time for the configuration period (ISO 8601 format) + +## Data and Transforms + +The integration creates the following transforms to aggregate cost and usage data: + +1. **billing_cluster_cost** - Aggregates daily ECU usage per deployment from ESS Billing data, with support for deployment groups via `chargeback_group` tags +2. **cluster_deployment_contribution** - Calculates per-deployment usage metrics (indexing time, query time, storage) from Elasticsearch monitoring data +3. **cluster_datastream_contribution** - Aggregates usage per data stream for detailed cost attribution +4. **cluster_tier_contribution** - Aggregates usage per data tier (hot, warm, cold, frozen) +5. **cluster_tier_and_ds_contribution** - Combined view of usage by both tier and data stream + +These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. + +### Starting the Transforms + +After installing the integration, you need to manually start the four usage-related transforms: + +1. Navigate to **Stack Management → Transforms** +2. Filter for `chargeback` to see all Chargeback transforms +3. Start the following transforms: + - `cluster_deployment_contribution` + - `cluster_datastream_contribution` + - `cluster_tier_contribution` + - `cluster_tier_and_ds_contribution` + +The `billing_cluster_cost` transform starts automatically and does not require manual intervention. + +### Transform Health Monitoring + +To set up alerts that notify you when transforms are not working: + +1. Navigate to **Stack Management → Transforms** +2. Filter for `chargeback` to see all Chargeback transforms +3. Select a transform and click the **Actions** menu +4. Select **Create alert rule** +5. Configure the alert rule to notify when the transform health status changes + +This will create a transform health rule that monitors the selected transform and sends notifications when issues are detected. + +## Dashboard + +Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard, which provides: + +- Cost breakdown by deployment, data tier, and data stream +- Time-series cost trends +- Deployment group filtering for team/project-based analysis +- Blended cost metrics combining indexing, querying, and storage usage +- ECU consumption vs. monetary cost comparison ![Cost and Consumption breakdown](../img/chargeback.png) +## Deployment Groups + +The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by teams, projects, or any organizational structure. + +To assign a deployment to a chargeback group, add a tag to your deployment in the Elastic Cloud console in the format: +``` +chargeback_group: +``` + +For example: `chargeback_group:team-search` or `chargeback_group:project-analytics` + +The `billing_cluster_cost` transform automatically extracts these tags from the `deployment_tags` field in ESS Billing data using runtime mappings. The dashboard includes a deployment group filter to view costs by specific groups, making it easy to track expenses per team or project. + +**Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. + +## Observability Alerts + +The following are sample observability alerts that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: + +### Alert 1: New Chargeback Group Detected + +Detects when a new `chargeback_group` tag is added to a deployment, allowing teams to be notified when new cost allocation groups are created. + +**To create this alert**, navigate to **Dev Tools** in Kibana and run: +```json +POST kbn:/api/alerting/rule/chargeback_new_group_detected +{ + "name": "[Chargeback] New chargeback group detected", + "tags": ["Chargeback"], + "consumer": "alerts", + "rule_type_id": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "size": 100, + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group" + }, + "threshold": [0], + "timeField": "@timestamp", + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "thresholdComparator": ">" + }, + "notify_when": "onActionGroupChange", + "actions": [] +} +``` + +### Alert 2: Deployment with Chargeback Group Missing Usage Data + +Alerts when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. + +**To create this alert**, navigate to **Dev Tools** in Kibana and run: +```json +POST kbn:/api/alerting/rule/chargeback_deployment_missing_usage_data +{ + "name": "[Chargeback] Deployment with chargeback group missing usage data", + "tags": ["Chargeback"], + "consumer": "alerts", + "rule_type_id": ".es-query", + "schedule": { + "interval": "6h" + }, + "params": { + "size": 0, + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | WHERE deployment_group != \"\" | LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key | WHERE cluster_name IS NULL | STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group" + }, + "threshold": [0], + "timeField": "@timestamp", + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "thresholdComparator": ">" + }, + "actions": [] +} +``` + ## Requirements To use this integration, the following prerequisites must be met: -- The monitoring cluster, where this integration is installed, must be on version 8.18.0+ due to its use of [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join). -- The [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and running. -- The [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all relevant deployments. -- The Transform named `logs-elasticsearch.index_pivot-default-{VERSION}` must be running, which is an asset of the **Elasticsearch** integration. +**Monitoring Cluster:** +- Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries +- This is where the Chargeback integration should be installed + +**Required Integrations:** +- [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and collecting billing data from your Elastic Cloud organization +- [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all deployments you want to include in chargeback calculations + +**Required Transforms:** +- The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index +- All Chargeback transforms will be automatically created and started when the integration is installed + +**Data Flow:** +1. ESS Billing data is collected into `metrics-ess_billing.billing-*` +2. Elasticsearch usage data is collected into `metrics-elasticsearch.stack_monitoring.*` (or `monitoring-indices` for Stack Monitoring) +3. Chargeback transforms process and correlate this data +4. Dashboard queries the resulting lookup indices using ES|QL -This integration must be installed on the **Monitoring cluster** where the above mentioned relevant usage and billing data is collected. \ No newline at end of file +**Note:** This integration must be installed on a centralized monitoring cluster that has visibility to both billing and usage data from your deployments. \ No newline at end of file diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml index 0ca7285c698..865b3377602 100644 --- a/packages/chargeback/changelog.yml +++ b/packages/chargeback/changelog.yml @@ -1,24 +1,29 @@ # newer versions go on top +- version: 0.2.7 + changes: + - description: "Add observability alerting rule templates and documentation for monitoring new chargeback groups and missing usage data. Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support." + type: enhancement + link: https://github.com/elastic/integrations/pull/11111 - version: 0.2.6 changes: - description: "Fixing bug around sku based cost allocation" - type: enhancement - link: https://github.com/elastic/integrations/pull/14545 + type: bugfix + link: https://github.com/elastic/integrations/pull/16192 - version: 0.2.5 changes: - description: "Add deployment_group field extracted from ESS Billing deployment tags using runtime mappings to enable tag-based cost allocation and filtering. Fix transforms to use correct field type for elasticsearch.cluster.name." type: enhancement - link: https://github.com/elastic/integrations/pull/14545 + link: https://github.com/elastic/integrations/pull/16185 - version: 0.2.4 changes: - description: "Adding sku and cost_type to the billing_cluster_cost_lookup for future utilization" type: enhancement - link: https://github.com/elastic/integrations/pull/14545 + link: https://github.com/elastic/integrations/pull/16182 - version: 0.2.3 changes: - description: "Adding deployment filter, dataview and moving config portion to bottom of dashboard for better usability." type: enhancement - link: https://github.com/elastic/integrations/pull/14545 + link: https://github.com/elastic/integrations/pull/16153 - version: 0.2.2 changes: - description: "Allow setting the Conversion Rate per time window in the configuration lookup index and adding collapsable sections in the dashboard for better usability." @@ -47,27 +52,27 @@ - version: 0.1.5 changes: - description: "Fixing the control error in the dashboard by adding a data view." - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/14545 - version: 0.1.4 changes: - description: "Consistent naming of `datastream`. Add `| LIMIT 5000` to ESQL top query to cater for large organisations." - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/14545 - version: 0.1.3 changes: - description: "Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard." - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/14545 - version: 0.1.2 changes: - description: "Added the necessary fields to the billing_cluster_cost_lookup in the Elasticsearch transform to allow for correlation with the ES integration." - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/14545 - version: 0.1.1 changes: - description: "Fixed the dashboard chargeback timeframe calculation for cost and ECU utilisation" - type: bugfix + type: enhancement link: https://github.com/elastic/integrations/pull/14545 - version: 0.1.0 changes: diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md index d4ee6fda934..903984ca406 100644 --- a/packages/chargeback/docs/README.md +++ b/packages/chargeback/docs/README.md @@ -2,7 +2,9 @@ _Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._ -The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier accross the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. +The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier across the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. + +The integration creates several transforms that aggregate billing and usage data into lookup indices optimized for cost analysis and chargeback reporting. ## What is FinOps? @@ -28,9 +30,11 @@ Currently, Chargeback calculations consider only Elasticsearch data nodes. Contr This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment. -Chargeback is also present based on a configured rate and unit. These are used to display cost in the local currency, for instance `EUR`, with a rate of `0.85`. +Chargeback costs are presented based on a configured rate and unit. These are used to display cost in your local currency, for instance `EUR`, with a rate of `0.85` per ECU. + +## Configuration -All configuration values can be updated, as follows: +All configuration values can be updated via the configuration lookup index: ``` POST chargeback_conf_lookup/_update/config @@ -40,22 +44,174 @@ POST chargeback_conf_lookup/_update/config "conf_ecu_rate_unit": "EUR", "conf_indexing_weight": 50, "conf_query_weight": 20, - "conf_storage_weight": 40 + "conf_storage_weight": 40, + "conf_start_date": "2024-01-01T00:00:00.000Z", + "conf_end_date": "2024-12-31T23:59:59.999Z" } } ``` -Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard. +**Configuration Options:** +- `conf_ecu_rate`: The monetary value per ECU (e.g., 0.85) +- `conf_ecu_rate_unit`: The currency code (e.g., "EUR", "USD", "GBP") +- `conf_indexing_weight`: Weight for indexing operations (default: 20, only applies to hot tier) +- `conf_query_weight`: Weight for query operations (default: 20) +- `conf_storage_weight`: Weight for storage (default: 40) +- `conf_start_date`: Start date/time for the configuration period (ISO 8601 format) +- `conf_end_date`: End date/time for the configuration period (ISO 8601 format) + +## Data and Transforms + +The integration creates the following transforms to aggregate cost and usage data: + +1. **billing_cluster_cost** - Aggregates daily ECU usage per deployment from ESS Billing data, with support for deployment groups via `chargeback_group` tags +2. **cluster_deployment_contribution** - Calculates per-deployment usage metrics (indexing time, query time, storage) from Elasticsearch monitoring data +3. **cluster_datastream_contribution** - Aggregates usage per data stream for detailed cost attribution +4. **cluster_tier_contribution** - Aggregates usage per data tier (hot, warm, cold, frozen) +5. **cluster_tier_and_ds_contribution** - Combined view of usage by both tier and data stream + +These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. + +### Starting the Transforms + +After installing the integration, you need to manually start the four usage-related transforms: + +1. Navigate to **Stack Management → Transforms** +2. Filter for `chargeback` to see all Chargeback transforms +3. Start the following transforms: + - `cluster_deployment_contribution` + - `cluster_datastream_contribution` + - `cluster_tier_contribution` + - `cluster_tier_and_ds_contribution` + +The `billing_cluster_cost` transform starts automatically and does not require manual intervention. + +### Transform Health Monitoring + +To set up alerts that notify you when transforms are not working: + +1. Navigate to **Stack Management → Transforms** +2. Filter for `chargeback` to see all Chargeback transforms +3. Select a transform and click the **Actions** menu +4. Select **Create alert rule** +5. Configure the alert rule to notify when the transform health status changes + +This will create a transform health rule that monitors the selected transform and sends notifications when issues are detected. + +## Dashboard + +Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard, which provides: + +- Cost breakdown by deployment, data tier, and data stream +- Time-series cost trends +- Deployment group filtering for team/project-based analysis +- Blended cost metrics combining indexing, querying, and storage usage +- ECU consumption vs. monetary cost comparison ![Cost and Consumption breakdown](../img/chargeback.png) +## Deployment Groups + +The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by teams, projects, or any organizational structure. + +To assign a deployment to a chargeback group, add a tag to your deployment in the Elastic Cloud console in the format: +``` +chargeback_group: +``` + +For example: `chargeback_group:team-search` or `chargeback_group:project-analytics` + +The `billing_cluster_cost` transform automatically extracts these tags from the `deployment_tags` field in ESS Billing data using runtime mappings. The dashboard includes a deployment group filter to view costs by specific groups, making it easy to track expenses per team or project. + +**Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. + +## Observability Alerts + +The following are sample observability alerts that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: + +### Alert 1: New Chargeback Group Detected + +Detects when a new `chargeback_group` tag is added to a deployment, allowing teams to be notified when new cost allocation groups are created. + +**To create this alert**, navigate to **Dev Tools** in Kibana and run: +```json +POST kbn:/api/alerting/rule/chargeback_new_group_detected +{ + "name": "[Chargeback] New chargeback group detected", + "tags": ["Chargeback"], + "consumer": "alerts", + "rule_type_id": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "size": 100, + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group" + }, + "threshold": [0], + "timeField": "@timestamp", + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "thresholdComparator": ">" + }, + "notify_when": "onActionGroupChange", + "actions": [] +} +``` + +### Alert 2: Deployment with Chargeback Group Missing Usage Data + +Alerts when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. + +**To create this alert**, navigate to **Dev Tools** in Kibana and run: +```json +POST kbn:/api/alerting/rule/chargeback_deployment_missing_usage_data +{ + "name": "[Chargeback] Deployment with chargeback group missing usage data", + "tags": ["Chargeback"], + "consumer": "alerts", + "rule_type_id": ".es-query", + "schedule": { + "interval": "6h" + }, + "params": { + "size": 0, + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | WHERE deployment_group != \"\" | LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key | WHERE cluster_name IS NULL | STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group" + }, + "threshold": [0], + "timeField": "@timestamp", + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "thresholdComparator": ">" + }, + "actions": [] +} +``` + ## Requirements To use this integration, the following prerequisites must be met: -- The monitoring cluster, where this integration is installed, must be on version 8.18.0+ due to its use of [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join). -- The [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and running. -- The [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all relevant deployments. -- The Transform named `logs-elasticsearch.index_pivot-default-{VERSION}` must be running, which is an asset of the **Elasticsearch** integration. +**Monitoring Cluster:** +- Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries +- This is where the Chargeback integration should be installed + +**Required Integrations:** +- [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and collecting billing data from your Elastic Cloud organization +- [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all deployments you want to include in chargeback calculations + +**Required Transforms:** +- The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index +- All Chargeback transforms will be automatically created and started when the integration is installed + +**Data Flow:** +1. ESS Billing data is collected into `metrics-ess_billing.billing-*` +2. Elasticsearch usage data is collected into `metrics-elasticsearch.stack_monitoring.*` (or `monitoring-indices` for Stack Monitoring) +3. Chargeback transforms process and correlate this data +4. Dashboard queries the resulting lookup indices using ES|QL -This integration must be installed on the **Monitoring cluster** where the above mentioned relevant usage and billing data is collected. \ No newline at end of file +**Note:** This integration must be installed on a centralized monitoring cluster that has visibility to both billing and usage data from your deployments. \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml index 3dd03b1bd74..3bc79b887ba 100644 --- a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml @@ -22,7 +22,7 @@ source: emit(''); dest: index: billing_cluster_cost_lookup - pipeline: 0.2.6-billing + pipeline: 0.2.7-billing frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml index cffb1e19b68..2c8d3270dd0 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_datastream_contribution_lookup - pipeline: 0.2.6-usage + pipeline: 0.2.7-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml index c2b5bfcc051..3dd5cf97cb1 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_deployment_contribution_lookup - pipeline: 0.2.6-usage + pipeline: 0.2.7-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml index 42cc3927e0b..77881d59e97 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_tier_and_datastream_contribution_lookup - pipeline: 0.2.6-usage + pipeline: 0.2.7-usage frequency: 60m sync: time: diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml index 99b14d3b52d..61d102dd35b 100644 --- a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml @@ -4,7 +4,7 @@ source: - monitoring-indices # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. dest: index: cluster_tier_contribution_lookup - pipeline: 0.2.6-usage + pipeline: 0.2.7-usage frequency: 60m sync: time: diff --git a/packages/chargeback/manifest.yml b/packages/chargeback/manifest.yml index ce23f0fa9a2..ff7270d44fc 100644 --- a/packages/chargeback/manifest.yml +++ b/packages/chargeback/manifest.yml @@ -1,7 +1,7 @@ format_version: 3.4.0 name: chargeback title: "Chargeback" -version: 0.2.6 +version: 0.2.7 description: "This package calculates chargeback based on billing and consumption data" type: integration categories: From f4f555bbb27f8ab2e57b4ab90bc5f26e87637c9a Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Tue, 2 Dec 2025 17:11:02 +0200 Subject: [PATCH 2/6] Update changelog with PR #16205 --- packages/chargeback/changelog.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml index 865b3377602..bb12d8af9df 100644 --- a/packages/chargeback/changelog.yml +++ b/packages/chargeback/changelog.yml @@ -3,7 +3,7 @@ changes: - description: "Add observability alerting rule templates and documentation for monitoring new chargeback groups and missing usage data. Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support." type: enhancement - link: https://github.com/elastic/integrations/pull/11111 + link: https://github.com/elastic/integrations/pull/16205 - version: 0.2.6 changes: - description: "Fixing bug around sku based cost allocation" From 162e8cca0d787e218cb5773259339e339eb1f881 Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Tue, 2 Dec 2025 17:19:44 +0200 Subject: [PATCH 3/6] Remove wrong information --- packages/chargeback/_dev/build/docs/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index 903984ca406..a3d34144fcb 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -206,7 +206,6 @@ To use this integration, the following prerequisites must be met: **Required Transforms:** - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index -- All Chargeback transforms will be automatically created and started when the integration is installed **Data Flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*` From dff89bde0040493543791be2f460b1efbb2f84b6 Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Tue, 2 Dec 2025 17:20:48 +0200 Subject: [PATCH 4/6] Update chargeback README documentation --- packages/chargeback/docs/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md index 903984ca406..a3d34144fcb 100644 --- a/packages/chargeback/docs/README.md +++ b/packages/chargeback/docs/README.md @@ -206,7 +206,6 @@ To use this integration, the following prerequisites must be met: **Required Transforms:** - The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index -- All Chargeback transforms will be automatically created and started when the integration is installed **Data Flow:** 1. ESS Billing data is collected into `metrics-ess_billing.billing-*` From efaa7fc71bb0394d0a150eddb3f28ee5bdcc216c Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Wed, 3 Dec 2025 10:04:35 +0200 Subject: [PATCH 5/6] Improve observability alert action message formatting --- packages/chargeback/_dev/build/docs/README.md | 47 ++++++++++++++----- 1 file changed, 35 insertions(+), 12 deletions(-) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index a3d34144fcb..45675e1f950 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -125,11 +125,11 @@ The `billing_cluster_cost` transform automatically extracts these tags from the **Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. -## Observability Alerts +## Observability Rules -The following are sample observability alerts that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: +The following are sample observability rules that can help ensure data validity by notifying you when events occur that could compromise the accuracy of your chargeback data: -### Alert 1: New Chargeback Group Detected +### Rule 1: New Chargeback Group Detected Detects when a new `chargeback_group` tag is added to a deployment, allowing teams to be notified when new cost allocation groups are created. @@ -147,23 +147,23 @@ POST kbn:/api/alerting/rule/chargeback_new_group_detected "params": { "size": 100, "esqlQuery": { - "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group" + "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group | KEEP deployment_group" }, "threshold": [0], "timeField": "@timestamp", "searchType": "esqlQuery", "timeWindowSize": 3, "timeWindowUnit": "d", - "thresholdComparator": ">" + "thresholdComparator": ">", + "excludeHitsFromPreviousRun": true }, - "notify_when": "onActionGroupChange", "actions": [] } ``` -### Alert 2: Deployment with Chargeback Group Missing Usage Data +### Rule 2: Deployment with Chargeback Group Missing Usage Data -Alerts when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. +Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data. This indicates a potential configuration issue or data collection problem. **To create this alert**, navigate to **Dev Tools** in Kibana and run: ```json @@ -174,24 +174,47 @@ POST kbn:/api/alerting/rule/chargeback_deployment_missing_usage_data "consumer": "alerts", "rule_type_id": ".es-query", "schedule": { - "interval": "6h" + "interval": "1h" }, "params": { - "size": 0, + "size": 100, "esqlQuery": { - "esql": "FROM billing_cluster_cost_lookup | WHERE deployment_group != \"\" | LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key | WHERE cluster_name IS NULL | STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group" + "esql": """FROM billing_cluster_cost_lookup +| WHERE deployment_group != "" +| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key +| WHERE cluster_name IS NULL +| INLINE STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group +| EVAL result = CONCAT("Deployment `", deployment_name,"` (`", deployment_id,"`) in deployment group `", deployment_group, "` did not have usage data since ", left(composite_key,10),".") +| STATS result = VALUES(result) +| MV_EXPAND result""" }, "threshold": [0], "timeField": "@timestamp", "searchType": "esqlQuery", "timeWindowSize": 3, "timeWindowUnit": "d", - "thresholdComparator": ">" + "thresholdComparator": ">", + "excludeHitsFromPreviousRun": true }, "actions": [] } ``` +### Alert actions + +**Configure an action** with the following message template appended to the default content (keep the new lines, as it helps with legibility): + +``` +Details: + +{{#context.hits}} +• {{_source}} + +{{/context.hits}} + +Total: {{context.hits.length}} +``` + ## Requirements To use this integration, the following prerequisites must be met: From 9262914f2d2f12b33ed80ceaae32d37985b00a92 Mon Sep 17 00:00:00 2001 From: Johannes Mahne Date: Wed, 3 Dec 2025 11:11:16 +0200 Subject: [PATCH 6/6] Clarify configuration update vs add new period documentation --- packages/chargeback/_dev/build/docs/README.md | 29 +++++++++++++++++-- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md index 45675e1f950..f66fb208b4e 100755 --- a/packages/chargeback/_dev/build/docs/README.md +++ b/packages/chargeback/_dev/build/docs/README.md @@ -34,7 +34,11 @@ Chargeback costs are presented based on a configured rate and unit. These are us ## Configuration -All configuration values can be updated via the configuration lookup index: +Configuration values are stored in the `chargeback_conf_lookup` index. The dashboard automatically applies the correct configuration based on the billing date falling within the `conf_start_date` and `conf_end_date` range. + +### Update the default configuration: + +Using `_update/config` updates the document with ID `config`: ``` POST chargeback_conf_lookup/_update/config @@ -42,15 +46,34 @@ POST chargeback_conf_lookup/_update/config "doc": { "conf_ecu_rate": 0.85, "conf_ecu_rate_unit": "EUR", - "conf_indexing_weight": 50, + "conf_indexing_weight": 20, "conf_query_weight": 20, "conf_storage_weight": 40, "conf_start_date": "2024-01-01T00:00:00.000Z", - "conf_end_date": "2024-12-31T23:59:59.999Z" + "conf_end_date": "2024-12-31T23:tie" } } ``` +### Add a new configuration period (for time-based rate changes): + +Using `_doc` creates a new document with an auto-generated ID: + +``` +POST chargeback_conf_lookup/_doc +{ + "conf_ecu_rate": 0.95, + "conf_ecu_rate_unit": "EUR", + "conf_indexing_weight": 20, + "conf_query_weight": 20, + "conf_storage_weight": 40, + "conf_start_date": "2025-01-01T00:00:00.000Z", + "conf_end_date": "2025-12-31T23:59:59.999Z" +} +``` + +This allows you to have different rates for different time periods (e.g., quarterly or annual rate changes). + **Configuration Options:** - `conf_ecu_rate`: The monetary value per ECU (e.g., 0.85) - `conf_ecu_rate_unit`: The currency code (e.g., "EUR", "USD", "GBP")