feat(cockpit): retention updates (#4778)

Author: nerda-codes

changelog/may2024/2024-05-24-cockpit-added-cockpit-available-in-all-region.mdx

@@ -9,5 +9,5 @@ category: observability
 product: cockpit
 ---
 
-Cockpit is now available in all three Scaleway regions: Paris, Amsterdam, and Warsaw. From the Scaleway console, you can now decide in which region to enable the [alert manager](/cockpit/concepts/#alert-manager) and your [preconfigured alerts](/cockpit/concepts/#preconfigured-alerts). You can also choose the regions in which to create your [data types](/cockpit/concepts/#data-types), [data sources](/cockpit/concepts/#data-sources), and [tokens](/cockpit/concepts/#tokens).
+Cockpit is now available in all three Scaleway regions: Paris, Amsterdam, and Warsaw. From the Scaleway console, you can now decide in which region to enable the [alert manager](/cockpit/concepts/#alert-manager) and your [preconfigured alerts](/cockpit/concepts/#preconfigured-alerts). You can also choose the regions in which to create your [data types](/cockpit/concepts/#data-types), [data sources](/cockpit/concepts/#data-sources), and [tokens](/cockpit/concepts/#cockpit-tokens).
 

macros/cockpit/plan-deprecation.mdx

@@ -2,11 +2,18 @@
 macro: cockpit-plan-deprecation
 ---
 
-<Message type="important">
+<Message type="note">
 
- **Cockpit pricing plans were deprecated on January 1st 2025** <br /><br />
+ **Pricing update for Cockpit custom retention** <br /><br />
 
- The [retention](/cockpit/concepts/#retention) period previously set for your Scaleway and custom logs, metrics and traces remains the same. <br /><br />
- You can change the retention period for your metrics, logs, and traces for free during the retention period edition beta (starting January 1st 2025). <br /><br />
- Find out [how to change data retention period](/cockpit/how-to/change-data-retention/) in the dedicated documentation.
+ On January 1st, 2025, Cockpit pricing plans were deprecated and replaced by the custom [retention](/cockpit/concepts/#retention) feature, available for free during beta.
+ On May 1st, 2025, this feature reaches general availability and **becomes billable**. <br /><br />
+
+ **Logs and traces**: free retention for 7 days, then charged €0.002/GB/day
+
+ **Metrics**: free retention for 31 days, then charged €0.0002/10 million samples/day
+
+ Ingestion of custom data remains billable at [the current pricing](/cockpit/faq/#how-am-i-billed-for-using-cockpit-with-custom-data).
+
+ Refer to the [dedicated documentation](/cockpit/how-to/change-data-retention/) to adjust retention settings.
 </Message>

pages/cockpit/concepts.mdx

@@ -7,15 +7,11 @@ content:
   paragraph: Explore Scaleway Observability concepts including metrics, logs, and alerts management through Cockpit. Learn about agents, alerting rules, Grafana integration, and data types for comprehensive monitoring.
 tags: observability alert-manager contact-points endpoint grafana loki logql prometheus promql tokens
 dates:
-  validation: 2024-12-26
+  validation: 2025-04-03
 categories:
   - observability
 ---
 
-## Active series
-
-Active series refer to [time series](#time-series) for which the latest [samples](#samples) received by your Cockpit are less than 10 minutes old.
-
 ## Agent
 
 An agent is a software component that runs on your systems to gather [data types](#data-types) from the host system or applications running on it. The agent then forwards this data to Cockpit for analysis and visualization.
@@ -28,16 +24,12 @@ An agent is a software component that runs on your systems to gather [data types
 
 Alerting detects complex conditions defined by a rule and keeps you aware of issues in your environments. When a condition defined by a rule is met, the rule tracks it as an alert and responds by triggering one or more actions.
 
-<Message type="important">
- The **Grafana alert manager** on the Grafana interface is inactive. We strongly recommend that you select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
-</Message>
-
 ## Alert manager
 
 Scaleway's regionalized alert manager allows you to manage and respond to alerts according to the regions you have enabled it in. It handles alerts sent when the alerting rules we run are firing. The alert manager triggers alerts (e.g. emails or texts) if a criteria you have configured on your applications' metrics and logs is activated.
 
 <Message type="important">
- The **Grafana alert manager** on the Grafana interface is inactive. We strongly recommend that you select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
+ Scaleway does not support the **Grafana alert manager** on the Grafana interface. You must select the **Scaleway Alerting** alert manager if you want to manage your alerts using Grafana.
 </Message>
 
 ## Alerting rules
@@ -52,10 +44,23 @@ Alerting rules allow you to define criteria that determine whether an alert is t
 
 A Cockpit is an instance of the Observability product that stores metrics, logs, and traces and provides a dedicated dashboarding system on Grafana to visualize them. A Scaleway Project can have only one Cockpit, which is automatically activated when you are using [Scaleway resources that are integrated into Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit).
 
+## Cockpit tokens
+
+Tokens are regionalized secret keys that allow you to authenticate against the endpoints of your Cockpit's [data sources](#data-sources) (metrics, logs, alerts). You can generate new tokens and select token permissions such as:
+
+- **Push**: allows you to send your metrics, logs, and traces to your Cockpit.
+- **Query**: allows you to fetch your metrics, logs, and traces from your Cockpit.
+- **Rules**: allow you to configure alerting and recording rules.
+- **Alerts**: allow you to set up the alert manager.
+
 ## Contact points
 
 Contact points define who is notified when an alert fires, according to the region in which you have added them. Contact points include emails, Slack, on-call systems, and texts. When an alert fires, all contact points are notified.
 
+## Custom data
+
+Custom data is any data (metrics, logs or traces) that you may push to Cockpit. It can be data collected from your applications hosted at Scaleway or elsewhere.
+
 ## Data sources
 
 Data sources are regionalized backends that allow you to store and fetch your metrics, logs, and traces. By default, **Scaleway data sources** are enabled if you are using [Scaleway resources integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) on your Project.
@@ -102,14 +107,6 @@ A Grafana user is any individual who can log in to [Grafana](https://grafana.com
 
 Loki is the log aggregation system used by [Grafana](https://grafana.com/docs/grafana/latest/introduction/) to store and query your logs.
 
-## Loki Remote Write
-
-`Loki Remote Write` is the protocol used to push your logs to your Cockpit's logs' endpoint.
-
-## LogQL
-
-LogQL is [Grafana Loki’s language](https://grafana.com/docs/loki/latest/logql/) for querying logs. LogQL uses labels and operators for filtering.
-
 ## Logs
 
 Logs are a data type that provides a record of all events and errors taking place during the lifecycle of your resources. They represent an excellent source of visibility if you want to know when a problem occurred, or which events correlate with it.
@@ -138,22 +135,10 @@ You can push metrics with any Prometheus-compatible agent such as [Prometheus](h
 
 A preconfigured dashboard is a set of one or more panels that Scaleway sets up and updates for you to visualize the metrics and logs associated with your Scaleway products.
 
-## Prometheus Remote Write
-
-`Prometheus Remote Write` is the protocol used to push your metrics to your Cockpit's metrics' endpoint.
-
-## PromQL
-
-PromQL, short for Prometheus Querying Language, is the main way to query metrics within Prometheus. It is designed for building queries for graphs and alerts.
-
 ## Receivers
 
 Receivers are hubs consisting of contact points. You can associate one or several alerts with one or more receivers. This allows you to diversify your alerts.
 
-## Recording rules
-
-Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their results as a new set of time series.
-
 ## Region
 
 A region is the geographical area in which your Cockpit data is stored. They are duplicated on all Availability Zones of the selected regions (Paris, Amsterdam, or Warsaw).
@@ -162,45 +147,37 @@ You can decide in which region to enable the [alert manager](#alert-manager) and
 
 ## Retention
 
-Retention or data retention refers to the duration for which data, such as metrics, logs, or traces, is stored before being automatically deleted. Retention allows you to manage the lifecycle of your Scaleway and custom data by selecting storage periods that align with your needs.
+Retention or data retention refers to the duration for which the data (metrics, logs, and traces) pushed to Cockpit (by Scaleway products or yourself) is stored before being automatically deleted. Retention allows you to manage the lifecycle of your Scaleway and custom data by selecting storage periods that align with your needs.
 
 The minimum and maximum retention periods for each data source type are as follows:
 
-| Custom metrics                                 | Custom logs/traces                          | Scaleway metrics                               | Scaleway logs/traces                        |
-|------------------------------------------------|---------------------------------------------|------------------------------------------------|---------------------------------------------|
-| Minimum retention period: 1 day                | Minimum retention period: 1 day             | Minimum retention period: 31 days              | Minimum retention period: 1 day             |
-| Maximum retention period: 365 days (12 months) | Maximum retention period: 31 days (1 month) | Maximum retention period: 365 days (12 months) | Maximum retention period: 31 days (1 month) |
-| Default retention period: 31 days              | Default retention period: 7 days            | Default retention period: 31 days              | Default retention period: 7 days            |
+| Retention period  | Custom metrics    | Custom logs/traces | Scaleway metrics  | Scaleway logs        |
+|-------------------|-------------------|--------------------|-------------------|----------------------|
+| Minimum retention | 1 day             | 1 days             | 31 days           | 1 day                |
+| Maximum retention | 365 days (1 year) | 31 days            | 365 days (1 year) | 31 days              |
+| Default retention | 31 days           | 7 days             | 31 days           | 7 days               |
+
+**Starting May 1, 2025, the maximum retention period for custom and Scaleway metrics, logs and traces will be of 1825 days (5 years)**. The information on the table above will be updated accordingly.
 
 ## Samples
 
 A sample is a unique measuring point on a time series.
 
+## Scaleway data
+
+Scaleway data is the observability data (metrics and/or logs) that is natively collected by all Scaleway products, integrated with Cockpit.
+Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) to find out which resources are integrated with Cockpit.
+
 ## Tempo
 
 Tempo is [Grafana's open source](https://grafana.com/docs/tempo/latest/) tracing tool that allows you to search for traces, generate metrics from them, and link your tracing data with logs and metrics. Tempo can be used with open-source tracing protocols such as [Jaeger](https://www.jaegertracing.io/docs/1.50/), [Zipkin](https://zipkin.io/), and [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/).
 
-<Message type="important">
- During the beta of the traces feature, only the [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) HTTP push path is supported. Scaleway is working on implementing the [Open Telemetry gRPC](https://opentelemetry.io/docs/specs/semconv/rpc/grpc/), [Zipkin](https://zipkin.io/) and [Jaeger](https://www.jaegertracing.io/docs/1.50/) protocols.
-</Message>
+Scaleway only supports the OpenTelemetry HTTP push path.
 
 ## Time series
 
 A time series is a sequence of numerical data points in successive order, that tracks the value of a parameter over time. Example of parameter: the disk usage of a machine hosting a database, expressed in percentage.
 
-## Tokens
-
-Tokens are regionalized secret keys that allow you to authenticate against your Cockpit's endpoints (metrics, logs, alerts). You can generate new tokens and select token permissions such as:
-
-- **Push**: allows you to send your metrics, logs, and traces to your Cockpit.
-- **Query**: allows you to fetch your metrics, logs, and traces from your Cockpit.
-- **Rules**: allow you to configure alerting and recording rules.
-- **Alerts**: allow you to set up the alert manager.
-
-## TraceQL
-
-[TraceQL](https://grafana.com/docs/tempo/latest/traceql/) is Grafana Tempo's query language designed for searching and extracting traces.
-
 ## Traces
 
 Traces are detailed records of your **requests' behavior**, as they move through distributed systems such as microservices and containers.

pages/cockpit/faq.mdx

@@ -5,7 +5,7 @@ meta:
 content:
   h1: Cockpit FAQ
 dates:
-  validation: 2024-11-25
+  validation: 2025-03-27
 category: observability
 productIcon: CockpitProductIcon
 ---
@@ -23,27 +23,37 @@ Cockpit integrates seamlessly with Scaleway’s ecosystem. It provides pre-built
 
 Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) for the whole list of Scaleway resources integrated with Cockpit.
 
-## How am I billed for using Cockpit with my Scaleway data?
+## What is the difference between Scaleway and custom data?
+
+[Scaleway data](#scaleway-data) is collected automatically from Scaleway products, while [custom data](#custom-data) is manually pushed from your own sources.
+
+### Scaleway data
+
+Scaleway data is the observability data (metrics and/or logs) that is natively collected by all [Scaleway products integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit).
 
-Using Cockpit with your Scaleway data is free of charge.
+### Custom data
 
-During the beta phase of the retention period edition feature, the retention period you set for your data - whether from [Scaleway resources integrated with Cockpit](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) or custom data - will be free of charge.
+Custom data refers to any metrics, logs, or traces that you manually push to Cockpit. This can include data from applications hosted at Scaleway or any other platform.
 
-Additional costs apply for sending custom data to Cockpit. This means that you will be billed for sending **data from Scaleway resources that are not integrated with Cockpit**, or data from any other external resources.
+## How am I billed for using Cockpit with my Scaleway data?
+
+Scaleway data is collected and available in Cockpit for free. Retention is also free as long as it stays within the default period: 31 days for metrics and 7 days for logs.
+
+You can [adjust the retention period](/cockpit/how-to/change-data-retention) from 1 day (for logs) or 31 days (for metrics) to 1 year. Refer to the [dedicated documentation](/cockpit/concepts/#retention) for more information on available retention periods. However, starting May 1, 2025, data stored beyond the default period will incur charges based on daily storage volume.
 
 ## How am I billed for using Cockpit with custom data?
 
-Custom data is any data that is external to Scaleway, or **data from Scaleway resources not integrated with Cockpit**. Refer to the [dedicated documentation page](/cockpit/reference-content/cockpit-limitations/#product-integration-into-cockpit) to find out which resources are integrated with Cockpit.
+Sending custom data to Cockpit such as data from non-integrated Scaleway resources or external sources incurs additional costs.
 
-During the beta of the retention period edition feature, the retention period you set for your custom data is free of charge.
+Retention of custom data is free within the default period: 31 days for custom metrics and 7 days for custom logs and traces. You can [adjust the retention period](/cockpit/how-to/change-data-retention) from 1 day (for logs and traces) and 31 days (for metrics) to 1 year. However, starting May 1, 2025, data stored beyond the default period will be charged based on daily storage volume.
 
-You will not be charged for **querying** custom data.
+Querying custom data is always free.
 
-Data ingestion is billed as follows:
+Ingestion for custom data is billed as follows:
 
-- Custom metrics are billed €0.15 per million samples of metrics ingested, per month.
-- Custom logs are billed €0.35 per GB ingested, per month.
-- Custom traces are billed €0.35 per GB ingested, per month.
+- Custom metrics are billed €0.15 per million samples of metrics ingested
+- Custom logs are billed €0.35 per GB ingested
+- Custom traces are billed €0.35 per GB ingested
 
 Scaleway applies volume discounts to bill custom metrics. This means that the monthly pricing for custom metrics will be calculated in levels. **Scaleway applies six levels of volume discounts.**
 
@@ -59,14 +69,40 @@ Scaleway applies volume discounts to bill custom metrics. This means that the mo
 
 Here is an example of how you would be billed for **sending 52 billion custom metrics samples per month**:
 
-Here is an example of how you would be billed for sending 52 billion custom metrics samples per month:
 - 10 billion samples at €0.15 per million samples: €1.500
 - 15 billion samples at €0.13 per million samples: €1.950
 - 25 billion samples at €0.12 per million samples: €3.000
 - 2 billion samples at €0.11 per million samples: €220
 
 **Total:** €6.650 per month with volume discounts instead of €7.800 without volume discounts.
 
+## How am I billed for increasing data retention period?
+
+All ingested data, whether from Scaleway or custom sources, is retained for free within the default periods:
+- **Metrics:** 31 days
+- **Logs and traces:** 7 days
+
+**Starting May 1, 2025, data retained beyond these periods will be charged** based on daily storage volume:
+
+- **Metrics:** €0.0002 per 10 million samples/day
+- **Logs and traces:** €0.002 per GB/day
+
+If you delete your data source or reduce its retention period below the default value, data will be deleted and you will no longer be charged for extended retention.
+
+<Concept>
+  ## See custom retention pricing examples
+  ### For 10 days of retention
+  If you ingest an average of **2GB of logs daily** and increase retention to **10 days**. Retention is free for the first **7 days**, so you are only charged for the extra **3 days**:
+
+  **Monthly estimated cost:**
+    `retention_cost` = 2 GB x (10 - 7) x 0.002€ x 30 days = 0.36€/month
+
+  ### For 90 days of retention
+  If you extend retention to **90 days**, you are charged for the extra **83 days** beyond the free period:
+  **Monthly estimated cost:**
+   `retention_cost` = 2 GB x (90 - 7) x 0.002€ x 30 days = 9.96€/month
+</Concept>
+
 ## Why are Cockpit pricing plans being deprecated?
 
 From January 1st 2025, Cockpit is transitioning away from fixed pricing plans to offer you more flexibility and granularity for managing data [retention](/cockpit/concepts/#retention).