Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: move federated rule groups documentation to its own section #1906

Merged
merged 2 commits into from
May 30, 2022
Merged

docs: move federated rule groups documentation to its own section #1906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
55eb13c
docs: move federated rule groups documentation to its own section
ortuman May 20, 2022
3e44647
addressed PR feedback
ortuman May 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/sources/operators-guide/architecture/components/ruler/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,47 @@ The ruler then notifies Alertmanagers of any **FIRING** (`firing`) alerts.
Configure the addresses of Alertmanagers with the `-ruler.alertmanager-url` flag, which supports the DNS service discovery format.
For more information about DNS service discovery, refer to [Supported discovery modes]({{< relref "../../../configuring/about-dns-service-discovery.md" >}}).

## Federated rule groups

A federated rule group is a rule group with a non-empty `source_tenants`.

The `source_tenants` field allows aggregating data from multiple tenants while evaluating a rule group. The expressions
of each rule in the group will be evaluated against the data of all tenants in `source_tenants`. If `source_tenants` is
empty or omitted, then the tenant under which the group is created will be treated as the `source_tenant`.

Below is an example of how a federated rule group would look like:

```yaml
name: MyGroupName
source_tenants: ["tenant-a", "tenant-b"]
rules:
- record: sum:metric
expr: sum(metric)
```

_In this example `MyGroupName` rules will be evaluated against `tenant-a` and `tenant-b` tenants._

Federated rule groups are skipped during evaluation by default. This feature depends on
the cross-tenant query federation feature. To enable federated rules
set `-ruler.tenant-federation.enabled=true` and `-tenant-federation.enabled=true` CLI flags (or their respective YAML
config options).

During evaluation query limits applied to single tenants are also applied to each query in the rule group. For example,
if `tenant-a` has a federated rule group with `source_tenants: [tenant-b, tenant-c]`, then query limits for `tenant-b`
and `tenant-c` will be applied. If any of these limits is exceeded, the whole evaluation will fail. No partial results
will be saved. The same "no partial results" guarantee applies to queries failing for other reasons (e.g. ingester
unavailability).

The time series used during evaluation of federated rules will have the `__tenant_id__` label, similar to how it is
present on series returned with cross-tenant query federation.

> **Note**: Federated rule groups allow data from multiple source tenants to be written into a single
> destination tenant. This makes the existing separation of tenants' data less clear. For example, `tenant-a` has a
> federated rule group that aggregates over `tenant-b`'s data (e.g. `sum(metric_b)`) and writes the result back
> into `tenant-a`'s storage (e.g. as metric `sum:metric_b`). Now part of `tenant-b`'s data is copied to `tenant-a` (albeit
> aggregated). Have this in mind when configuring the access control layer in front of mimir and when enabling federated
> rules via `-ruler.tenant-federation.enabled`.

## Sharding

The ruler supports multi-tenancy and horizontal scalability.
Expand Down
48 changes: 0 additions & 48 deletions docs/sources/operators-guide/reference-http-api/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -705,54 +705,6 @@ rules:
severity: warning
```

#### Federated rule groups

A federated rule groups is a rule group with a non-empty `source_tenants`.

The `source_tenants` field allows aggregating data from multiple tenants while evaluating a rule group. The expressions
of each rule in the group will be evaluated against the data of all tenants in `source_tenants`. If `source_tenants` is
empty or omitted, then the tenant under which the group is created will be treated as the `source_tenant`.

Federated rule groups are skipped during evaluation by default. This feature depends on
the cross-tenant query federation feature. To enable federated rules
set `-ruler.tenant-federation.enabled=true` and `-tenant-federation.enabled=true` CLI flags (or their respective YAML
config options).

During evaluation query limits applied to single tenants are also applied to each query in the rule group. For example,
if `tenant-a` has a federated rule group with `source_tenants: [tenant-b, tenant-c]`, then query limits for `tenant-b`
and `tenant-c` will be applied. If any of these limits is exceeded, the whole evaluation will fail. No partial results
will be saved. The same "no partial results" guarantee applies to queries failing for other reasons (e.g. ingester
unavailability).

The time series used during evaluation of federated rules will have the `__tenant_id__` label, similar to how it is
present on series returned with cross-tenant query federation.

**Considerations:** Federated rule groups allow data from multiple source tenants to be written into a single
destination tenant. This makes the existing separation of tenants' data less clear. For example, `tenant-a` has a
federated rule group that aggregates over `tenant-b`'s data (e.g. `sum(metric_b)`) and writes the result back
into `tenant-a`'s storage (e.g. as metric `sum:metric_b`). Now part of `tenant-b`'s data is copied to `tenant-a` (albeit
aggregated). Have this in mind when configuring the access control layer in front of mimir and when enabling federated
rules via `-ruler.tenant-federation.enabled`.

#### Example "federated rules group" request body

```yaml
name: <string>
interval: <duration;optional>
source_tenants:
ortuman marked this conversation as resolved.
Show resolved Hide resolved
- <string>
rules:
- record: <string>
expr: <string>
- alert: <string>
expr: <string>
for: <duration>
annotations:
<annotation_name>: <string>
labels:
<label_name>: <string>
```

### Delete rule group

```
Expand Down
2 changes: 1 addition & 1 deletion docs/sources/release-notes/v2.0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ These features and enhancements distinguish Grafana Mimir from Cortex 1.10.0:

- **Query sharding for improved query speeds**: Grafana Mimir introduces query sharding to accelerate the execution of high-cardinality or CPU-intensive queries. Query sharding distributes the execution of a single query across multiple machines, to significantly reduce query execution time. We have seen speedups of 10 to 30x in our Grafana Cloud Metrics clusters. Refer to [query sharding]({{< relref "../operators-guide/architecture/query-sharding/index.md" >}}) to learn more.

- **Federated rule groups**: Grafana Mimir makes it possible to write alerting and recording rules that use metrics data from multiple tenants. For example, a user can now create a recording rule that adds metricA from tenantA to metricB from tenantB and writes the result to tenantC. This feature is experimental. For more information on how to use it, refer to [federated rule groups]({{< relref "../operators-guide/reference-http-api/index.md#federated-rule-groups" >}}).
- **Federated rule groups**: Grafana Mimir makes it possible to write alerting and recording rules that use metrics data from multiple tenants. For example, a user can now create a recording rule that adds metricA from tenantA to metricB from tenantB and writes the result to tenantC. This feature is experimental. For more information on how to use it, refer to [federated rule groups]({{< relref "../operators-guide/architecture/components/ruler/index.md#federated-rule-groups" >}}).

- **Understand your metrics cardinality**: Grafana Mimir adds two API endpoints to help users identify high-cardinality metrics. The [`label_names` endpoint]({{< relref "../operators-guide/reference-http-api/index.md#label-names-cardinality" >}}) takes a metric name and returns all label names applied to that metric, as well as the count of values for each label name. When run without a metric name, it returns the highest-cardinality label names. The [`label_values` endpoint]({{< relref "../operators-guide/reference-http-api/index.md#label-values-cardinality" >}}) returns the highest-cardinality metrics, and can be used to get a count of how many series have a given label-value pair applied. The new [custom tracker feature]({{< relref "../operators-guide/configuring/configuring-custom-trackers.md" >}}) takes this one step further by allowing you to track the count of active series over time that match a specific label matcher.

Expand Down