Skip to content

Commit

Permalink
remove the forwarding feature (#4876)
Browse files Browse the repository at this point in the history 
* remove the forwarding feature

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove forwardig tests

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update alerts and dashboards

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* make lint

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update mixin

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* build-helm-tests

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update docs, 2nd attempt

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove forwarding integration test

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update changelog

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* fix changelog position

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* remove deprecated limits config

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

* update mimirtool config convert to not support forwarding rules

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>

---------

Signed-off-by: Mauro Stettler <mauro.stettler@gmail.com>
  • Loading branch information
@replay
replay committed May 2, 2023
1 parent e5bf3b8 commit ce87556
Show file tree
Hide file tree
Showing 23 changed files with 24 additions and 3,555 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
### Grafana Mimir

* [CHANGE] Store-gateway: change expanded postings and postings index cache key format. These caches will be invalidated when rolling out the new Mimir version. #4770
* [CHANGE] Distributor: remove the "forwarding" feature as it isn't necessary anymore. #4876
* [ENHANCEMENT] Add per-tenant limit `-validation.max-native-histogram-buckets` to be able to ignore native histogram samples that have too many buckets. #4765
* [ENHANCEMENT] Store-gateway: reduce memory usage in some LabelValues calls. #4789
* [ENHANCEMENT] Store-gateway: add a `stage` label to the metric `cortex_bucket_store_series_data_touched`. This label now applies to `data_type="chunks"` and `data_type="series"`. The `stage` label has 2 values: `processed` - the number of series that parsed - and `returned` - the number of series selected from the processed bytes to satisfy the query. #4797 #4830
Expand Down
288 changes: 0 additions & 288 deletions cmd/mimir/config-descriptor.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1445,267 +1445,6 @@
],
"fieldValue": null,
"fieldDefaultValue": null
},
{
"kind": "block",
"name": "forwarding",
"required": false,
"desc": "",
"blockEntries": [
{
"kind": "field",
"name": "enabled",
"required": false,
"desc": "Enables the feature to forward certain metrics in remote_write requests, depending on defined rules.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "distributor.forwarding.enabled",
"fieldType": "boolean",
"fieldCategory": "experimental"
},
{
"kind": "field",
"name": "request_concurrency",
"required": false,
"desc": "Maximum concurrency at which forwarding requests get performed.",
"fieldValue": null,
"fieldDefaultValue": 10,
"fieldFlag": "distributor.forwarding.request-concurrency",
"fieldType": "int",
"fieldCategory": "experimental"
},
{
"kind": "field",
"name": "request_timeout",
"required": false,
"desc": "Timeout for requests to ingestion endpoints to which we forward metrics.",
"fieldValue": null,
"fieldDefaultValue": 2000000000,
"fieldFlag": "distributor.forwarding.request-timeout",
"fieldType": "duration",
"fieldCategory": "experimental"
},
{
"kind": "field",
"name": "propagate_errors",
"required": false,
"desc": "If disabled then forwarding requests are always considered to be successful, errors are ignored.",
"fieldValue": null,
"fieldDefaultValue": true,
"fieldFlag": "distributor.forwarding.propagate-errors",
"fieldType": "boolean",
"fieldCategory": "experimental"
},
{
"kind": "block",
"name": "grpc_client",
"required": false,
"desc": "",
"blockEntries": [
{
"kind": "field",
"name": "max_recv_msg_size",
"required": false,
"desc": "gRPC client max receive message size (bytes).",
"fieldValue": null,
"fieldDefaultValue": 104857600,
"fieldFlag": "distributor.forwarding.grpc-client.grpc-max-recv-msg-size",
"fieldType": "int",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "max_send_msg_size",
"required": false,
"desc": "gRPC client max send message size (bytes).",
"fieldValue": null,
"fieldDefaultValue": 104857600,
"fieldFlag": "distributor.forwarding.grpc-client.grpc-max-send-msg-size",
"fieldType": "int",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "grpc_compression",
"required": false,
"desc": "Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression)",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.grpc-compression",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "rate_limit",
"required": false,
"desc": "Rate limit for gRPC client; 0 means disabled.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldFlag": "distributor.forwarding.grpc-client.grpc-client-rate-limit",
"fieldType": "float",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "rate_limit_burst",
"required": false,
"desc": "Rate limit burst for gRPC client.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldFlag": "distributor.forwarding.grpc-client.grpc-client-rate-limit-burst",
"fieldType": "int",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "backoff_on_ratelimits",
"required": false,
"desc": "Enable backoff and retry when we hit ratelimits.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "distributor.forwarding.grpc-client.backoff-on-ratelimits",
"fieldType": "boolean",
"fieldCategory": "advanced"
},
{
"kind": "block",
"name": "backoff_config",
"required": false,
"desc": "",
"blockEntries": [
{
"kind": "field",
"name": "min_period",
"required": false,
"desc": "Minimum delay when backing off.",
"fieldValue": null,
"fieldDefaultValue": 100000000,
"fieldFlag": "distributor.forwarding.grpc-client.backoff-min-period",
"fieldType": "duration",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "max_period",
"required": false,
"desc": "Maximum delay when backing off.",
"fieldValue": null,
"fieldDefaultValue": 10000000000,
"fieldFlag": "distributor.forwarding.grpc-client.backoff-max-period",
"fieldType": "duration",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "max_retries",
"required": false,
"desc": "Number of times to backoff and retry before failing.",
"fieldValue": null,
"fieldDefaultValue": 10,
"fieldFlag": "distributor.forwarding.grpc-client.backoff-retries",
"fieldType": "int",
"fieldCategory": "advanced"
}
],
"fieldValue": null,
"fieldDefaultValue": null
},
{
"kind": "field",
"name": "tls_enabled",
"required": false,
"desc": "Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "distributor.forwarding.grpc-client.tls-enabled",
"fieldType": "boolean",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_cert_path",
"required": false,
"desc": "Path to the client certificate, which will be used for authenticating with the server. Also requires the key path to be configured.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-cert-path",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_key_path",
"required": false,
"desc": "Path to the key for the client certificate. Also requires the client certificate to be configured.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-key-path",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_ca_path",
"required": false,
"desc": "Path to the CA certificates to validate server certificate against. If not set, the host's root CA certificates are used.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-ca-path",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_server_name",
"required": false,
"desc": "Override the expected name on the server certificate.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-server-name",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_insecure_skip_verify",
"required": false,
"desc": "Skip validating server certificate.",
"fieldValue": null,
"fieldDefaultValue": false,
"fieldFlag": "distributor.forwarding.grpc-client.tls-insecure-skip-verify",
"fieldType": "boolean",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_cipher_suites",
"required": false,
"desc": "Override the default cipher suite list (separated by commas). Allowed values:\n\nSecure Ciphers:\n- TLS_RSA_WITH_AES_128_CBC_SHA\n- TLS_RSA_WITH_AES_256_CBC_SHA\n- TLS_RSA_WITH_AES_128_GCM_SHA256\n- TLS_RSA_WITH_AES_256_GCM_SHA384\n- TLS_AES_128_GCM_SHA256\n- TLS_AES_256_GCM_SHA384\n- TLS_CHACHA20_POLY1305_SHA256\n- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA\n- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA\n- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA\n- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA\n- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\n- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\n- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\n- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\n- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256\n- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256\n\nInsecure Ciphers:\n- TLS_RSA_WITH_RC4_128_SHA\n- TLS_RSA_WITH_3DES_EDE_CBC_SHA\n- TLS_RSA_WITH_AES_128_CBC_SHA256\n- TLS_ECDHE_ECDSA_WITH_RC4_128_SHA\n- TLS_ECDHE_RSA_WITH_RC4_128_SHA\n- TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA\n- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\n- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\n",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-cipher-suites",
"fieldType": "string",
"fieldCategory": "advanced"
},
{
"kind": "field",
"name": "tls_min_version",
"required": false,
"desc": "Override the default minimum TLS version. Allowed values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldFlag": "distributor.forwarding.grpc-client.tls-min-version",
"fieldType": "string",
"fieldCategory": "advanced"
}
],
"fieldValue": null,
"fieldDefaultValue": null
}
],
"fieldValue": null,
"fieldDefaultValue": null
}
],
"fieldValue": null,
Expand Down Expand Up @@ -3634,33 +3373,6 @@
"fieldDefaultValue": 0,
"fieldFlag": "alertmanager.max-alerts-size-bytes",
"fieldType": "int"
},
{
"kind": "field",
"name": "forwarding_endpoint",
"required": false,
"desc": "Remote-write endpoint where metrics specified in forwarding_rules are forwarded to. If set, takes precedence over endpoints specified in forwarding rules.",
"fieldValue": null,
"fieldDefaultValue": "",
"fieldType": "string"
},
{
"kind": "field",
"name": "forwarding_drop_older_than",
"required": false,
"desc": "If set, forwarding drops samples that are older than this duration. If unset or 0, no samples get dropped.",
"fieldValue": null,
"fieldDefaultValue": 0,
"fieldType": "int"
},
{
"kind": "field",
"name": "forwarding_rules",
"required": false,
"desc": "Rules based on which the Distributor decides whether a metric should be forwarded to an alternative remote_write API endpoint.",
"fieldValue": null,
"fieldDefaultValue": {},
"fieldType": "map of string to validation.ForwardingRule"
}
],
"fieldValue": null,
Expand Down
42 changes: 0 additions & 42 deletions cmd/mimir/help-all.txt.tmpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -991,48 +991,6 @@ Usage of ./cmd/mimir/mimir:
How frequently to clean up clients for ingesters that have gone away. (default 15s)
-distributor.drop-label string
This flag can be used to specify label names that to drop during sample ingestion within the distributor and can be repeated in order to drop multiple labels.
-distributor.forwarding.enabled
[experimental] Enables the feature to forward certain metrics in remote_write requests, depending on defined rules.
-distributor.forwarding.grpc-client.backoff-max-period duration
Maximum delay when backing off. (default 10s)
-distributor.forwarding.grpc-client.backoff-min-period duration
Minimum delay when backing off. (default 100ms)
-distributor.forwarding.grpc-client.backoff-on-ratelimits
Enable backoff and retry when we hit ratelimits.
-distributor.forwarding.grpc-client.backoff-retries int
Number of times to backoff and retry before failing. (default 10)
-distributor.forwarding.grpc-client.grpc-client-rate-limit float
Rate limit for gRPC client; 0 means disabled.
-distributor.forwarding.grpc-client.grpc-client-rate-limit-burst int
Rate limit burst for gRPC client.
-distributor.forwarding.grpc-client.grpc-compression string
Use compression when sending messages. Supported values are: 'gzip', 'snappy' and '' (disable compression)
-distributor.forwarding.grpc-client.grpc-max-recv-msg-size int
gRPC client max receive message size (bytes). (default 104857600)
-distributor.forwarding.grpc-client.grpc-max-send-msg-size int
gRPC client max send message size (bytes). (default 104857600)
-distributor.forwarding.grpc-client.tls-ca-path string
Path to the CA certificates to validate server certificate against. If not set, the host's root CA certificates are used.
-distributor.forwarding.grpc-client.tls-cert-path string
Path to the client certificate, which will be used for authenticating with the server. Also requires the key path to be configured.
-distributor.forwarding.grpc-client.tls-cipher-suites string
Override the default cipher suite list (separated by commas).
-distributor.forwarding.grpc-client.tls-enabled
Enable TLS in the GRPC client. This flag needs to be enabled when any other TLS flag is set. If set to false, insecure connection to gRPC server will be used.
-distributor.forwarding.grpc-client.tls-insecure-skip-verify
Skip validating server certificate.
-distributor.forwarding.grpc-client.tls-key-path string
Path to the key for the client certificate. Also requires the client certificate to be configured.
-distributor.forwarding.grpc-client.tls-min-version string
Override the default minimum TLS version. Allowed values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
-distributor.forwarding.grpc-client.tls-server-name string
Override the expected name on the server certificate.
-distributor.forwarding.propagate-errors
[experimental] If disabled then forwarding requests are always considered to be successful, errors are ignored. (default true)
-distributor.forwarding.request-concurrency int
[experimental] Maximum concurrency at which forwarding requests get performed. (default 10)
-distributor.forwarding.request-timeout duration
[experimental] Timeout for requests to ingestion endpoints to which we forward metrics. (default 2s)
-distributor.ha-tracker.cluster string
Prometheus label to look for in samples to identify a Prometheus HA cluster. (default "cluster")
-distributor.ha-tracker.consul.acl-token string
Expand Down
15 changes: 0 additions & 15 deletions docs/sources/mimir/operators-guide/mimir-runbooks/_index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1168,21 +1168,6 @@ How to **investigate**:
1. The alert fired because of a bug in Mimir: fix it.
1. The alert fired because of a bug or edge case in the continuous test tool, causing a false positive: fix it.

### MimirDistributorForwardingErrorRate

This alert fires when the Distributor is trying to forward samples to a forwarding target, but the forwarding requests
result in errors at a high rate.

How it **works**:

- The alert compares the total rate of forwarding requests to the rate of forwarding requests which result in an error.

How to **investigate**:

- Check the `Mimir / Writes` dashboard, it should have a row named `Distributor Forwarding` which also shows the type of error if an HTTP status code was returned.
- Check the Distributor logs, depending on the type of errors which occur the Distributor might log information about the errors.
- Check what the forwarding targets are in use, this can be seen in the runtime config under the key `forwarding_endpoint`, then check the logs of the forwarding target(s).

### MimirRingMembersMismatch

This alert fires when the number of ring members does not match the number of running replicas.
Expand Down
Loading

0 comments on commit ce87556

Please sign in to comment.