From 95e8b8874cc75342e196681f8b5a9ae233b8ed87 Mon Sep 17 00:00:00 2001 From: shivam Date: Fri, 2 Dec 2022 16:26:23 +0530 Subject: [PATCH 1/6] Remove backoff from pubsub Signed-off-by: shivam --- .../supported-pubsub/setup-mqtt.md | 6 ------ .../supported-pubsub/setup-pulsar.md | 14 +------------- 2 files changed, 1 insertion(+), 19 deletions(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md index 1fbfa6105d1..801319d103b 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md @@ -28,8 +28,6 @@ spec: value: "false" - name: cleanSession value: "false" - - name: backOffMaxRetries - value: "0" ``` {{% alert title="Warning" color="warning" %}} @@ -70,8 +68,6 @@ spec: value: "false" - name: cleanSession value: "false" - - name: backoffMaxRetries - value: "0" - name: caCert value: ${{ myLoadedCACert }} - name: clientCert @@ -109,8 +105,6 @@ spec: value: "false" - name: cleanSession value: "true" - - name: backoffMaxRetries - value: "0" ``` {{% alert title="Warning" color="warning" %}} diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md index df12e402712..2838587f210 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md @@ -32,10 +32,6 @@ spec: value: "default" - name: persistent value: "true" - - name: backOffPolicy - value: "constant" - - name: backOffMaxRetries - value: "-1" - name: disableBatching value: "false" ``` @@ -49,15 +45,7 @@ spec: | token | N | Enable Authentication. | [How to create pulsar token](https://pulsar.apache.org/docs/en/security-jwt/#generate-tokens)| | tenant | N | The topic tenant within the instance. Tenants are essential to multi-tenancy in Pulsar, and spread across clusters. Default: `"public"` | `"public"` | | namespace | N | The administrative unit of the topic, which acts as a grouping mechanism for related topics. Default: `"default"` | `"default"` -| persistent | N | Pulsar supports two kind of topics: [persistent](https://pulsar.apache.org/docs/en/concepts-architecture-overview#persistent-storage) and [non-persistent](https://pulsar.apache.org/docs/en/concepts-messaging/#non-persistent-topics). With persistent topics, all messages are durably persisted on disks (if the broker is not standalone, messages are durably persisted on multiple disks), whereas data for non-persistent topics is not persisted to storage disks. Note: the default retry behavior is to retry until it succeeds, so when you use a non-persistent theme, you can reduce or prohibit retries by defining `backOffMaxRetries` to `0`. Default: `"true"` | `"true"`, `"false"` -| backOffPolicy | N | Retry policy, `"constant"` is a backoff policy that always returns the same backoff delay. `"exponential"` is a backoff policy that increases the backoff period for each retry attempt using a randomization function that grows exponentially. Defaults to `"constant"`. | `constant`、`exponential` | -| backOffDuration | N | The fixed interval only takes effect when the `backOffPolicy` is `"constant"`. There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Defaults to `"5s"`. | `"5s"`、`"5000"` | -| backOffInitialInterval | N | The backoff initial interval on retry. Only takes effect when the `backOffPolicy` is `"exponential"`. There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Defaults to `"500"` | `"50"` | -| backOffMaxInterval | N | The backoff initial interval on retry. Only takes effect when the `backOffPolicy` is `"exponential"`. There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Defaults to `"60s"` | `"60000"` | -| backOffMaxRetries | N | The maximum number of retries to process the message before returning an error. Defaults to `"0"` which means the component will not retry processing the message. `"-1"` will retry indefinitely until the message is processed or the application is shutdown. Any positive number is treated as the maximum retry count. | `"3"` | -| backOffRandomizationFactor | N | Randomization factor, between 1 and 0, including 0 but not 1. Randomized interval = RetryInterval * (1 ± backOffRandomizationFactor). Defaults to `"0.5"`. | `"0.5"` | -| backOffMultiplier | N | Backoff multiplier for the policy. Increments the interval by multiplying it with the multiplier. Defaults to `"1.5"` | `"1.5"` | -| backOffMaxElapsedTime | N | After MaxElapsedTime the ExponentialBackOff returns Stop. There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Defaults to `"15m"` | `"15m"` | +| persistent | N | Pulsar supports two kind of topics: [persistent](https://pulsar.apache.org/docs/en/concepts-architecture-overview#persistent-storage) and [non-persistent](https://pulsar.apache.org/docs/en/concepts-messaging/#non-persistent-topics). With persistent topics, all messages are durably persisted on disks (if the broker is not standalone, messages are durably persisted on multiple disks), whereas data for non-persistent topics is not persisted to storage disks. | disableBatching | N | disable batching.When batching enabled default batch delay is set to 10 ms and default batch size is 1000 messages,Setting `disableBatching: true` will make the producer to send messages individually. Default: `"false"` | `"true"`, `"false"`| | batchingMaxPublishDelay | N | batchingMaxPublishDelay set the time period within which the messages sent will be batched,if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or batchingMaxMessages (see below) or batchingMaxSize (see below). There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Default: `"10ms"` | `"10ms"`, `"10"`| | batchingMaxMessages | N | batchingMaxMessages set the maximum number of messages permitted in a batch.If set to a value greater than 1, messages will be queued until this threshold is reached or batchingMaxSize (see below) has been reached or the batch interval has elapsed. Default: `"1000"` | `"1000"`| From 291574a77368140a735f27ee44c644a0725b6bad Mon Sep 17 00:00:00 2001 From: Shivam Kumar Date: Sun, 11 Dec 2022 20:09:15 +0530 Subject: [PATCH 2/6] Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Shivam Kumar --- .../components-reference/supported-pubsub/setup-pulsar.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md index 2838587f210..4366249d45a 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md @@ -45,7 +45,7 @@ spec: | token | N | Enable Authentication. | [How to create pulsar token](https://pulsar.apache.org/docs/en/security-jwt/#generate-tokens)| | tenant | N | The topic tenant within the instance. Tenants are essential to multi-tenancy in Pulsar, and spread across clusters. Default: `"public"` | `"public"` | | namespace | N | The administrative unit of the topic, which acts as a grouping mechanism for related topics. Default: `"default"` | `"default"` -| persistent | N | Pulsar supports two kind of topics: [persistent](https://pulsar.apache.org/docs/en/concepts-architecture-overview#persistent-storage) and [non-persistent](https://pulsar.apache.org/docs/en/concepts-messaging/#non-persistent-topics). With persistent topics, all messages are durably persisted on disks (if the broker is not standalone, messages are durably persisted on multiple disks), whereas data for non-persistent topics is not persisted to storage disks. +| persistent | N | Pulsar supports two kinds of topics: [persistent](https://pulsar.apache.org/docs/en/concepts-architecture-overview#persistent-storage) and [non-persistent](https://pulsar.apache.org/docs/en/concepts-messaging/#non-persistent-topics). With persistent topics, all messages are durably persisted on disks (if the broker is not standalone, messages are durably persisted on multiple disks), whereas data for non-persistent topics is not persisted to storage disks. | disableBatching | N | disable batching.When batching enabled default batch delay is set to 10 ms and default batch size is 1000 messages,Setting `disableBatching: true` will make the producer to send messages individually. Default: `"false"` | `"true"`, `"false"`| | batchingMaxPublishDelay | N | batchingMaxPublishDelay set the time period within which the messages sent will be batched,if batch messages are enabled. If set to a non zero value, messages will be queued until this time interval or batchingMaxMessages (see below) or batchingMaxSize (see below). There are two valid formats, one is the fraction with a unit suffix format, and the other is the pure digital format that is processed as milliseconds. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Default: `"10ms"` | `"10ms"`, `"10"`| | batchingMaxMessages | N | batchingMaxMessages set the maximum number of messages permitted in a batch.If set to a value greater than 1, messages will be queued until this threshold is reached or batchingMaxSize (see below) has been reached or the batch interval has elapsed. Default: `"1000"` | `"1000"`| From d7e71a6fc7e0c96be15fb7cb96cc3ae8bcace7a8 Mon Sep 17 00:00:00 2001 From: Shivam Kumar Date: Sat, 14 Jan 2023 18:58:27 +0530 Subject: [PATCH 3/6] Adding retry messages Signed-off-by: Shivam Kumar --- .../supported-pubsub/setup-mqtt.md | 13 +++++++++++++ .../supported-pubsub/setup-pulsar.md | 4 ++++ 2 files changed, 17 insertions(+) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md index 801319d103b..1206d056737 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md @@ -47,6 +47,19 @@ The above example uses secrets as plain strings. It is recommended to use a secr | caCert | Required for using TLS | Certificate Authority (CA) certificate in PEM format for verifying server TLS certificates. | `"-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----"` | clientCert | Required for using TLS | TLS client certificate in PEM format. Must be used with `clientKey`. | `"-----BEGIN CERTIFICATE-----\n\n-----END CERTIFICATE-----"` | clientKey | Required for using TLS | TLS client key in PEM format. Must be used with `clientCert`. Can be `secretKeyRef` to use a secret reference. | `"-----BEGIN RSA PRIVATE KEY-----\n\n-----END RSA PRIVATE KEY-----"` + +### Enabling message delivery retries + +The MQTT pub/sub component has no built-in support for retry strategies. This means that sidecar sends a message to the service only once. If the service marks the message as not processed, the message won't be acknowledged to the broker. Only if broker resends the message, it would be retried. + +To make Dapr use more spohisticated retry policies, you can apply a [retry resiliency policy]({{< ref "policies.md#retries" >}}) to the MQTT pub/sub component. + +There is a crucial difference between the two ways of retries: + +1. Re-delivery of unacknowledged messages is completely dependent to the broker. Dapr does not guarantee it. Some brokers like [emqx](https://www.emqx.io/), [vernemq](https://vernemq.com/) etc. support it but it not a part of [MQTT3 spec](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718103). + +2. Using Resiliency makes the same Dapr sidecar retry redelivering the messages. So it will be the same Dapr sidecar and the same app receiving the same message. + ### Communication using TLS To configure communication using TLS, ensure that the MQTT broker (e.g. mosquitto) is configured to support certificates and provide the `caCert`, `clientCert`, `clientKey` metadata in the component configuration. For example: diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md index 4366249d45a..b965e75b432 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md @@ -51,6 +51,10 @@ spec: | batchingMaxMessages | N | batchingMaxMessages set the maximum number of messages permitted in a batch.If set to a value greater than 1, messages will be queued until this threshold is reached or batchingMaxSize (see below) has been reached or the batch interval has elapsed. Default: `"1000"` | `"1000"`| | batchingMaxSize | N | batchingMaxSize sets the maximum number of bytes permitted in a batch. If set to a value greater than 1, messages will be queued until this threshold is reached or batchingMaxMessages (see above) has been reached or the batch interval has elapsed. Default: `"128KB"` | `"131072"`| +### Enabling message delivery retries + +The Pulsar pub/sub component has no built-in support for retry strategies. This means that sidecar sends a message to the service only once and is not retried in case of failures. To make Dapr use more spohisticated retry policies, you can apply a [retry resiliency policy]({{< ref "policies.md#retries" >}}) to the MQTT pub/sub component. Note that it will be the same Dapr sidecar retry redelivering the message to the same app instance and not other instances. + ### Delay queue When invoking the Pulsar pub/sub, it's possible to provide an optional delay queue by using the `metadata` query parameters in the request url. From 784387dab109d90d9feb4d2ef039cf7e0636d13e Mon Sep 17 00:00:00 2001 From: Mark Fussell Date: Fri, 20 Jan 2023 12:11:20 -0800 Subject: [PATCH 4/6] Update daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md Co-authored-by: Deepanshu Agarwal Signed-off-by: Mark Fussell --- .../components-reference/supported-pubsub/setup-mqtt.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md index 1206d056737..43882cc8416 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md @@ -56,7 +56,7 @@ To make Dapr use more spohisticated retry policies, you can apply a [retry resil There is a crucial difference between the two ways of retries: -1. Re-delivery of unacknowledged messages is completely dependent to the broker. Dapr does not guarantee it. Some brokers like [emqx](https://www.emqx.io/), [vernemq](https://vernemq.com/) etc. support it but it not a part of [MQTT3 spec](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718103). +1. Re-delivery of unacknowledged messages is completely dependent on the broker. Dapr does not guarantee it. Some brokers like [emqx](https://www.emqx.io/), [vernemq](https://vernemq.com/) etc. support it but it not a part of [MQTT3 spec](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718103). 2. Using Resiliency makes the same Dapr sidecar retry redelivering the messages. So it will be the same Dapr sidecar and the same app receiving the same message. From 01b7887e526ab610c4ad01dfa6ceac575ab311f4 Mon Sep 17 00:00:00 2001 From: Shivam Kumar Date: Tue, 24 Jan 2023 14:21:32 +0530 Subject: [PATCH 5/6] Apply suggestions from code review Co-authored-by: Mark Fussell Signed-off-by: Shivam Kumar --- .../components-reference/supported-pubsub/setup-mqtt.md | 4 ++-- .../components-reference/supported-pubsub/setup-pulsar.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md index 43882cc8416..fe2843c3b94 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-mqtt.md @@ -50,7 +50,7 @@ The above example uses secrets as plain strings. It is recommended to use a secr ### Enabling message delivery retries -The MQTT pub/sub component has no built-in support for retry strategies. This means that sidecar sends a message to the service only once. If the service marks the message as not processed, the message won't be acknowledged to the broker. Only if broker resends the message, it would be retried. +The MQTT pub/sub component has no built-in support for retry strategies. This means that the sidecar sends a message to the service only once. If the service marks the message as not processed, the message won't be acknowledged back to the broker. Only if broker resends the message, would it would be retried. To make Dapr use more spohisticated retry policies, you can apply a [retry resiliency policy]({{< ref "policies.md#retries" >}}) to the MQTT pub/sub component. @@ -58,7 +58,7 @@ There is a crucial difference between the two ways of retries: 1. Re-delivery of unacknowledged messages is completely dependent on the broker. Dapr does not guarantee it. Some brokers like [emqx](https://www.emqx.io/), [vernemq](https://vernemq.com/) etc. support it but it not a part of [MQTT3 spec](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc398718103). -2. Using Resiliency makes the same Dapr sidecar retry redelivering the messages. So it will be the same Dapr sidecar and the same app receiving the same message. +2. Using a [retry resiliency policy]({{< ref "policies.md#retries" >}}) makes the same Dapr sidecar retry redelivering the messages. So it is the same Dapr sidecar and the same app receiving the same message. ### Communication using TLS diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md index b965e75b432..128d66715f0 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-pulsar.md @@ -53,7 +53,7 @@ spec: ### Enabling message delivery retries -The Pulsar pub/sub component has no built-in support for retry strategies. This means that sidecar sends a message to the service only once and is not retried in case of failures. To make Dapr use more spohisticated retry policies, you can apply a [retry resiliency policy]({{< ref "policies.md#retries" >}}) to the MQTT pub/sub component. Note that it will be the same Dapr sidecar retry redelivering the message to the same app instance and not other instances. +The Pulsar pub/sub component has no built-in support for retry strategies. This means that sidecar sends a message to the service only once and is not retried in case of failures. To make Dapr use more spohisticated retry policies, you can apply a [retry resiliency policy]({{< ref "policies.md#retries" >}}) to the MQTT pub/sub component. Note that it will be the same Dapr sidecar retrying the redelivery the message to the same app instance and not other instances. ### Delay queue From d78a25553c94fb86b61c8bfca57493e93f2a6503 Mon Sep 17 00:00:00 2001 From: Shivam Kumar Date: Wed, 1 Feb 2023 14:58:11 +0530 Subject: [PATCH 6/6] Fixing broken link Signed-off-by: Shivam Kumar --- .../components-reference/supported-pubsub/setup-apache-kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md index 86d4a350c58..a18b8c26105 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-apache-kafka.md @@ -337,7 +337,7 @@ To run without Docker, see the getting started guide [here](https://kafka.apache {{% /codetab %}} {{% codetab %}} -To run Kafka on Kubernetes, you can use any Kafka operator, such as [Strimzi](https://strimzi.io/docs/operators/latest/quickstart.html#ref-install-prerequisites-str). +To run Kafka on Kubernetes, you can use any Kafka operator, such as [Strimzi](https://strimzi.io/quickstarts/). {{% /codetab %}} {{< /tabs >}}