diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md index 5393fd7efce..c11534932d0 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md @@ -1,42 +1,41 @@ --- type: docs -title: "How-To: Publish a message and subscribe to a topic" -linkTitle: "How-To: Publish & subscribe" +title: "How to: Publish a message and subscribe to a topic" +linkTitle: "How to: Publish & subscribe to topics" weight: 2000 description: "Learn how to send messages to a topic with one service and subscribe to that topic in another service" --- -## Introduction +Now that you've learned what the Dapr pub/sub building block provides, learn how it can work in your service. The below code example loosely describes an application that processes orders with two services, each with Dapr sidecars: -Pub/Sub is a common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. -Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers. +- A checkout service using Dapr to subscribe to the topic in the message queue. +- An order processing service using Dapr to publish a message to RabbitMQ. -Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. -Dapr provides components for pub/sub, that enable operators to use their preferred infrastructure, for example Redis Streams, Kafka, etc. -## Content Types +Diagram showing state management of example service -When publishing a message, it's important to specify the content type of the data being sent. -Unless specified, Dapr will assume `text/plain`. When using Dapr's HTTP API, the content type can be set in a `Content-Type` header. -gRPC clients and SDKs have a dedicated content type parameter. +Dapr automatically wraps the user payload in a CloudEvents v1.0 compliant envelope, using `Content-Type` header value for `datacontenttype` attribute. [Learn more about messages with CloudEvents.]({{< ref pubsub-cloudevents.md >}}) -## Example: +The following example demonstrates how your applications publish and subscribe to a topic called `orders`. -The below code example loosely describes an application that processes orders. In the example, there are two services - an order processing service and a checkout service. Both services have Dapr sidecars. The order processing service uses Dapr to publish a message to RabbitMQ and the checkout service subscribes to the topic in the message queue. +{{% alert title="Note" color="primary" %}} + If you haven't already, [try out the pub/sub quickstart]({{< ref pubsub-quickstart.md >}}) for a quick walk-through on how to use pub/sub. -Diagram showing state management of example service +{{% /alert %}} -## Step 1: Setup the Pub/Sub component -The following example creates applications to publish and subscribe to a topic called `orders`. +## Set up the Pub/Sub component -The first step is to setup the Pub/Sub component: +The first step is to set up the pub/sub component: {{< tabs "Self-Hosted (CLI)" Kubernetes >}} {{% codetab %}} -The pubsub.yaml is created by default on your local machine when running `dapr init`. Verify by opening your components file under `%UserProfile%\.dapr\components\pubsub.yaml` on Windows or `~/.dapr/components/pubsub.yaml` on Linux/MacOS. +When you run `dapr init`, Dapr creates a default Redis `pubsub.yaml` and runs a Redis container on your local machine, located: + +- On Windows, under `%UserProfile%\.dapr\components\pubsub.yaml` +- On Linux/MacOS, under `~/.dapr/components/pubsub.yaml` -In this example, RabbitMQ is used for publish and subscribe. Replace `pubsub.yaml` file contents with the below contents. +With the `pubsub.yaml` component, you can easily swap out underlying components without application code changes. In this example, RabbitMQ is used. ```yaml apiVersion: dapr.io/v1alpha1 @@ -64,11 +63,55 @@ scopes: - checkout ``` -You can override this file with another Redis instance or another [pubsub component]({{< ref setup-pubsub >}}) by creating a `components` directory containing the file and using the flag `--components-path` with the `dapr run` CLI command. +You can override this file with another [pubsub component]({{< ref setup-pubsub >}}) by creating a components directory (in this example, `myComponents`) containing the file and using the flag `--components-path` with the `dapr run` CLI command. + +{{< tabs Dotnet Java Python Go Javascript >}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- dotnet run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- python3 app.py +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- go run app.go +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- npm start +``` +{{% /codetab %}} + +{{< /tabs >}} + {{% /codetab %}} {{% codetab %}} -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of your [desired pubsub component]({{< ref setup-pubsub >}}) in the yaml below, save as `pubsub.yaml`, and run `kubectl apply -f pubsub.yaml`. +To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of the [pub/sub component]({{< ref setup-pubsub >}}) in the YAML below, save as `pubsub.yaml`, and run `kubectl apply -f pubsub.yaml`. ```yaml apiVersion: dapr.io/v1alpha1 @@ -96,26 +139,21 @@ scopes: - orderprocessing - checkout ``` + {{% /codetab %}} {{< /tabs >}} +## Subscribe to topics -## Step 2: Subscribe to topics - -Dapr allows two methods by which you can subscribe to topics: +Dapr provides two methods by which you can subscribe to topics: - **Declaratively**, where subscriptions are defined in an external file. - **Programmatically**, where subscriptions are defined in user code. -{{% alert title="Note" color="primary" %}} - Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependency from your code and allows, for example, existing applications to subscribe to topics, without having to change code. The programmatic approach implements the subscription in your code. - -{{% /alert %}} - -### Declarative subscriptions +Learn more in the [declarative and programmatic subscriptions doc]({{< ref subscription-methods.md >}}). This example demonstrates a **declarative** subscription. -You can subscribe to a topic using the following Custom Resources Definition (CRD). Create a file named `subscription.yaml` and paste the following: +Create a file named `subscription.yaml` and paste the following: ```yaml apiVersion: dapr.io/v1alpha1 @@ -132,69 +170,13 @@ scopes: ``` The example above shows an event subscription to topic `orders`, for the pubsub component `order-pub-sub`. + - The `route` field tells Dapr to send all topic messages to the `/checkout` endpoint in the app. - The `scopes` field enables this subscription for apps with IDs `orderprocessing` and `checkout`. -Set the component with: - -Place the CRD in your `./components` directory. When Dapr starts up, it loads subscriptions along with components. - -Note: By default, Dapr loads components from `$HOME/.dapr/components` on MacOS/Linux and `%USERPROFILE%\.dapr\components` on Windows. - -You can also override the default directory by pointing the Dapr CLI to a components path: - -{{< tabs Dotnet Java Python Go Javascript Kubernetes>}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- dotnet run -``` - -{{% /codetab %}} +Place `subscription.yaml` in the same directory as your `pubsub.yaml` component. When Dapr starts up, it loads subscriptions along with the components. -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- python3 app.py -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- go run app.go -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- npm start -``` - -{{% /codetab %}} - -{{% codetab %}} -In Kubernetes, save the CRD to a file and apply it to the cluster: -```bash -kubectl apply -f subscription.yaml -``` -{{% /codetab %}} - -{{< /tabs >}} - -Below are code examples that leverage Dapr SDKs to subscribe to a topic. +Below are code examples that leverage Dapr SDKs to subscribe to the topic you defined in `subscription.yaml`. {{< tabs Dotnet Java Python Go Javascript>}} @@ -226,7 +208,7 @@ namespace CheckoutService.controller } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 --app-ssl dotnet run @@ -266,7 +248,7 @@ public class CheckoutServiceController { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 mvn spring-boot:run @@ -295,7 +277,7 @@ def mytopic(event: v1.Event) -> None: app.run(6002) ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --app-protocol grpc -- python3 CheckoutService.py @@ -340,7 +322,7 @@ func eventHandler(ctx context.Context, e *common.TopicEvent) (retry bool, err er } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 go run CheckoutService.go @@ -380,7 +362,7 @@ async function start(orderId) { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 npm start @@ -390,44 +372,44 @@ dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-por {{< /tabs >}} -The `/checkout` endpoint matches the `route` defined in the subscriptions and this is where Dapr will send all topic messages to. - -## Step 3: Publish a topic +## Publish a message Start an instance of Dapr with an app-id called `orderprocessing`: ```bash dapr run --app-id orderprocessing --dapr-http-port 3601 ``` + +Then publish a message to the `orders` topic: + {{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} {{% codetab %}} -Then publish a message to the `orders` topic: - ```bash dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"orderId": "100"}' ``` + {{% /codetab %}} {{% codetab %}} -Then publish a message to the `orders` topic: + ```bash curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/json" -d '{"orderId": "100"}' ``` + {{% /codetab %}} {{% codetab %}} -Then publish a message to the `orders` topic: + ```powershell Invoke-RestMethod -Method Post -ContentType 'application/json' -Body '{"orderId": "100"}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' ``` + {{% /codetab %}} {{< /tabs >}} -Dapr automatically wraps the user payload in a Cloud Events v1.0 compliant envelope, using `Content-Type` header value for `datacontenttype` attribute. - Below are code examples that leverage Dapr SDKs to publish a topic. {{< tabs Dotnet Java Python Go Javascript>}} @@ -470,7 +452,7 @@ namespace EventService } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --app-ssl dotnet run @@ -520,7 +502,7 @@ public class OrderProcessingServiceApplication { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 mvn spring-boot:run @@ -557,7 +539,7 @@ while True: logging.info('Published data: ' + str(orderId)) ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --app-protocol grpc python3 OrderProcessingService.py @@ -605,7 +587,7 @@ func main() { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 go run OrderProcessingService.go @@ -648,7 +630,7 @@ function sleep(ms) { main(); ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 npm start @@ -658,50 +640,13 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{< /tabs >}} -## Step 4: ACK-ing a message +## Message acknowledgement and retries In order to tell Dapr that a message was processed successfully, return a `200 OK` response. If Dapr receives any other return status code than `200`, or if your app crashes, Dapr will attempt to redeliver the message following at-least-once semantics. -## Sending a custom CloudEvent - -Dapr automatically takes the data sent on the publish request and wraps it in a CloudEvent 1.0 envelope. -If you want to use your own custom CloudEvent, make sure to specify the content type as `application/cloudevents+json`. - -Read about content types [here](#content-types), and about the [Cloud Events message format]({{< ref "pubsub-overview.md#cloud-events-message-format" >}}). - -#### Example - -{{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```bash -dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -``` -{{% /codetab %}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```bash -curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/cloudevents+json" -d '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -``` -{{% /codetab %}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```powershell -Invoke-RestMethod -Method Post -ContentType 'application/cloudevents+json' -Body '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' -``` -{{% /codetab %}} - -{{< /tabs >}} - ## Next steps -- Try the [Pub/Sub quickstart sample](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) -- Learn about [PubSub routing]({{< ref howto-route-messages >}}) -- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -- List of [pub/sub components]({{< ref setup-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Try the [pub/sub tutorial]({{https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub}}). +- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md index 5fac6c647b2..be9c37f7966 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md @@ -2,23 +2,23 @@ type: docs title: "How-To: Route messages to different event handlers" linkTitle: "How-To: Route events" -weight: 2100 +weight: 2300 description: "Learn how to route messages from a topic to different event handlers based on CloudEvent fields" --- {{% alert title="Preview feature" color="warning" %}} -Pub/Sub message routing is currently in [preview]({{< ref preview-features.md >}}). +Pub/sub message routing is currently in [preview]({{< ref preview-features.md >}}). {{% /alert %}} -## Introduction +Pub/sub routing is an implementation of [content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html), a messaging pattern that utilizes a DSL instead of imperative application code. With pub/sub routing, you use expressions to route [CloudEvents](https://cloudevents.io) (based on their contents) to different URIs/paths and event handlers in your application. If no route matches, then an optional default route is used. This proves useful as your applications expand to support multiple event versions or special cases. -[Content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html) is a messaging pattern that utilizes a DSL instead of imperative application code. PubSub routing is an implementation of this pattern that allows developers to use expressions to route [CloudEvents](https://cloudevents.io) based on their contents to different URIs/paths and event handlers in your application. If no route matches, then an optional default route is used. This becomes useful as your applications expands to support multiple event versions, or special cases. Routing can be implemented with code; however, keeping routing rules external from the application can improve portability. +While routing can be implemented with code, keeping routing rules external from the application can improve portability. -This feature is available to both the declarative and programmatic subscription approaches. +This feature is available to both the [declarative and programmatic subscription approaches]({{< ref subscription-methods.md >}}). ## Enable message routing -This is a preview feature. To enable it, add the `PubSub.Routing` feature entry to your application configuration like so: +To enable this preview feature, add the `PubSub.Routing` feature entry to your application configuration: ```yaml apiVersion: dapr.io/v1alpha1 @@ -30,10 +30,12 @@ spec: - name: PubSub.Routing enabled: true ``` -Learn more about enabling [preview features]({{}}). + +Learn more about enabling [preview features]({{< ref preview-features >}}). + ## Declarative subscription -For declarative subscriptions, you must use `dapr.io/v2alpha1` as the `apiVersion`. Here is an example of `subscriptions.yaml` using routing. +For declarative subscriptions, use `dapr.io/v2alpha1` as the `apiVersion`. Here is an example of `subscriptions.yaml` using routing: ```yaml apiVersion: dapr.io/v2alpha1 @@ -57,7 +59,7 @@ scopes: ## Programmatic subscription -Alternatively, the programattic approach varies slightly in that the `routes` structure is returned instead of `route`. The JSON structure matches the declarative YAML. +In the programmatic approach, the `routes` structure is returned instead of `route`. The JSON structure matches the declarative YAML: {{< tabs Python Node "C#" Go PHP>}} @@ -268,27 +270,34 @@ $app->start(); ## Common Expression Language (CEL) -In these examples, depending on the type of the event (`event.type`), the application will be called on `/widgets`, `/gadgets` or `/products`. The expressions are written as [Common Expression Language (CEL)](https://github.com/google/cel-spec) where `event` represents the cloud event. Any of the attributes from the [CloudEvents core specification](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) can be referenced in the expression. +In these examples, depending on the `event.type`, the application will be called on: + +- `/widgets` +- `/gadgets` +- `/products` + +The expressions are written as [Common Expression Language (CEL)](https://github.com/google/cel-spec) where `event` represents the cloud event. Any of the attributes from the [CloudEvents core specification](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) can be referenced in the expression. ### Example expressions -Match "important" messages +Match "important" messages: ```javascript has(event.data.important) && event.data.important == true ``` -Match deposits greater than $10000 +Match deposits greater than $10,000: ```javascript event.type == "deposit" && event.data.amount > 10000 ``` -Match multiple versions of a message +Match multiple versions of a message: ```javascript event.type == "mymessage.v1" ``` + ```javascript event.type == "mymessage.v2" ``` @@ -301,201 +310,154 @@ For reference, the following attributes are from the CloudEvents specification. #### data -As defined by the term Data, CloudEvents MAY include domain-specific information about the occurrence. When present, this information will be encapsulated within `data`. +As defined by the term **data**, CloudEvents _may_ include domain-specific information about the occurrence. When present, this information will be encapsulated within `data`. -- Description: The event payload. This specification does not place any restriction on the type of this information. It is encoded into a media format which is specified by the `datacontenttype` attribute (e.g. application/json), and adheres to the `dataschema` format when those respective attributes are present. -- Constraints: +- **Description:** The event payload. This specification places no restriction on the information type. It is encoded into a media format, specified by the `datacontenttype` attribute (e.g. application/json), and adheres to the `dataschema` format when those respective attributes are present. +- **Constraints:** - OPTIONAL {{% alert title="Limitation" color="warning" %}} -Currently, it is only possible to access the attributes inside data if it is nested JSON values and not JSON escaped in a string. +Currently, you can only access the attributes inside data if it is nested JSON values and not JSON escaped in a string. {{% /alert %}} ### REQUIRED Attributes -The following attributes are REQUIRED to be present in all CloudEvents: +The following attributes are **required** in all CloudEvents: #### id -- Type: `String` -- Description: Identifies the event. Producers MUST ensure that `source` + `id` - is unique for each distinct event. If a duplicate event is re-sent (e.g. due - to a network error) it MAY have the same `id`. Consumers MAY assume that - Events with identical `source` and `id` are duplicates. -- Constraints: +- **Type:** `String` +- **Description:** Identifies the event. Producers _must_ ensure that `source` + `id` + are unique for each distinct event. If a duplicate event is re-sent (e.g. due + to a network error), it may have the same `id`. Consumers may assume that + events with identical `source` and `id` are duplicates. +- **Constraints:** - REQUIRED - - MUST be a non-empty string - - MUST be unique within the scope of the producer -- Examples: + - Must be a non-empty string + - Must be unique within the scope of the producer +- **Examples:** - An event counter maintained by the producer - A UUID #### source -- Type: `URI-reference` -- Description: Identifies the context in which an event happened. Often this - will include information such as the type of the event source, the - organization publishing the event or the process that produced the event. The - exact syntax and semantics behind the data encoded in the URI is defined by - the event producer. +- **Type:** `URI-reference` +- **Description:** Identifies the context in which an event happened. Often this includes information such as: + - The type of the event source + - The organization publishing the event + - The process that produced the event + + The exact syntax and semantics behind the data encoded in the URI is defined by the event producer. - Producers MUST ensure that `source` + `id` is unique for each distinct event. + Producers _must_ ensure that `source` + `id` are unique for each distinct event. - An application MAY assign a unique `source` to each distinct producer, which - makes it easy to produce unique IDs since no other producer will have the same - source. The application MAY use UUIDs, URNs, DNS authorities or an - application-specific scheme to create unique `source` identifiers. + An application may: + - Assign a unique `source` to each distinct producer, making it easier to produce unique IDs and preventing other producers from having the same `source`. + - Use UUIDs, URNs, DNS authorities, or an application-specific scheme to create unique `source` identifiers. - A source MAY include more than one producer. In that case the producers MUST - collaborate to ensure that `source` + `id` is unique for each distinct event. + A source may include more than one producer. In this case, the producers _must_ collaborate to ensure that `source` + `id` are unique for each distinct event. -- Constraints: +- **Constraints:** - REQUIRED - - MUST be a non-empty URI-reference + - Must be a non-empty URI-reference - An absolute URI is RECOMMENDED -- Examples - - Internet-wide unique URI with a DNS authority. +- **Examples:** + - Internet-wide unique URI with a DNS authority: - https://github.com/cloudevents - mailto:cncf-wg-serverless@lists.cncf.io - Universally-unique URN with a UUID: - urn:uuid:6e8bc430-9c3a-11d9-9669-0800200c9a66 - - Application-specific identifiers + - Application-specific identifiers: - /cloudevents/spec/pull/123 - /sensors/tn-1234567/alerts - 1-555-123-4567 #### specversion -- Type: `String` -- Description: The version of the CloudEvents specification which the event - uses. This enables the interpretation of the context. Compliant event - producers MUST use a value of `1.0` when referring to this version of the - specification. +- **Type:** `String` +- **Description:** The version of the CloudEvents specification used by the event. This enables the interpretation of the context. Compliant event producers _must_ use a value of `1.0` when referring to this version of the specification. + + Currently, this attribute only includes the 'major' and 'minor' version numbers. This allows patch changes to the specification to be made without changing this property's value in the serialization. - Currently, this attribute will only have the 'major' and 'minor' version - numbers included in it. This allows for 'patch' changes to the specification - to be made without changing this property's value in the serialization. - Note: for 'release candidate' releases a suffix might be used for testing + Note: for 'release candidate' releases, a suffix might be used for testing purposes. -- Constraints: +- **Constraints:** - REQUIRED - - MUST be a non-empty string + - Must be a non-empty string #### type -- Type: `String` -- Description: This attribute contains a value describing the type of event - related to the originating occurrence. Often this attribute is used for - routing, observability, policy enforcement, etc. The format of this is - producer defined and might include information such as the version of the - `type` - see - [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) - for more information. -- Constraints: +- **Type:** `String` +- **Description:** Contains a value describing the event type related to the originating occurrence. Often, this attribute is used for routing, observability, policy enforcement, etc. The format is producer-defined and might include information like the version of the `type`. See [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) for more information. +- **Constraints:** - REQUIRED - - MUST be a non-empty string - - SHOULD be prefixed with a reverse-DNS name. The prefixed domain dictates the - organization which defines the semantics of this event type. -- Examples + - Must be a non-empty string + - Should be prefixed with a reverse-DNS name. The prefixed domain dictates the + organization, which defines the semantics of this event type. +- **Examples:** - com.github.pull_request.opened - com.example.object.deleted.v2 ### OPTIONAL Attributes -The following attributes are OPTIONAL to appear in CloudEvents. See the -[Notational Conventions](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#notational-conventions) section for more information -on the definition of OPTIONAL. +The following attributes are **optional** to appear in CloudEvents. See the [Notational Conventions](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#notational-conventions) section for more information on the definition of OPTIONAL. #### datacontenttype -- Type: `String` per [RFC 2046](https://tools.ietf.org/html/rfc2046) -- Description: Content type of `data` value. This attribute enables `data` to - carry any type of content, whereby format and encoding might differ from that - of the chosen event format. For example, an event rendered using the - [JSON envelope](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#3-envelope) format might carry an XML payload - in `data`, and the consumer is informed by this attribute being set to - "application/xml". The rules for how `data` content is rendered for different - `datacontenttype` values are defined in the event format specifications; for - example, the JSON event format defines the relationship in - [section 3.1](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#31-handling-of-data). - - For some binary mode protocol bindings, this field is directly mapped to the - respective protocol's content-type metadata property. Normative rules for the - binary mode and the content-type metadata mapping can be found in the - respective protocol. - - In some event formats the `datacontenttype` attribute MAY be omitted. For - example, if a JSON format event has no `datacontenttype` attribute, then it is - implied that the `data` is a JSON value conforming to the "application/json" - media type. In other words: a JSON-format event with no `datacontenttype` is - exactly equivalent to one with `datacontenttype="application/json"`. - - When translating an event message with no `datacontenttype` attribute to a - different format or protocol binding, the target `datacontenttype` SHOULD be - set explicitly to the implied `datacontenttype` of the source. - -- Constraints: +- **Type:** `String` per [RFC 2046](https://tools.ietf.org/html/rfc2046) +- **Description:** Content type of `data` value. This attribute enables `data` to carry any type of content, whereby format and encoding might differ from that of the chosen event format. + + For example, an event rendered using the [JSON envelope](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#3-envelope) format might carry an XML payload in `data`. The consumer is informed by this attribute being set to `"application/xml"`. + + The rules for how `data` content is rendered for different `datacontenttype` values are defined in the event format specifications. For example, the JSON event format defines the relationship in [section 3.1](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#31-handling-of-data). + + For some binary mode protocol bindings, this field is directly mapped to the respective protocol's content-type metadata property. You can find normative rules for the binary mode and the content-type metadata mapping in the respective protocol. + + In some event formats, you may omit the `datacontenttype` attribute. For example, if a JSON format event has no `datacontenttype` attribute, it's implied that the `data` is a JSON value conforming to the `"application/json"` media type. In other words: a JSON-format event with no `datacontenttype` is exactly equivalent to one with `datacontenttype="application/json"`. + + When translating an event message with no `datacontenttype` attribute to a different format or protocol binding, the target `datacontenttype` should be set explicitly to the implied `datacontenttype` of the source. + +- **Constraints:** - OPTIONAL - - If present, MUST adhere to the format specified in - [RFC 2046](https://tools.ietf.org/html/rfc2046) -- For Media Type examples see - [IANA Media Types](http://www.iana.org/assignments/media-types/media-types.xhtml) + - If present, must adhere to the format specified in [RFC 2046](https://tools.ietf.org/html/rfc2046) +- For Media Type examples, see [IANA Media Types](http://www.iana.org/assignments/media-types/media-types.xhtml) #### dataschema -- Type: `URI` -- Description: Identifies the schema that `data` adheres to. Incompatible - changes to the schema SHOULD be reflected by a different URI. See - [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) - for more information. -- Constraints: +- **Type:** `URI` +- **Description:** Identifies the schema that `data` adheres to. Incompatible changes to the schema should be reflected by a different URI. See [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) for more information. +- **Constraints:** - OPTIONAL - - If present, MUST be a non-empty URI + - If present, must be a non-empty URI #### subject -- Type: `String` -- Description: This describes the subject of the event in the context of the - event producer (identified by `source`). In publish-subscribe scenarios, a - subscriber will typically subscribe to events emitted by a `source`, but the - `source` identifier alone might not be sufficient as a qualifier for any - specific event if the `source` context has internal sub-structure. - - Identifying the subject of the event in context metadata (opposed to only in - the `data` payload) is particularly helpful in generic subscription filtering - scenarios where middleware is unable to interpret the `data` content. In the - above example, the subscriber might only be interested in blobs with names - ending with '.jpg' or '.jpeg' and the `subject` attribute allows for - constructing a simple and efficient string-suffix filter for that subset of - events. - -- Constraints: +- **Type:** `String` +- **Description:** This describes the event subject in the context of the event producer (identified by `source`). In publish-subscribe scenarios, a subscriber will typically subscribe to events emitted by a `source`. The `source` identifier alone might not be sufficient as a qualifier for any specific event if the `source` context has internal sub-structure. + + Identifying the subject of the event in context metadata (opposed to only in the `data` payload) is helpful in generic subscription filtering scenarios, where middleware is unable to interpret the `data` content. In the above example, the subscriber might only be interested in blobs with names ending with '.jpg' or '.jpeg'. With the `subject` attribute, you can construct a simple and efficient string-suffix filter for that subset of events. + +- **Constraints:** - OPTIONAL - - If present, MUST be a non-empty string -- Example: - - A subscriber might register interest for when new blobs are created inside a - blob-storage container. In this case, the event `source` identifies the - subscription scope (storage container), the `type` identifies the "blob - created" event, and the `id` uniquely identifies the event instance to - distinguish separate occurrences of a same-named blob having been created; - the name of the newly created blob is carried in `subject`: - - `source`: https://example.com/storage/tenant/container - - `subject`: mynewfile.jpg + - If present, must be a non-empty string +- **Example:** + A subscriber might register interest for when new blobs are created inside a blob-storage container. In this case: + - The event `source` identifies the subscription scope (storage container) + - The event `type` identifies the "blob created" event + - The event `id` uniquely identifies the event instance to distinguish separately created occurrences of a same-named blob. + + The name of the newly created blob is carried in `subject`: + - `source`: https://example.com/storage/tenant/container + - `subject`: mynewfile.jpg #### time -- Type: `Timestamp` -- Description: Timestamp of when the occurrence happened. If the time of the - occurrence cannot be determined then this attribute MAY be set to some other - time (such as the current time) by the CloudEvents producer, however all - producers for the same `source` MUST be consistent in this respect. In other - words, either they all use the actual time of the occurrence or they all use - the same algorithm to determine the value used. -- Constraints: +- **Type:** `Timestamp` +- **Description:** Timestamp of when the occurrence happened. If the time of the occurrence cannot be determined, then this attribute may be set to some other time (such as the current time) by the CloudEvents producer. However, all producers for the same `source` _must_ be consistent in this respect. In other words, either they all use the actual time of the occurrence or they all use the same algorithm to determine the value used. +- **Constraints:** - OPTIONAL - - If present, MUST adhere to the format specified in - [RFC 3339](https://tools.ietf.org/html/rfc3339) + - If present, must adhere to the format specified in [RFC 3339](https://tools.ietf.org/html/rfc3339) {{% alert title="Limitation" color="warning" %}} Currently, comparisons to time (e.g. before or after "now") are not supported. @@ -511,9 +473,8 @@ Watch [this video](https://www.youtube.com/watch?v=QqJgRmbH82I&t=1063s) on how t ## Next steps -- Try the [Pub/Sub routing sample](https://github.com/dapr/samples/tree/master/pub-sub-routing) -- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -- List of [pub/sub components]({{< ref setup-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Try the [pub/sub routing sample](https://github.com/dapr/samples/tree/master/pub-sub-routing). +- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) and [message time-to-live]({{< ref pubsub-message-ttl.md >}}). +- [Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md new file mode 100644 index 00000000000..e2ab04d5709 --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md @@ -0,0 +1,109 @@ +--- +type: docs +title: "Publishing & subscribing messages with Cloudevents" +linkTitle: "Messages with Cloudevents" +weight: 2100 +description: "Learn why Dapr uses CloudEvents, how they work in Dapr pub/sub, and how to create CloudEvents." +--- + +To enable message routing and provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a CloudEvents envelope, using the [`Content-Type` header value]({{< ref "pubsub-overview.md#content-types" >}}) for `datacontenttype` attribute. + +Dapr uses CloudEvents to provide additional context to the event payload, enabling features like: + +- Tracing +- Deduplication by message Id +- Content-type for proper deserialization of event data + +## CloudEvents example + +Dapr implements the following CloudEvents fields when creating a message topic. + +- `id` +- `source` +- `specversion` +- `type` +- `traceparent` +- `datacontenttype` (optional) + +The following example demonstrates an `orders` topic message sent by Dapr that includes a W3C `traceid` unique to the message, the `data` and the fields for the CloudEvent where the data content is serialized as JSON. + +```json +{ + "topic": "orders", + "pubsubname": "order_pub_sub", + "traceid": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01", + "tracestate": "", + "data": { + "orderId": 1 + }, + "id": "5929aaac-a5e2-4ca1-859c-edfe73f11565", + "specversion": "1.0", + "datacontenttype": "application/json; charset=utf-8", + "source": "checkout", + "type": "com.dapr.event.sent", + "traceparent": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01" +} +``` + +As another example of a v1.0 CloudEvent, the following shows data as XML content in a CloudEvent message serialized as JSON: + +```json +{ + "specversion" : "1.0", + "type" : "xml.message", + "source" : "https://example.com/message", + "subject" : "Test XML Message", + "id" : "id-1234-5678-9101", + "time" : "2020-09-23T06:23:21Z", + "datacontenttype" : "text/xml", + "data" : "User1user2hi" +} +``` + +## Publish your own CloudEvent + +If you want to use your own CloudEvent, make sure to specify the [`datacontenttype`]({{< ref "pubsub-overview.md#setting-message-content-types" >}}) as `application/cloudevents+json`. + +### Example + +{{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```bash +dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' +``` + +{{% /codetab %}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```bash +curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/cloudevents+json" -d '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' +``` + +{{% /codetab %}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```powershell +Invoke-RestMethod -Method Post -ContentType 'application/cloudevents+json' -Body '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' +``` + +{{% /codetab %}} + +{{< /tabs >}} + +## Next steps + +- Learn why you might [not want to use CloudEvents]({{< ref pubsub-raw.md >}}) +- Try out the [pub/sub Quickstart]({{< ref pubsub-quickstart.md >}}) +- List of [pub/sub components]({{< ref setup-pubsub >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) + diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md index a30acdc5253..d6029842c73 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -3,7 +3,7 @@ type: docs title: "Message Time-to-Live (TTL)" linkTitle: "Message TTL" weight: 6000 -description: "Use time-to-live in Pub/Sub messages." +description: "Use time-to-live in pub/sub messages." --- ## Introduction @@ -18,6 +18,11 @@ In some components, such as Kafka, time-to-live can be configured in the topic v When message time-to-live has native support in the pub/sub component, Dapr simply forwards the time-to-live configuration without adding any extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component. For example, with Azure Service Bus, where expired messages are stored in the dead letter queue and are not simply deleted. +{{% alert title="Note" color="primary" %}} + You can also set message TTL for a given message broker at creation. Look at the specific characteristic of the component that you are using to see if this is suitable. + +{{% /alert %}} + ### Supported components #### Azure Service Bus @@ -85,6 +90,6 @@ See [this guide]({{< ref pubsub_api.md >}}) for a reference on the pub/sub API. ## Next steps - Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +- Learn [how to configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) - Read the [API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index 262e790b3d7..5b8aa616a3e 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -3,118 +3,134 @@ type: docs title: "Publish and subscribe overview" linkTitle: "Overview" weight: 1000 -description: "Overview of the Pub/Sub API building block" +description: "Overview of the pub/sub API building block" --- -## Introduction +## Publish and subscribe pattern -The [publish/subscribe pattern](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) allows microservices to communicate with each other using messages. The **producer or publisher** sends messages to a **topic** without knowledge of what application will receive them. This involves writing them to an input channel. Similarly, a **consumer or subscriber** subscribes to the topic and receive its messages without any knowledge of what service produced these messages. This involves receiving messages from an output channel. An intermediary message broker is responsible for copying each message from an input channel to an output channels for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another. +The publish and subscribe pattern (pub/sub) enables microservices to communicate with each other using messages for event-driven architectures. -The publish/subscribe API in Dapr provides an at-least-once guarantee and integrates with various message brokers and queuing systems. The specific implementation used by your service is pluggable and configured as a Dapr pub/sub component at runtime. This approach removes the dependency from your service and, as a result, makes your service more portable and flexible to changes. +- The producer, or **publisher**, writes messages to an input channel and sends them to a topic, unaware which application will receive them. +- The consumer, or **subscriber**, subscribes to the topic and receives messages from an output channel, unaware which service produced these messages. -The complete list of Dapr pub/sub components is [here]({{< ref supported-pubsub >}}). +An intermediary message broker copies each message from a publisher's input channel to an output channel for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.

-The Dapr pub/sub building block provides a platform-agnostic API to send and receive messages. Your services publish messages to a named topic and also subscribe to a topic to consume the messages. +## Pub/sub API in Dapr -The service makes a network call to a Dapr pub/sub building block, exposed as a sidecar. This building block then makes calls into a Dapr pub/sub component that encapsulates a specific message broker product. To receive topics, Dapr subscribes to the Dapr pub/sub component on behalf of your service and delivers the messages to an endpoint when they arrive. +The pub/sub API in Dapr: +- Provides a platform-agnostic API to send and receive messages. +- Offers at-least-once message delivery guarantee. +- Integrates with various message brokers and queuing systems. -The diagram below shows an example of a "shipping" service and an "email" service that have both subscribed to topics that are published by the "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component, for example Redis Streams, NATS Streaming, Azure Service Bus, or GCP Pub/Sub. +The specific message broker used by your service is pluggable and configured as a Dapr pub/sub component at runtime. This removes the dependency from your service and makes your service more portable and flexible to changes. + +When using pub/sub in Dapr: + +1. Your service makes a network call to a Dapr pub/sub building block API. +1. The pub/sub building block makes calls into a Dapr pub/sub component that encapsulates a specific message broker. +1. To receive messages on a topic, Dapr subscribes to the pub/sub component on behalf of your service with a topic and delivers the messages to an endpoint on your service when they arrive. + +In the diagram below, a "shipping" service and an "email" service have both subscribed to topics published by a "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component; for example: Redis Streams, NATS Streaming, Azure Service Bus, or GCP pub/sub.

-The diagram below has the same services, however this time showing the Dapr publish API that sends an "order" topic and order endpoints on the subscribing services that these topic messages are delivered posted to by Dapr. +In the diagram below, the Dapr API posts an "order" topic from the publishing "cart" service to "order" endpoints on the "shipping" and "email" subscribing services.

-## Features -The pub/sub building block provides several features to your application. +[View the complete list of pub/sub components that Dapr supports]({{< ref supported-pubsub >}}). -### Cloud Events message format +## Dapr pub/sub API features -To enable message routing and to provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically "wrapped" in a Cloud Events envelope, using `Content-Type` header value for `datacontenttype` attribute. +The pub/sub building block brings several features to your application. -Dapr implements the following Cloud Events fields: +### Sending messages using Cloud Events -* `id` -* `source` -* `specversion` -* `type` -* `datacontenttype` (Optional) +To enable message routing and provide additional context with each message between services, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a Cloud Events envelope, using [`Content-Type` header value]({{< ref "pubsub-overview.md#content-types" >}}) for `datacontenttype` attribute. -The following example shows an XML content in CloudEvent v1.0 serialized as JSON: +For more information, read about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}), or [sending raw messages without CloudEvents]({{< ref pubsub-raw.md >}}). -```json -{ - "specversion" : "1.0", - "type" : "xml.message", - "source" : "https://example.com/message", - "subject" : "Test XML Message", - "id" : "id-1234-5678-9101", - "time" : "2020-09-23T06:23:21Z", - "datacontenttype" : "text/xml", - "data" : "User1user2hi" -} -``` - -### Message subscription +### Communication with applications not using Dapr and CloudEvents -Dapr applications can subscribe to published topics. Dapr allows two methods by which your applications can subscribe to topics: +If one of your applications uses Dapr while another doesn't, you can disable the CloudEvent wrapping for a publisher or subscriber. This allows partial adoption of Dapr pub/sub in applications that cannot adopt Dapr all at once. - - **Declarative**, where a subscription is defined in an external file, - - **Programmatic**, where a subscription is defined in the user code. +For more information, read [how to use pub/sub without CloudEvents]({{< ref pubsub-raw.md >}}). - Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. The programmatic approach implements the subscription in your code. +### Setting message content types - For more information read [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe >}}). +When publishing a message, it's important to specify the content type of the data being sent. Unless specified, Dapr will assume `text/plain`. +- HTTP client: the content type can be set in a `Content-Type` header +- gRPC client and SDK: have a dedicated content type parameter ### Message delivery -In principle, Dapr considers message successfully delivered when the subscriber responds with a non-error response after processing the message. For more granular control, Dapr's publish/subscribe API also provides explicit statuses, defined in the response payload, which the subscriber can use to indicate the specific handling instructions to Dapr (e.g. `RETRY` or `DROP`). For more information on message routing read [Dapr publish/subscribe API documentation]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}}) +In principle, Dapr considers a message successfully delivered once the subscriber processes the message and responds with a non-error response. For more granular control, Dapr's pub/sub API also provides explicit statuses, defined in the response payload, with which the subscriber indicates specific handling instructions to Dapr (for example, `RETRY` or `DROP`). + +### Receiving messages with topic subscriptions + +Dapr applications can subscribe to published topics via two methods that support the same features: declarative and programmatic. + +| Subscription method | Description | +| ------------------- | ----------- | +| **Declarative** | Subscription is defined in an **external file**. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. | +| **Programmatic** | Subscription is defined in the **user code**. The programmatic approach implements the subscription in your code. | + +For more information, read [about the subscriptions in Subscription Methods]({{< ref subscription-methods.md >}}). + +### Message routing + +Dapr provides [content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html) pattern. [Pub/sub routing]({{< ref howto-route-messages.md >}}) is an implementation of this pattern that allows developers to use expressions to route [CloudEvents](https://cloudevents.io) based on their contents to different URIs/paths and event handlers in your application. If no route matches, an optional default route is used. This is useful as your applications expands to support multiple event versions or special cases. + +This feature is available to both the declarative and programmatic subscription approaches. + +For more information on message routing, read [Dapr pub/sub API reference]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}}) ### At-least-once guarantee -Dapr guarantees "At-Least-Once" semantics for message delivery. That means that when an application publishes a message to a topic using the publish/subscribe API, Dapr ensures that this message will be delivered at least once to every subscriber. +Dapr guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the pub/sub API, Dapr ensures the message is delivered *at least once* to every subscriber. ### Consumer groups and competing consumers pattern -The burden of dealing with concepts like consumer groups and multiple application instances using a single consumer group is all handled automatically by Dapr. When multiple instances of the same application (running same app-IDs) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This is commonly known as the competing consumers pattern and is illustrated in the diagram below. +Dapr automatically handles the burden of dealing with concepts like consumer groups and competing consumers pattern. The competing consumers pattern refers to multiple application instances using a single consumer group. When multiple instances of the same application (running same Dapr app ID) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This concept is illustrated in the diagram below.

-Similarly, if two different applications (different app-IDs) subscribe to the same topic, Dapr deliver each message to *only one instance of **each** application*. +Similarly, if two different applications (with different app-IDs) subscribe to the same topic, Dapr delivers each message to *only one instance of **each** application*. -### Topic scoping +### Scoping topics for added security -By default, all topics backing the Dapr pub/sub component (e.g. Kafka, Redis Stream, RabbitMQ) are available to every application configured with that component. To limit which application can publish or subscribe to topics, Dapr provides topic scoping. This enables you to say which topics an application is allowed to publish and which topics an application is allowed to subscribe to. For more information read [publish/subscribe topic scoping]({{< ref pubsub-scopes.md >}}). +By default, all topic messages associated with an instance of a pub/sub component are available to every application configured with that component. You can limit which application can publish or subscribe to topics with Dapr topic scoping. For more information, read: [pub/sub topic scoping]({{< ref pubsub-scopes.md >}}). ### Message Time-to-Live (TTL) -Dapr can set a timeout message on a per message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This is to prevent the build up of messages that are not read. A message that has been in the queue for longer than the configured TTL is said to be dead. For more information read [publish/subscribe message time-to-live]({{< ref pubsub-message-ttl.md >}}). -- Note: Message TTL can also be set for a given queue at the time of component creation. Look at the specific characteristic of the component that you are using. +Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). -### Communication with applications not using Dapr and CloudEvents -For scenarios where one application uses Dapr but another doesn't, CloudEvent wrapping can be disabled for a publisher or subscriber. This allows partial adoption of Dapr pubsub in applications that cannot adopt Dapr all at once. For more information read [how to use pubsub without CloudEvent]({{< ref pubsub-raw.md >}}). +## Try out pub/sub + +### Quickstarts and tutorials + +Want to put the Dapr pub/sub API to the test? Walk through the following quickstart and tutorials to see pub/sub in action: + +| Quickstart/tutorial | Description | +| ------------------- | ----------- | +| [Pub/sub quickstart]({{< ref pubsub-quickstart.md >}}) | Send and receive messages using the publish and subscribe API. | +| [Pub/sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) | Demonstrates how to use Dapr to enable pub-sub applications. Uses Redis as a pub-sub component. | -### Publish/Subscribe API +### Start using pub/sub directly in your app -The publish/subscribe API is located in the [API reference]({{< ref pubsub_api.md >}}). +Want to skip the quickstarts? Not a problem. You can try out the pub/sub building block directly in your application to publish messages and subscribe to a topic. After [Dapr is installed]({{< ref "getting-started/_index.md" >}}), you can begin using the pub/sub API starting with [the pub/sub how-to guide]({{< ref howto-publish-subscribe.md >}}). ## Next steps -* Follow these guides on: - * [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe.md >}}) - * [How-To: Configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -* Try out the [Pub/Sub quickstart sample](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) -* Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -* Learn about [message time-to-live (TTL)]({{< ref pubsub-message-ttl.md >}}) -* Learn about [pubsub without CloudEvent]({{< ref pubsub-raw.md >}}) -* List of [pub/sub components]({{< ref supported-pubsub.md >}}) -* Read the [pub/sub API reference]({{< ref pubsub_api.md >}}) +- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}). +- Follow [How-To: Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md index f4d542f2daf..d5a0fbe61b0 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md @@ -1,32 +1,23 @@ --- type: docs -title: "Pub/Sub without CloudEvents" -linkTitle: "Pub/Sub without CloudEvents" -weight: 7000 -description: "Use Pub/Sub without CloudEvents." +title: "Publishing & subscribing messages without CloudEvents" +linkTitle: "Messages without CloudEvents" +weight: 2200 +description: "Learn when you might not use CloudEvents and how to disable them." --- -## Introduction +When adding Dapr to your application, some services may still need to communicate via pub/sub messages not encapsulated in CloudEvents, due to either compatibility reasons or some apps not using Dapr. These are referred to as "raw" pub/sub messages. Dapr enables apps to [publish and subscribe to raw events]({{< ref "pubsub-cloudevents.md#publishing-raw-messages" >}}) not wrapped in a CloudEvent for compatibility. -Dapr uses CloudEvents to provide additional context to the event payload, enabling features like: -* Tracing -* Deduplication by message Id -* Content-type for proper deserialization of event's data +## Publishing raw messages -For more information about CloudEvents, read the [CloudEvents specification](https://github.com/cloudevents/spec). +Dapr apps are able to publish raw events to pub/sub topics without CloudEvent encapsulation, for compatibility with non-Dapr apps. -When adding Dapr to your application, some services may still need to communicate via raw pub/sub messages not encapsulated in CloudEvents. This may be for compatibility reasons, or because some apps are not using Dapr. Dapr enables apps to publish and subscribe to raw events that are not wrapped in a CloudEvent. +Diagram showing how to publish with Dapr when subscriber does not use Dapr or CloudEvent {{% alert title="Warning" color="warning" %}} Not using CloudEvents disables support for tracing, event deduplication per messageId, content-type metadata, and any other features built using the CloudEvent schema. {{% /alert %}} -## Publishing raw messages - -Dapr apps are able to publish raw events to pub/sub topics without CloudEvent encapsulation, for compatibility with non-Dapr apps. - -Diagram showing how to publish with Dapr when subscriber does not use Dapr or CloudEvent - To disable CloudEvent wrapping, set the `rawPayload` metadata to `true` as part of the publishing request. This allows subscribers to receive these messages without having to parse the CloudEvent schema. {{< tabs curl "Python SDK" "PHP SDK">}} @@ -147,7 +138,7 @@ $app->start(); ## Declaratively subscribe to raw events -Similarly, you can subscribe to raw events declaratively by adding the `rawPayload` metadata entry to your Subscription Custom Resource Definition (CRD): +Similarly, you can subscribe to raw events declaratively by adding the `rawPayload` metadata entry to your subscription specification. ```yaml apiVersion: dapr.io/v1alpha1 @@ -167,6 +158,6 @@ scopes: ## Next steps -- Learn more about [how to publish and subscribe]({{< ref howto-publish-subscribe.md >}}) +- Learn more about [publishing and subscribing messages]({{< ref pubsub-overview.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md index 34ffa9d8879..df45ad8f376 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md @@ -1,9 +1,9 @@ --- type: docs -title: "Scope Pub/Sub topic access" +title: "Scope Pub/sub topic access" linkTitle: "Scope topic access" weight: 5000 -description: "Use scopes to limit Pub/Sub topics to specific applications" +description: "Use scopes to limit pub/sub topics to specific applications" --- ## Introduction @@ -164,7 +164,7 @@ The table below shows which application is allowed to subscribe to the topics: ## Next steps -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +- Learn [how to configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) - Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) - Read the [API reference]({{< ref pubsub_api.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md new file mode 100644 index 00000000000..7c4d240195e --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md @@ -0,0 +1,353 @@ +--- +type: docs +title: "Declarative and programmatic subscription methods" +linkTitle: "Subscription methods" +weight: 3000 +description: "Learn more about the two methods by which Dapr allows you to subscribe to topics." +--- + +## Pub/sub API subscription methods + +Dapr applications can subscribe to published topics via two methods that support the same features: declarative and programmatic. + +| Subscription method | Description | +| ------------------- | ----------- | +| [**Declarative**]({{< ref "subscription-methods.md#declarative-subscriptions" >}}) | Subscription is defined in an **external file**. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. | +| [**Programmatic**]({{< ref "subscription-methods.md#programmatic-subscriptions" >}}) | Subscription is defined in the **user code**. The programmatic approach implements the subscription in your code. | + +The examples below demonstrate pub/sub messaging between a `checkout` app and an `orderprocessing` app via the `orders` topic. The examples demonstrate the same Dapr pub/sub component used first declaratively, then programmatically. + +### Declarative subscriptions + +You can subscribe declaratively to a topic using an external component file. This example uses a YAML component file named `subscription.yaml`: + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Subscription +metadata: + name: order_pub_sub +spec: + topic: orders + route: /checkout + pubsubname: order_pub_sub +scopes: +- orderprocessing +- checkout +``` + +Notice, the pub/sub component `order_pub_sub` subscribes to topic `orders`. +- The `route` field tells Dapr to send all topic messages to the `/checkout` endpoint in the app. +- The `scopes` field enables this subscription for apps with IDs `orderprocessing` and `checkout`. + +When running Dapr, call out the YAML component file path to point Dapr to the component. + +{{< tabs ".NET" Java Python JavaScript Go Kubernetes>}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- dotnet run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- python3 app.py +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- npm start +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- go run app.go +``` + +{{% /codetab %}} + +{{% codetab %}} + +In Kubernetes, apply the component to the cluster: + +```bash +kubectl apply -f subscription.yaml +``` + +{{% /codetab %}} + +{{< /tabs >}} + +In your application code, subscribe to the topic specified in the Dapr pub/sub component. + +{{< tabs ".NET" Java Python JavaScript Go >}} + +{{% codetab %}} + +```csharp + //Subscribe to a topic +[Topic("order_pub_sub", "orders")] +[HttpPost("checkout")] +public void getCheckout([FromBody] int orderId) +{ + Console.WriteLine("Subscriber received : " + orderId); +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```java + //Subscribe to a topic +@Topic(name = "orders", pubsubName = "order_pub_sub") +@PostMapping(path = "/checkout") +public Mono getCheckout(@RequestBody(required = false) CloudEvent cloudEvent) { + return Mono.fromRunnable(() -> { + try { + log.info("Subscriber received: " + cloudEvent.getData()); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```python +#Subscribe to a topic +@app.subscribe(pubsub_name='order_pub_sub', topic='orders') +def mytopic(event: v1.Event) -> None: + data = json.loads(event.Data()) + logging.info('Subscriber received: ' + str(data)) + +app.run(6002) +``` + +{{% /codetab %}} + +{{% codetab %}} + +```javascript +//Subscribe to a topic +await server.pubsub.subscribe("order_pub_sub", "orders", async (orderId) => { + console.log(`Subscriber received: ${JSON.stringify(orderId)}`) +}); +await server.startServer(); +``` + +{{% /codetab %}} + +{{% codetab %}} + +```go +//Subscribe to a topic +if err := s.AddTopicEventHandler(sub, eventHandler); err != nil { + log.Fatalf("error adding topic subscription: %v", err) +} +if err := s.Start(); err != nil && err != http.ErrServerClosed { + log.Fatalf("error listenning: %v", err) +} + +func eventHandler(ctx context.Context, e *common.TopicEvent) (retry bool, err error) { + log.Printf("Subscriber received: %s", e.Data) + return false, nil +} +``` + +{{% /codetab %}} + +{{< /tabs >}} + +The `/checkout` endpoint matches the `route` defined in the subscriptions and this is where Dapr will send all topic messages to. + +### Programmatic subscriptions + +The programmatic approach returns the `routes` JSON structure within the code, unlike the declarative approach's `route` YAML structure. In the example below, we define the values found in the [declarative YAML subscription](#declarative-subscriptions) above within the application code. + +{{< tabs ".NET" Java Python JavaScript Go>}} + +{{% codetab %}} + +```csharp +[Topic("order_pub_sub", "checkout", event.type ==\"order\"")] +[HttpPost("orders")] +public async Task> HandleCheckout(Checkout checkout, [FromServices] DaprClient daprClient) +{ + // Logic + return stock; +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```java + +``` + +{{% /codetab %}} + +{{% codetab %}} + +```python +@app.route('/dapr/subscribe', methods=['GET']) +def subscribe(): + subscriptions = [ + { + 'pubsubname': 'order_pub_sub', + 'topic': 'checkout', + 'routes': { + 'rules': [ + { + 'match': 'event.type == "order"', + 'path': '/orders' + }, + ], + 'default': '/orders' + } + }] + return jsonify(subscriptions) + +@app.route('/orders', methods=['POST']) +def ds_subscriber(): + print(request.json, flush=True) + return json.dumps({'success':True}), 200, {'ContentType':'application/json'} +app.run() +``` + +{{% /codetab %}} + +{{% codetab %}} + +```javascript +const express = require('express') +const bodyParser = require('body-parser') +const app = express() +app.use(bodyParser.json({ type: 'application/*+json' })); + +const port = 3000 + +app.get('/dapr/subscribe', (req, res) => { + res.json([ + { + pubsubname: "order_pub_sub", + topic: "checkout", + routes: { + rules: [ + { + match: 'event.type == "order"', + path: '/orders' + }, + ], + default: '/products' + } + } + ]); +}) + +app.post('/orders', (req, res) => { + console.log(req.body); + res.sendStatus(200); +}); + +app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) +``` + +{{% /codetab %}} + +{{% codetab %}} + +```go +package main + + "encoding/json" + "fmt" + "log" + "net/http" + + "github.com/gorilla/mux" +) + +const appPort = 3000 + +type subscription struct { + PubsubName string `json:"pubsubname"` + Topic string `json:"topic"` + Metadata map[string]string `json:"metadata,omitempty"` + Routes routes `json:"routes"` +} + +type routes struct { + Rules []rule `json:"rules,omitempty"` + Default string `json:"default,omitempty"` +} + +type rule struct { + Match string `json:"match"` + Path string `json:"path"` +} + +// This handles /dapr/subscribe +func configureSubscribeHandler(w http.ResponseWriter, _ *http.Request) { + t := []subscription{ + { + PubsubName: "order_pub_sub", + Topic: "checkout", + Routes: routes{ + Rules: []rule{ + { + Match: `event.type == "order"`, + Path: "/orders", + }, + }, + Default: "/orders", + }, + }, + } + + w.WriteHeader(http.StatusOK) + json.NewEncoder(w).Encode(t) +} + +func main() { + router := mux.NewRouter().StrictSlash(true) + router.HandleFunc("/dapr/subscribe", configureSubscribeHandler).Methods("GET") + log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", appPort), router)) +} +``` +{{% /codetab %}} + +{{< /tabs >}} + +## Next Steps + +* Try out the [pub/sub Quickstart]({{< ref pubsub-quickstart.md >}}) +* Follow: [How-To: Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +* Learn more about [declarative and programmatic subscription methods]({{< ref subscription-methods >}}). +* Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) +* Learn about [message TTL]({{< ref pubsub-message-ttl.md >}}) +* Learn more about [pub/sub with and without CloudEvent]({{< ref pubsub-cloudevents.md >}}) +* List of [pub/sub components]({{< ref supported-pubsub.md >}}) +* Read the [pub/sub API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/reference/api/pubsub_api.md b/daprdocs/content/en/reference/api/pubsub_api.md index a09ea6a8e37..f4d8e47fa3a 100644 --- a/daprdocs/content/en/reference/api/pubsub_api.md +++ b/daprdocs/content/en/reference/api/pubsub_api.md @@ -163,7 +163,7 @@ other | warning is logged and message to be retried ## Message envelope -Dapr Pub/Sub adheres to version 1.0 of CloudEvents. +Dapr pub/sub adheres to version 1.0 of CloudEvents. ## Related links diff --git a/daprdocs/static/images/pubsub-overview-components.png b/daprdocs/static/images/pubsub-overview-components.png index fb95c647dc2..cc05cec2c9c 100644 Binary files a/daprdocs/static/images/pubsub-overview-components.png and b/daprdocs/static/images/pubsub-overview-components.png differ