diff --git a/daprdocs/content/en/concepts/components-concept.md b/daprdocs/content/en/concepts/components-concept.md index 16fb31bf6aa..d6e5532cb73 100644 --- a/daprdocs/content/en/concepts/components-concept.md +++ b/daprdocs/content/en/concepts/components-concept.md @@ -12,53 +12,61 @@ Dapr uses a modular design where functionality is delivered as a component. Each You can get a list of current components available in the hosting environment using the `dapr components` CLI command. +## Component specification + +Each component has a specification (or spec) that it conforms to. Components are configured at design-time with a YAML file which is stored in either a `components/local` folder within your solution, or globally in the `.dapr` folder created when invoking `dapr init`. These YAML files adhere to the generic [Dapr component schema]({{}}), but each is specific to the component specification. + +It is important to understand that the component spec values, particularly the spec `metadata`, can change between components of the same component type, for example between different state stores, and that some design-time spec values can be overridden at runtime when making requests to a component's API. As a result, it is strongly recommended to review a [component's specs]({{}}), paying particular attention to the sample payloads for requests to set the metadata used to interact with the component. + +## Available component types + The following are the component types provided by Dapr: -## State stores +### State stores State store components are data stores (databases, files, memory) that store key-value pairs as part of the [state management]({{< ref "state-management-overview.md" >}}) building block. - [List of state stores]({{< ref supported-state-stores >}}) - [State store implementations](https://github.com/dapr/components-contrib/tree/master/state) -## Name resolution +### Name resolution Name resolution components are used with the [service invocation]({{}}) building block to integrate with the hosting environment and provide service-to-service discovery. For example, the Kubernetes name resolution component integrates with the Kubernetes DNS service, self-hosted uses mDNS and clusters of VMs can use the Consul name resolution component. - [List of name resolution components]({{< ref supported-name-resolution >}}) - [Name resolution implementations](https://github.com/dapr/components-contrib/tree/master/nameresolution) -## Pub/sub brokers +### Pub/sub brokers Pub/sub broker components are message brokers that can pass messages to/from services as part of the [publish & subscribe]({{< ref pubsub-overview.md >}}) building block. - [List of pub/sub brokers]({{< ref supported-pubsub >}}) - [Pub/sub broker implementations](https://github.com/dapr/components-contrib/tree/master/pubsub) -## Bindings +### Bindings External resources can connect to Dapr in order to trigger a method on an application or be called from an application as part of the [bindings]({{< ref bindings-overview.md >}}) building block. - [List of supported bindings]({{< ref supported-bindings >}}) - [Binding implementations](https://github.com/dapr/components-contrib/tree/master/bindings) -## Secret stores +### Secret stores A [secret]({{}}) is any piece of private information that you want to guard against unwanted access. Secrets stores are used to store secrets that can be retrieved and used in applications. - [List of supported secret stores]({{< ref supported-secret-stores >}}) - [Secret store implementations](https://github.com/dapr/components-contrib/tree/master/secretstores) -## Configuration stores +### Configuration stores Configuration stores are used to save application data, which can then be read by application instances on startup or notified of when changes occur. This allows for dynamic configuration. - [List of supported configuration stores]({{< ref supported-configuration-stores >}}) - [Configuration store implementations](https://github.com/dapr/components-contrib/tree/master/configuration) -## Middleware +### Middleware -Dapr allows custom [middleware]({{}}) to be plugged into the HTTP request processing pipeline. Middleware can perform additional actions on an HTTP request, such as authentication, encryption and message transformation before the request is routed to the user code, or before the request is returned to the client. The middleware components are used with the [service invocation]({{}}) building block. +Dapr allows custom [middleware]({{}}) to be plugged into the HTTP request processing pipeline. Middleware can perform additional actions on an HTTP request, such as authentication, encryption and message transformation before the request is routed to the user code, or before the response is returned to the client. The middleware components are used with the [service invocation]({{}}) building block. - [List of supported middleware components]({{< ref supported-middleware >}}) - [Middleware implementations](https://github.com/dapr/components-contrib/tree/master/middleware) \ No newline at end of file diff --git a/daprdocs/content/en/concepts/dapr-services/sidecar.md b/daprdocs/content/en/concepts/dapr-services/sidecar.md index aaaf4edade8..cc082cf6067 100644 --- a/daprdocs/content/en/concepts/dapr-services/sidecar.md +++ b/daprdocs/content/en/concepts/dapr-services/sidecar.md @@ -6,8 +6,13 @@ weight: 100 description: "Overview of the Dapr sidecar process" --- -Dapr uses a [sidecar pattern]({{< ref "concepts/overview.md#sidecar-architecture" >}}), meaning the Dapr APIs are run and exposed on a separate process (i.e. the Dapr sidecar) running alongside your application. The Dapr sidecar process is named `daprd` and is launched in different ways depending on the hosting environment. +Dapr uses a [sidecar pattern]({{< ref "concepts/overview.md#sidecar-architecture" >}}), meaning the Dapr APIs are run and exposed on a separate process, the Dapr sidecar, running alongside your application. The Dapr sidecar process is named `daprd` and is launched in different ways depending on the hosting environment. +The Dapr sidecar exposes [building block APIs]({{}}) used by your application business logic, a [metadata API]({{}}) for discoverability of capabiliites and to set attributes and a [health API]({{}}) to determine health status. + + + +The sidecar APIs are called from your application over local http or gRPC endpoints. ## Self-hosted with `dapr run` @@ -20,13 +25,13 @@ On [Kubernetes]({{< ref kubernetes.md >}}), the Dapr control plane includes the ## Running the sidecar directly -In most cases you do not need to run `daprd` explicitly, as the sidecar is either launched by the CLI (self-hosted mode) or by the dapr-sidecar-injector service (Kubernetes). For advanced use cases (debugging, scripted deployments, etc.) the `daprd` process can be launched directly. +In most cases you do not need to run `daprd` explicitly, as the sidecar is either launched by the [CLI]({{}}) (self-hosted mode) or by the dapr-sidecar-injector service (Kubernetes). For advanced use cases (debugging, scripted deployments, etc.) the `daprd` process can be launched directly. For a detailed list of all available arguments run `daprd --help` or see this [table]({{< ref arguments-annotations-overview.md >}}) which outlines how the `daprd` arguments relate to the CLI arguments and Kubernetes annotations. ### Examples -1. Start a sidecar along an application by specifying its unique ID. Note `--app-id` is a required field: +1. Start a sidecar along with an application by specifying its unique ID. Note `--app-id` is a required field: ```bash daprd --app-id myapp diff --git a/daprdocs/content/en/concepts/security-concept.md b/daprdocs/content/en/concepts/security-concept.md index 8f3fb26e134..ac9574dbabd 100644 --- a/daprdocs/content/en/concepts/security-concept.md +++ b/daprdocs/content/en/concepts/security-concept.md @@ -6,9 +6,9 @@ weight: 600 description: How Dapr is designed with security in mind --- -Security is fundamental to Dapr. This article describes the security features and capabiliies when using Dapr in a distributed application. These can be divided into: +Security is fundamental to Dapr. This article describes the security features and capabilities when using Dapr in a distributed application. These can be divided into: -- Secure communication with service invocation and pub/sub APIs +- Secure communication with service invocation and pub/sub APIs. - Security policies on components and applied through configuration. - Operational security practices. - State security, focusing on data at rest. @@ -16,151 +16,196 @@ Security is fundamental to Dapr. This article describes the security features an An example application is used to illustrate many of the security features available in Dapr. # Secure communication -Dapr has end-to-end security with the service invocation API, the ability to authentication an application with Dapr and the to set endpoint access policies. This is shown in the diagram below. + +Dapr provides end-to-end security with the service invocation API, with the ability to authenticate an application with Dapr and set endpoint access policies. This is shown in the diagram below. ## Service invocation scoping access policy -Dapr applications can be scoped to namespaces for deployment and security and you can call between services deployed to different namespaces. Read the [Service invocation across namespaces]({{< ref "service-invocation-namespaces.md" >}}) for more details. -Dapr applications can restrict which operations can be called, including which applications are allowed (or denied) to call it. Read [How-To: Apply access control list configuration for service invocation]({{}}) for more details. +Dapr applications can be scoped to namespaces for deployment and security. You can call between services deployed to different namespaces. Read the [Service invocation across namespaces]({{< ref "service-invocation-namespaces.md" >}}) article for more details. + +Dapr applications can restrict which operations can be called, including which applications are allowed (or denied) to call it. Read [How-To: Apply access control list configuration for service invocation]({{< ref invoke-allowlist.md >}}) for more details. + +## Pub/sub topic scoping access policy -## Pub/Sub topic scoping access policy -For pub/sub components you can limit which topic types and which applications are allowed to published and subscribed to specific topics. Read [Scope Pub/Sub topic access]({{< ref "pubsub-scopes.md" >}}) for more details. +For pub/sub components, you can limit which topic types and applications are allowed to publish and subscribe to specific topics. Read [Scope Pub/Sub topic access]({{< ref "pubsub-scopes.md" >}}) for more details. ## Encryption of data using mTLS -One of the security mechanisms that Dapr employs for encrypting data in transit is [mutual authentication TLS](https://en.wikipedia.org/wiki/Mutual_authentication) or mTLS. mTLS offers a few key features for network traffic inside your application: -- Two way authentication, the client proving its identity to the server, and vice-versa -- An encrypted channel for all in-flight communication, after two-way authentication is established +One of Dapr's security mechanisms for encrypting data in transit is [mutual authentication TLS](https://en.wikipedia.org/wiki/Mutual_authentication) or mTLS. mTLS offers a few key features for network traffic inside your application: -Mutual TLS is useful in almost all scenarios, but especially so for systems subject to regulations such as [HIPAA](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act) and [PCI](https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard). +- **Two way authentication**, with the client proving its identity to the server, and vice-versa. +- **An encrypted channel** for all in-flight communication, after two-way authentication is established. + +mTLS is useful in almost all scenarios, but especially for systems subject to regulations such as [HIPAA](https://en.wikipedia.org/wiki/Health_Insurance_Portability_and_Accountability_Act) and [PCI](https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard). ## Secure Dapr to Dapr communication -Dapr enables mTLS with no extra code or complex configuration inside your production systems. Equally Dapr side cars prevent all IP addresses by default other than localhost from calling it, unless explicitly listed. -Dapr includes an "on by default", automatic mutual TLS that provides in-transit encryption for traffic between Dapr sidecars. To achieve this, Dapr leverages a system service named `Sentry` which acts as a Certificate Authority (CA)/Identity Provider and signs workload (app) certificate requests originating from the Dapr sidecar. +Dapr enables mTLS with no extra code or complex configuration inside your production systems. Equally, Dapr sidecars prevent all IP addresses by default other than `localhost` from calling it, unless explicitly listed. + +Dapr includes an "on by default", automatic mTLS that provides in-transit encryption for traffic between Dapr sidecars. To achieve this, Dapr leverages a system service named `Sentry`, which acts as a Certificate Authority (CA)/Identity Provider and signs workload (app) certificate requests originating from the Dapr sidecar. -By default, a workload cert is valid for 24 hours and the clock skew is set to 15 minutes. +By default, a workload certificate is valid for 24 hours and the clock skew is set to 15 minutes. -The Sentry service, automatically creates and persists self-signed root certificates valid for one year, unless existing root certs have been provided by the user. Dapr manages workload certificate rotation, and if you bring your own certificates, does so with zero downtime to the application. +Unless you've provided existing root certificates, the Sentry service automatically creates and persists self-signed root certificates valid for one year. Dapr manages workload certificate rotation; if you bring your own certificates, Dapr does so with zero downtime to the application. -When root certs are replaced (secret in Kubernetes mode and filesystem for self-hosted mode), the Sentry picks them up and rebuilds the trust chain without needing to restart, with zero downtime to Sentry. +When root certificates are replaced (secret in Kubernetes mode and filesystem for self-hosted mode), Sentry picks them up and rebuilds the trust chain, without restart and with zero downtime to Sentry. -When a new Dapr sidecar initializes, it first checks if mTLS is enabled. If it is, an ECDSA private key and certificate signing request are generated and sent to Sentry via a gRPC interface. The communication between the Dapr sidecar and Sentry is authenticated using the trust chain cert, which is injected into each Dapr instance by the Dapr Sidecar Injector system service. +When a new Dapr sidecar initializes, it checks if mTLS is enabled. If so, an ECDSA private key and certificate signing request are generated and sent to Sentry via a gRPC interface. The communication between the Dapr sidecar and Sentry is authenticated using the trust chain certificate, which is injected into each Dapr instance by the Dapr Sidecar Injector system service. ### Configuring mTLS -Mutual TLS can be turned off/on by editing the default configuration that is deployed with Dapr via the `spec.mtls.enabled` field. -This can be done for both Kubernetes and self-hosted modes. Details for how to do this can be found [here]({{< ref mtls.md >}}). +mTLS can be turned on/off by editing the default configuration deployed with Dapr via the `spec.mtls.enabled` field. + +[You can do this for both Kubernetes and self-hosted modes]({{< ref mtls.md >}}). #### mTLS in self hosted mode -The diagram below shows how the Sentry system service issues certificates for applications based on the root/issuer certificate that is provided by an operator or generated by the Sentry service as stored in a file. + +The diagram below shows how the Sentry system service issues certificates for applications based on the root/issuer certificate provided by an operator or generated by the Sentry service as stored in a file. #### mTLS in Kubernetes mode -In a Kubernetes cluster, the secret that holds the root certificates is scoped to the namespace in which the Dapr components are deployed and is only accessible by the Dapr control plane system pods. -Dapr also supports strong identities when deployed on Kubernetes, relying on a pod's Service Account token which is sent as part of the certificate signing request (CSR) to Sentry. +In a Kubernetes cluster, the secret that holds the root certificates is: + +- Scoped to the namespace in which the Dapr components are deployed. +- Only accessible by the Dapr control plane system pods. + +Dapr also supports strong identities when deployed on Kubernetes, relying on a pod's Service Account token sent as part of the certificate signing request (CSR) to Sentry. -The diagram below shows how the Sentry system service issues certificates for applications based on the root/issuer certificate that is provided by an operator or generated by the Sentry service and stored as a Kubernetes secret +The diagram below shows how the Sentry system service issues certificates for applications based on the root/issuer certificate provided by an operator or generated by the Sentry service and stored as a Kubernetes secret ### Preventing IP addresses on Dapr -To prevent Dapr side cars from being called on any IP address especially in production environments such as Kubernetes, Dapr restricts its listening IP addresses to only local host. Before the v1.4 release any Dapr sidecar could call any other sidecar in a cluster by default. This is no longer possible and needs to be enabled explicitly. Use the [dapr-listen-addresses](https://docs.dapr.io/reference/arguments-annotations-overview/) setting if there are other addresses you need to enable. + +To prevent Dapr sidecars from being called on any IP address (especially in production environments such as Kubernetes), Dapr restricts its listening IP addresses only to `localhost`. Use the [dapr-listen-addresses](https://docs.dapr.io/reference/arguments-annotations-overview/) setting you need to enable other addresses. ## Secure Dapr to application communication -The Dapr sidecar runs close to the application through **localhost**, and is recommended to run under the same network boundary as the app. While many cloud-native systems today consider the pod level (on Kubernetes, for example) as a trusted security boundary, Dapr provides the app with API level authentication using tokens. This feature guarantees that even on localhost, only an authenticated application may call into Dapr and equally an application can check that Dpar is calling it back. For more details on configuring API token security read, + +The Dapr sidecar runs close to the application through `localhost`, and is recommended to run under the same network boundary as the app. While many cloud-native systems today consider the pod level (on Kubernetes, for example) as a trusted security boundary, Dapr provides the app with API level authentication using tokens. This feature guarantees that, even on `localhost`: + +- Only an authenticated application may call into Dapr +- An application can check that Dapr is calling it back + +For more details on configuring API token security, read: - [Using an API token to authentication requests from an application to Dapr]({{< ref api-token.md >}}). - [Using an API token to authentication requests from Dapr to the application]({{< ref app-api-token.md >}}) ## Secure Dapr to control plane communication -In addition to automatic mTLS between Dapr sidecars, Dapr offers mandatory mTLS between the Dapr sidecar and the Dapr control plane system services, namely the Sentry service (a Certificate Authority), the Placement service (actor placement) and the Kubernetes Operator service. +In addition to automatic mTLS between Dapr sidecars, Dapr offers mandatory mTLS between: -When mTLS is enabled, Sentry writes the root and issuer certificates to a Kubernetes secret that is scoped to the namespace where the control plane is installed. In self-hosted mode, Sentry writes the certificates to a configurable file system path. +- The Dapr sidecar +- The Dapr control plane system services, namely: + - The Sentry service (a Certificate Authority) + - The Placement service (actor placement) + - The Kubernetes Operator service -In Kubernetes, when Dapr system services start, they automatically mount the secret containing the root and issuer certs and use those to secure the gRPC server that is used by the Dapr sidecar. +When mTLS is enabled, Sentry writes the root and issuer certificates to a Kubernetes secret scoped to the namespace where the control plane is installed. In self-hosted mode, Sentry writes the certificates to a configurable file system path. -In self-hosted mode, each system service can be mounted to a filesystem path to get the credentials. +In Kubernetes, when Dapr system services start, they automatically mount and use the secret containing the root and issuer certs to secure the gRPC server used by the Dapr sidecar. In self-hosted mode, each system service can be mounted to a filesystem path to get the credentials. When the Dapr sidecar initializes, it authenticates with the system pods using the mounted leaf certificates and issuer private key. These are mounted as environment variables on the sidecar container. ### mTLS to system services in Kubernetes -The diagram below shows secure communication between the Dapr sidecar and the Dapr Sentry (Certificate Authority), Placement (actor placement) and the Kubernetes Operator system services + +The diagram below shows secure communication between the Dapr sidecar and the Dapr Sentry (Certificate Authority), Placement (actor placement), and the Kubernetes Operator system services.
+# Operational Security -# Operational Security -Dapr is designed for operators to manage mTLS certificate and enforce OAuth policies. +Dapr is designed for operators to manage mTLS certificates and enforce OAuth policies. ## mTLS Certificate deployment and rotation -Dapr allows operators and developers to bring in their own certificates, or instead let Dapr automatically create and persist self-signed root and issuer certificates. Read [Setup & configure mTLS certificates]({{}}) for more details. + +While operators and developers can bring their own certificates into Dapr, Dapr automatically creates and persists self-signed root and issuer certificates. Read [Setup & configure mTLS certificates]({{< ref mtls.md >}}) for more details. ## Middleware endpoint authorization with OAuth -Dapr OAuth 2.0 middleware allows you to enable OAuth authorization on Dapr endpoints for your APIs. Read [Configure endpoint authorization with OAuth]({{}}) for details. Dapr has other middleware components that can be used for OpenID Connect and OPA Policies which you can [read about]({{}}) for more details. + +With Dapr OAuth 2.0 middleware, you can enable OAuth authorization on Dapr endpoints for your APIs. Read [Configure endpoint authorization with OAuth]({{< ref oauth.md >}}) for details. Dapr has other middleware components that you can use for OpenID Connect and OPA Policies. For more details, [read about supported middleware]({{< ref supported-middleware.md >}}). ## Network security -You can adopt common network security technologies such as network security groups (NSGs), demilitarized zones (DMZs) and firewalls to provide layers of protection over your networked resources. For example, unless configured to talk to an external binding target, Dapr sidecars don’t open connections to the internet. And most binding implementations use outbound connections only. You can design your firewall rules to allow outbound connections only through designated ports. + +You can adopt common network security technologies, such as network security groups (NSGs), demilitarized zones (DMZs), and firewalls, to provide layers of protection over your networked resources. For example, unless configured to talk to an external binding target, Dapr sidecars don’t open connections to the internet and most binding implementations only use outbound connections. You can design your firewall rules to allow outbound connections only through designated ports. # Security policies -Dapr has an extensive set of security policies that can be applied to your applications to scope what they are able to do either through a policy setting in the sidecar configuration or with the component specification. + +Dapr has an extensive set of security policies you can apply to your applications. You can scope what they are able to do, either through a policy setting in the sidecar configuration, or with the component specification. ## API access policy -In certain scenarios such as zero trust networks or when exposing the Dapr sidecar to external traffic through a frontend, it’s recommended to only enable the Dapr sidecar APIs that are being used by the app. Doing so reduces the attack surface and helps keep the Dapr APIs scoped to the actual needs of the application. You can control which APIs are accessible to the application by setting an API allow list in configuration, as shown in the diagram below. + +In certain scenarios, such as with zero trust networks or when exposing the Dapr sidecar to external traffic through a frontend, it’s recommended to only enable the Dapr sidecar APIs currently used by the app. This reduces the attack surface and keeps the Dapr APIs scoped to the actual needs of the application. You can control which APIs are accessible to the application by setting an API allow list in configuration, as shown in the diagram below. -Read [How-To: Selectively enable Dapr APIs on the Dapr sidecar]({{}}) for more details. +Read [How-To: Selectively enable Dapr APIs on the Dapr sidecar]({{< ref api-allowlist.md >}}) for more details. ## Secret scoping access policy -To limit the secrets to which the Dapr application has access to, you can define secret scopes by adding a secret scope policy to the application configuration with restrictive permissions. Read [How To: Use secret scoping]({{}}) for more details. -## Component application scoping access policy and secret usage -Dapr components can be namespaced. That means a Dapr sidecar instance can only access the components that have been deployed to the same namespace. Read [How-To: Scope components to one or more applications using namespaces]({{}}) for more details. +To limit the Dapr application's access to secrets, you can define secret scopes. Add a secret scope policy to the application configuration with restrictive permissions. Read [How To: Use secret scoping]({{< ref secret-scope.md >}}) for more details. + +## Component application scoping access policy and secret usage -Dapr provides application-level scoping for components by allowing you to specify which applications can consume specific components and deny others. Read [restricting application access to components with scopes]({{}}) for more details. +Dapr components can be namespaced. That means a Dapr sidecar instance can only access the components that have been deployed to the same namespace. Read [How-To: Scope components to one or more applications using namespaces]({{< ref component-scopes.md >}}) for more details. -Dapr components can use Dapr's built-in secret management capability to manage secrets. Read the [secret store overview]({{}}) and [How-To: Reference secrets in components]({{}}) for more details. +Dapr provides application-level scoping for components by allowing you to specify which applications can consume specific components and deny others. Read [restricting application access to components with scopes]({{< ref "component-scopes.md#application-access-to-components-with-scopes" >}}) for more details. + +Dapr components can use Dapr's built-in secret management capability to manage secrets. Read the [secret store overview]({{< ref secrets-overview.md >}}) and [How-To: Reference secrets in components]({{< ref component-secrets.md >}}) for more details. ## Bindings security + Authentication with a binding target is configured by the binding’s configuration file. Generally, you should configure the minimum required access rights. For example, if you only read from a binding target, you should configure the binding to use an account with read-only access rights. # State security -## State store encrytion at rest -By default Dapr doesn't transform the state data from applications. This means Dapr doesn't attempt to encrypt/decrypt state data and your application can adopt encryption/decryption methods of your choice, where the state data remains opaque to Dapr. Dapr components can use a configured authentication method to authenticate with the underlying state store. Many state store implementations use official client libraries that generally use secured communication channels with the servers. +## State store encryption at rest -However application state often needs to get encrypted at rest to provide stronger security in enterprise workloads or regulated environments and Dapr does provide automatic client side state encryption based on AES256. Read [How-To: Encrypt application state]({{< ref howto-encrypt-state.md >}}) for more details. +By default Dapr doesn't transform the state data from applications. This means: + +- Dapr doesn't attempt to encrypt/decrypt state data +- Your application can adopt encryption/decryption methods of your choice, where the state data remains opaque to Dapr. + +Dapr components can use a configured authentication method to authenticate with the underlying state store. Many state store implementations use official client libraries that generally use secured communication channels with the servers. + +However, application state often needs to get encrypted at rest to provide stronger security in enterprise workloads or regulated environments. Dapr provides automatic client-side state encryption based on AES256. Read [How-To: Encrypt application state]({{< ref howto-encrypt-state.md >}}) for more details. ## Dapr Runtime state -Importantly the Dapr runtime does not store any data at rest, meaning that Dapr runtime has no dependency on any state stores for its operation and can be considered stateless. -# Using security capabilies in an example application -The diagram below shows a number of the security capabilities put together into an example application hosted on Kubernetes. You can see that the Dapr control plane, the Redis state store and each of the services have been deployed to their own namespaces. Also when deploying on Kubernetes, you can use regular Kubernetes RBAC to control access to management activities. +The Dapr runtime does not store any data at rest, meaning that Dapr runtime has no dependency on any state stores for its operation and can be considered stateless. + +# Using security capabilities in an example application -In the application, requests are received by the ingress reverse-proxy which has a Dapr sidecar running next to it. From the reverse proxy, Dapr uses service invocation to call onto service A which then publishes a message to Serivce B. Service B retrieves a secret in order to read and save state to a Redis state store. +The diagram below shows a number of security capabilities placed in an example application hosted on Kubernetes. In the example, the Dapr control plane, the Redis state store, and each of the services have been deployed to their own namespaces. When deploying on Kubernetes, you can use regular Kubernetes RBAC to control access to management activities. + +In the application, requests are received by the ingress reverse-proxy which has a Dapr sidecar running next to it. From the reverse proxy, Dapr uses service invocation to call onto Service A, which then publishes a message to Service B. Service B retrieves a secret in order to read and save state to a Redis state store. Let's go over each of the security capabilities and describe how they are protecting this application. -1. API Token authentication is used to ensure the the reverse-proxy knows it is communication with the correct Dapr sidecar instance. This prevents it forwarding message to anything expect this Dapr side car. -2. Service invocation mTLS is used for authentication between the reverse proxy and Service A and a in addition service access policy configured on Service A restricts it to only receive calls on a specific endpoint from the reverse proxy and no other services. -3. Service B uses a pub/sub topic security policy to indicate that it can only receive messages published from Service A. +1. API Token authentication ensures the reverse-proxy knows it is communicating with the correct Dapr sidecar instance. This prevents forwarding messages to anything except this Dapr sidecar. +2. Service invocation mTLS is used for authentication between the reverse proxy and Service A. A service access policy configured on Service A restricts it to only receive calls on a specific endpoint from the reverse proxy and no other services. +3. Service B uses a pub/sub topic security policy to indicate it can only receive messages published from Service A. 4. The Redis component definition uses a component scoping security policy to say only Service B is allowed to call it. -5. Service B restricts the Dapr sidecar to only use the pub/sub, state management and secrets APIs. All other API calls, for example service invocation, would fail. -6. A secrets security policy set in configuration restricts which secrets Service B it is able to access, in this case it can only read the secret needed to connect to the Redis state store component, and no others. -7. Service B is deployed to namespace "B" which further isolates it from other services meaning that even if the service invocation API was enabled on it, it could not be accidently called by being in the space namespace as Service A. Furthermore Service B must explicity set the Redis Host namespace in its component YAML file to call onto the "Redis" namespace, otherwise this call also fails. -8. The data in the Redis state store is encrypted at rest and can only be read using the correctly configured Dapr Redis state store component. +5. Service B restricts the Dapr sidecar to only use the pub/sub, state management, and secrets APIs. All other API calls (for example, service invocation) would fail. +6. A secrets security policy set in configuration restricts which secrets Service B can access. In this case, Service B can only read the secret needed to connect to the Redis state store component, and no others. +7. Service B is deployed to namespace "B", which further isolates it from other services. Even if the service invocation API was enabled on it, it could not be called accidentally by being in the same namespace as Service A. Service B must explicitly set the Redis Host namespace in its component YAML file to call onto the "Redis" namespace, otherwise this call also fails. +8. The data in the Redis state store is encrypted at rest and can only be read using the correctly configured Dapr Redis state store component. # Threat model -Threat modeling is a process by which potential threats, such as structural vulnerabilities or the absence of appropriate safeguards, can be identified and enumerated, and mitigations can be prioritized. The Dapr threat model is below. + +Threat modeling is a process by which: + +- Potential threats, like structural vulnerabilities or the absence of appropriate safeguards, can be identified and enumerated. +- Mitigation can be prioritized. + +The Dapr threat model is below. Dapr threat model @@ -168,36 +213,39 @@ Threat modeling is a process by which potential threats, such as structural vuln ### February 2021 -In February 2021, Dapr went through a 2nd security audit targeting it's 1.0 release by Cure53. +In February 2021, Dapr went through a 2nd security audit targeting its 1.0 release by Cure53. + The test focused on the following: -* Dapr runtime codebase evaluation since last audit -* Access control lists -* Secrets management -* Penetration testing -* Validating fixes for previous high/medium issues +- Dapr runtime codebase evaluation since last audit +- Access control lists +- Secrets management +- Penetration testing +- Validating fixes for previous high/medium issues -The full report can be found [here](/docs/Dapr-february-2021-security-audit-report.pdf). +You can find the full report [here](/docs/Dapr-february-2021-security-audit-report.pdf). One high issue was detected and fixed during the test. -As of February 16th 2021, Dapr has 0 criticals, 0 highs, 0 mediums, 2 lows, 2 infos. + +As of February 16, 2021, Dapr has 0 criticals, 0 highs, 0 mediums, 2 lows, 2 infos. ### June 2020 In June 2020, Dapr underwent a security audit from Cure53, a CNCF-approved cybersecurity firm. + The test focused on the following: -* Dapr runtime codebase evaluation -* Dapr components codebase evaluation -* Dapr CLI codebase evaluation -* Privilege escalation -* Traffic spoofing -* Secrets management -* RBAC -* Validating base assumptions: mTLS, scopes, API authentication -* Orchestration hardening (Kubernetes) -* DoS attacks -* Penetration testing +- Dapr runtime codebase evaluation +- Dapr components codebase evaluation +- Dapr CLI codebase evaluation +- Privilege escalation +- Traffic spoofing +- Secrets management +- RBAC +- Validating base assumptions: mTLS, scopes, API authentication +- Orchestration hardening (Kubernetes) +- DoS attacks +- Penetration testing The full report can be found [here](/docs/Dapr-july-2020-security-audit-report.pdf). @@ -206,4 +254,5 @@ The full report can be found [here](/docs/Dapr-july-2020-security-audit-report.p Visit [this page]({{< ref support-security-issues.md >}}) to report a security issue to the Dapr maintainers. ## Related links - - [Operational Security]({{< ref "security.md" >}}) \ No newline at end of file + +[Operational Security]({{< ref "security.md" >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-bindings.md b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-bindings.md index fa20a26c826..c3251351651 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-bindings.md +++ b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-bindings.md @@ -6,31 +6,28 @@ description: "Invoke external systems with output bindings" weight: 300 --- -Output bindings enable you to invoke external resources without taking dependencies on special SDK or libraries. -For a complete sample showing output bindings, visit this [link](https://github.com/dapr/quickstarts/tree/master/tutorials/bindings). - -## Example: - -The below code example loosely describes an application that processes orders. In the example, there is an order processing service which has a Dapr sidecar. The order processing service uses Dapr to invoke external resources, in this case a Kafka, via an output binding. +With output bindings, you can invoke external resources without depending on special SDK or libraries. An output binding represents a resource that Dapr uses to invoke and send messages to. For a complete sample showing output bindings, [walk through the tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/bindings). Diagram showing bindings of example service -## 1. Create a binding +This guide uses a Kafka binding as an example. You can find your preferred binding spec from [the list of bindings components]({{< ref setup-bindings >}}). -An output binding represents a resource that Dapr uses to invoke and send messages to. - -For the purpose of this guide, you'll use a Kafka binding. You can find a list of the different binding specs [here]({{< ref setup-bindings >}}). +## Create a binding Create a new binding component with the name of `checkout`. -Inside the `metadata` section, configure Kafka related properties such as the topic to publish the message to and the broker. +Within the `metadata` section, configure Kafka-related properties, such as: + +- The topic to which you'll publish the message +- The broker + +Create the following `binding.yaml` file and save to a `components` sub-folder in your application directory. {{< tabs "Self-Hosted (CLI)" Kubernetes >}} {{% codetab %}} -Create the following YAML file, named `binding.yaml`, and save this to a `components` sub-folder in your application directory. -(Use the `--components-path` flag with `dapr run` to point to your custom components dir) +Use the `--components-path` flag with `dapr run` to point to your custom components directory. ```yaml apiVersion: dapr.io/v1alpha1 @@ -60,8 +57,7 @@ spec: {{% codetab %}} -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of your [desired binding component]({{< ref setup-bindings >}}) in the yaml below (in this case kafka), save as `binding.yaml`, and run `kubectl apply -f binding.yaml`. - +To deploy the following `binding.yaml` file into a Kubernetes cluster, run `kubectl apply -f binding.yaml`. ```yaml apiVersion: dapr.io/v1alpha1 @@ -91,7 +87,7 @@ spec: {{< /tabs >}} -## 2. Send an event (Output binding) +## Send an event (output binding) Below are code examples that leverage Dapr SDKs to interact with an output binding. @@ -135,12 +131,6 @@ namespace EventService ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --app-ssl dotnet run -``` - {{% /codetab %}} {{% codetab %}} @@ -179,12 +169,6 @@ public class OrderProcessingServiceApplication { ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 mvn spring-boot:run -``` - {{% /codetab %}} {{% codetab %}} @@ -213,12 +197,6 @@ while True: ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --app-protocol grpc python3 OrderProcessingService.py -``` - {{% /codetab %}} {{% codetab %}} @@ -257,12 +235,6 @@ func main() { ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 go run OrderProcessingService.go -``` - {{% /codetab %}} {{% codetab %}} @@ -299,19 +271,14 @@ async function sendOrder(orderId) { function sleep(ms) { return new Promise(resolve => setTimeout(resolve, ms)); } -``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 npm start ``` {{% /codetab %}} {{< /tabs >}} -All that's left now is to invoke the output bindings endpoint on a running Dapr instance. +Invoke the output bindings endpoint on a running Dapr instance. You can also invoke the output bindings endpoint using HTTP: @@ -319,11 +286,13 @@ You can also invoke the output bindings endpoint using HTTP: curl -X POST -H 'Content-Type: application/json' http://localhost:3601/v1.0/bindings/checkout -d '{ "data": 100, "operation": "create" }' ``` -As seen above, you invoked the `/binding` endpoint with the name of the binding to invoke, in our case its `checkout`. -The payload goes inside the mandatory `data` field, and can be any JSON serializable value. +As seen above: + +1. The example invoked the `/binding` endpoint with `checkout`, the name of the binding to invoke. +1. The payload goes inside the mandatory `data` field, and can be any JSON serializable value. +1. The `operation` field tells the binding what action it needs to take. For example, [the Kafka binding supports the `create` operation]({{< ref "kafka.md#binding-support" >}}). -You'll also notice that there's an `operation` field that tells the binding what you need it to do. -You can check [here]({{< ref supported-bindings >}}) which operations are supported for every output binding. +You can check [which operations (specific to each component) are supported for every output binding]({{< ref supported-bindings >}}). Watch this [video](https://www.youtube.com/watch?v=ysklxm81MTs&feature=youtu.be&t=1960) on how to use bi-directional output bindings. diff --git a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md index 3264b34422c..79e698a1168 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md +++ b/daprdocs/content/en/developing-applications/building-blocks/bindings/howto-triggers.md @@ -8,7 +8,7 @@ weight: 200 Using bindings, your code can be triggered with incoming events from different resources which can be anything: a queue, messaging pipeline, cloud-service, filesystem etc. -This is ideal for event-driven processing, data pipelines or just generally reacting to events and doing further processing. +This is ideal for event-driven processing, data pipelines, or generally reacting to events and performing further processing. Dapr bindings allow you to: @@ -16,30 +16,25 @@ Dapr bindings allow you to: * Replace bindings without changing your code * Focus on business logic and not the event resource implementation -For more info on bindings, read [this overview]({{}}). - -## Example: - -The below code example loosely describes an application that processes orders. In the example, there is an order processing service which has a Dapr sidecar. The checkout service uses Dapr to trigger the application via an input binding. +An input binding represents a resource that Dapr uses to read events from and push to your application. Read the [bindings overview for more information]({{}}). Diagram showing bindings of example service -## 1. Create a binding - -An input binding represents a resource that Dapr uses to read events from and push to your application. +This guide uses a Kafka binding as an example. You can find your preferred binding spec from [the list of bindings components]({{< ref setup-bindings >}}). -For the purpose of this guide, you'll use a Kafka binding. You can find a list of supported binding components [here]({{< ref setup-bindings >}}). +## Create a binding Create a new binding component with the name of `checkout`. -Inside the `metadata` section, configure Kafka related properties, such as the topic to publish the message to and the broker. +Inside the `metadata` section, configure Kafka-related properties, such as the topic to publish the message to and the broker. + +Create the following `binding.yaml` file and save it to a `components` sub-folder in your application directory. {{< tabs "Self-Hosted (CLI)" Kubernetes >}} {{% codetab %}} -Create the following YAML file, named `binding.yaml`, and save this to a `components` sub-folder in your application directory. -(Use the `--components-path` flag with `dapr run` to point to your custom components dir) +Use the `--components-path` flag with the `dapr run` command to point to your custom components directory. ```yaml apiVersion: dapr.io/v1alpha1 @@ -69,8 +64,7 @@ spec: {{% codetab %}} -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of your [desired binding component]({{< ref setup-bindings >}}) in the yaml below (in this case kafka), save as `binding.yaml`, and run `kubectl apply -f binding.yaml`. - +To deploy into a Kubernetes cluster, run `kubectl apply -f binding.yaml`. ```yaml apiVersion: dapr.io/v1alpha1 @@ -100,9 +94,9 @@ spec: {{< /tabs >}} -## 2. Listen for incoming events (input binding) +## Listen for incoming events (input binding) -Now configure your application to receive incoming events. If using HTTP, you need to listen on a `POST` endpoint with the name of the binding as specified in `metadata.name` in the file. +Configure your application to receive incoming events. If using HTTP, you need to listen on a `POST` endpoint with the name of the binding, as specified in `metadata.name` in the `binding.yaml` file. Below are code examples that leverage Dapr SDKs to demonstrate an output binding. @@ -134,12 +128,6 @@ namespace CheckoutService.controller ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 --app-ssl dotnet run -``` - {{% /codetab %}} {{% codetab %}} @@ -165,12 +153,6 @@ public class CheckoutServiceController { ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 mvn spring-boot:run -``` - {{% /codetab %}} {{% codetab %}} @@ -192,12 +174,6 @@ app.run(6002) ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --app-protocol grpc -- python3 CheckoutService.py -``` - {{% /codetab %}} {{% codetab %}} @@ -232,12 +208,6 @@ func main() { ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 go run CheckoutService.go -``` - {{% /codetab %}} {{% codetab %}} @@ -265,28 +235,22 @@ async function start() { ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: - -```bash -dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 dotnet npm start -``` - {{% /codetab %}} {{< /tabs >}} ### ACK-ing an event -In order to tell Dapr that you successfully processed an event in your application, return a `200 OK` response from your HTTP handler. +Tell Dapr you've successfully processed an event in your application by returning a `200 OK` response from your HTTP handler. ### Rejecting an event -In order to tell Dapr that the event was not processed correctly in your application and schedule it for redelivery, return any response other than `200 OK`. For example, a `500 Error`. +Tell Dapr the event was not processed correctly in your application and schedule it for redelivery by returning any response other than `200 OK`. For example, a `500 Error`. ### Specifying a custom route By default, incoming events will be sent to an HTTP endpoint that corresponds to the name of the input binding. -You can override this by setting the following metadata property: +You can override this by setting the following metadata property in `binding.yaml`: ```yaml name: mybinding @@ -298,6 +262,7 @@ spec: ``` ### Event delivery Guarantees + Event delivery guarantees are controlled by the binding implementation. Depending on the binding implementation, the event delivery can be exactly once or at least once. ## References @@ -305,4 +270,4 @@ Event delivery guarantees are controlled by the binding implementation. Dependin * [Bindings building block]({{< ref bindings >}}) * [Bindings API]({{< ref bindings_api.md >}}) * [Components concept]({{< ref components-concept.md >}}) -* [Supported bindings]({{< ref supported-bindings >}}) +* [Supported bindings]({{< ref supported-bindings >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/configuration/howto-manage-configuration.md b/daprdocs/content/en/developing-applications/building-blocks/configuration/howto-manage-configuration.md index 6cc818d0154..524d54dfd64 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/configuration/howto-manage-configuration.md +++ b/daprdocs/content/en/developing-applications/building-blocks/configuration/howto-manage-configuration.md @@ -6,22 +6,21 @@ weight: 2000 description: "Learn how to get application configuration and subscribe for changes" --- -## Introduction -This HowTo uses the Redis configuration store component as an example on how to retrieve a configuration item. +This example uses the Redis configuration store component to demonstrate how to retrieve a configuration item. -*This API is currently in `Alpha` state and only available on gRPC. An HTTP1.1 supported version with this URL syntax `/v1.0/configuration` will be available before the API is certified into `Stable` state.* +{{% alert title="Note" color="primary" %}} +This API is currently in `Alpha` state and only available on gRPC. An HTTP1.1 supported version with this URL syntax `/v1.0/configuration` will be available before the API is certified into `Stable` state. -## Example: - -The below code examples loosely describe an application that processes orders. In the examples, there is an order processing service which has a Dapr sidecar. The order processing service uses Dapr to retrieve the configuration from a Redis configuration store. +{{% /alert %}} Diagram showing get configuration of example service -## Step 1: Create a configuration item in store +## Create a configuration item in store -First, create a configuration item in a supported configuration store. This can be a simple key-value item, with any key of your choice. For this example, we'll use the Redis configuration store component. +Create a configuration item in a supported configuration store. This can be a simple key-value item, with any key of your choice. As mentioned earlier, this example uses the Redis configuration store component. ### Run Redis with Docker + ``` docker run --name my-redis -p 6379:6379 -d redis ``` @@ -42,7 +41,15 @@ MSET orderId1 "101||1" orderId2 "102||1" ### Configure a Dapr configuration store -Save the following component file, for example to the [default components folder]({{}}) on your machine. You can use this as the Dapr component YAML for Kubernetes using `kubectl` or when running with the Dapr CLI. Note: The Redis configuration component has identical metadata to the Redis state store component, so you can simply copy and change the Redis state store component type if you already have a Redis state store YAML file. +Save the following component file to the [default components folder]({{< ref "install-dapr-selfhost.md#step-5-verify-components-directory-has-been-initialized" >}}) on your machine. You can use this as the Dapr component YAML: + +- For Kubernetes using `kubectl`. +- When running with the Dapr CLI. + +{{% alert title="Note" color="primary" %}} + Since the Redis configuration component has identical metadata to the Redis `statestore.yaml` component, you can simply copy/change the Redis state store component type if you already have a Redis `statestore.yaml`. + +{{% /alert %}} ```yaml apiVersion: dapr.io/v1alpha1 @@ -63,6 +70,7 @@ spec: {{< tabs Dotnet Java Python>}} {{% codetab %}} + ```csharp //dependencies using System; @@ -88,14 +96,10 @@ namespace ConfigurationApi } ``` -Navigate to the directory containing the above code and run the following command to launch the application along with a Dapr sidecar: - -```bash -dapr run --app-id orderprocessing --components-path ./components -- dotnet run -``` {{% /codetab %}} {{% codetab %}} + ```java //dependencies import io.dapr.client.DaprClientBuilder; @@ -125,14 +129,10 @@ public static void main(String[] args) throws Exception { } ``` -Navigate to the directory containing the above code and run the following command to launch the application along with a Dapr sidecar: - -```bash -dapr run --app-id orderprocessing --components-path ./components mvn spring-boot:run -``` {{% /codetab %}} {{% codetab %}} + ```python #dependencies from dapr.clients import DaprClient @@ -146,12 +146,6 @@ with DaprClient() as d: print(f"Got key={configuration.items[0].key} value={configuration.items[0].value} version={configuration.items[0].version}") ``` -Navigate to the directory containing the above code and run the following command to launch the application along with a Dapr sidecar: - -```bash -dapr run --app-id orderprocessing --components-path ./components python3 OrderProcessingService.py -``` - {{% /codetab %}} {{< /tabs >}} @@ -163,37 +157,45 @@ Using your [favorite language](https://grpc.io/docs/languages/), create a Dapr g {{< tabs Java Dotnet Python Javascript >}} {{% codetab %}} + ```java Dapr.ServiceBlockingStub stub = Dapr.newBlockingStub(channel); stub.GetConfigurationAlpha1(new GetConfigurationRequest{ StoreName = "redisconfigstore", Keys = new String[]{"myconfig"} }); ``` + {{% /codetab %}} {{% codetab %}} + ```csharp var call = client.GetConfigurationAlpha1(new GetConfigurationRequest { StoreName = "redisconfigstore", Keys = new String[]{"myconfig"} }); ``` + {{% /codetab %}} {{% codetab %}} + ```python response = stub.GetConfigurationAlpha1(request={ StoreName: 'redisconfigstore', Keys = ['myconfig'] }) ``` + {{% /codetab %}} {{% codetab %}} + ```javascript client.GetConfigurationAlpha1({ StoreName: 'redisconfigstore', Keys = ['myconfig'] }) ``` + {{% /codetab %}} {{< /tabs >}} -##### Watch configuration items +#### Watch configuration items -Create a Dapr gRPC client from the [Dapr proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto) using your [preferred language](https://grpc.io/docs/languages/). Then use the proto method `SubscribeConfigurationAlpha1` on your client stub to start subscribing to events. The method accepts the following request object: +Create a Dapr gRPC client from the [Dapr proto](https://github.com/dapr/dapr/blob/master/dapr/proto/runtime/v1/dapr.proto) using your [preferred language](https://grpc.io/docs/languages/). Use the `SubscribeConfigurationAlpha1` proto method on your client stub to start subscribing to events. The method accepts the following request object: ```proto message SubscribeConfigurationRequest { @@ -210,45 +212,11 @@ message SubscribeConfigurationRequest { } ``` -Using this method, you can subscribe to changes in specific keys for a given configuration store. gRPC streaming varies widely based on language - see the [gRPC examples here](https://grpc.io/docs/languages/) for usage. - -Below are the examples in sdks: - -{{< tabs Python>}} - -{{% codetab %}} -```python -#dependencies -import asyncio -from dapr.clients import DaprClient -#code -async def executeConfiguration(): - with DaprClient() as d: - CONFIG_STORE_NAME = 'configstore' - key = 'orderId' - # Subscribe to configuration by key. - configuration = await d.subscribe_configuration(store_name=CONFIG_STORE_NAME, keys=[key], config_metadata={}) - if configuration != None: - items = configuration.get_items() - for item in items: - print(f"Subscribe key={item.key} value={item.value} version={item.version}", flush=True) - else: - print("Nothing yet") - -asyncio.run(executeConfiguration()) -``` - -```bash -dapr run --app-id orderprocessing --components-path components/ -- python3 OrderProcessingService.py -``` - -{{% /codetab %}} +Using this method, you can subscribe to changes in specific keys for a given configuration store. gRPC streaming varies widely based on language. [See the gRPC examples](https://grpc.io/docs/languages/) for usage. -{{< /tabs >}} +#### Stop watching configuration items -##### Stop watching configuration items - -After you have subscribed to watch configuration items, the gRPC-server stream starts. This stream thread does not close itself, and you have to do by explicitly call the `UnSubscribeConfigurationRequest` API. This method accepts the following request object: +After you've subscribed to watch configuration items, the gRPC-server stream starts. Since this stream thread does not close itself, you have to explicitly call the `UnSubscribeConfigurationRequest` API to unsubscribe. This method accepts the following request object: ```proto // UnSubscribeConfigurationRequest is the message to stop watching the key-value configuration. @@ -265,4 +233,5 @@ message UnSubscribeConfigurationRequest { Using this unsubscribe method, you can stop watching configuration update events. Dapr locates the subscription stream based on the `store_name` and any optional keys supplied and closes it. ## Next steps -* Read [configuration API overview]({{< ref configuration-api-overview.md >}}) + +* Read [configuration API overview]({{< ref configuration-api-overview.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/observability/w3c-tracing/w3c-tracing-howto.md b/daprdocs/content/en/developing-applications/building-blocks/observability/w3c-tracing/w3c-tracing-howto.md index 04ad0a4fd02..d008fdbfbff 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/observability/w3c-tracing/w3c-tracing-howto.md +++ b/daprdocs/content/en/developing-applications/building-blocks/observability/w3c-tracing/w3c-tracing-howto.md @@ -135,7 +135,7 @@ You can create a trace context using the recommended OpenCensus SDKs. OpenCensus | Language | SDK | |:-------:|:----:| | Go | [Link](https://pkg.go.dev/go.opencensus.io?tab=overview) -| Java | [Link](https://www.javadoc.io/doc/io.opencensus/opencensus-api/latest/index.html) +| Java | [Link](https://github.com/census-instrumentation/opencensus-java) | C# | [Link](https://github.com/census-instrumentation/opencensus-csharp/) | C++ | [Link](https://github.com/census-instrumentation/opencensus-cpp) | Node.js | [Link](https://github.com/census-instrumentation/opencensus-node) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md index 5393fd7efce..fd5f3548b66 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-publish-subscribe.md @@ -1,42 +1,41 @@ --- type: docs -title: "How-To: Publish a message and subscribe to a topic" -linkTitle: "How-To: Publish & subscribe" +title: "How to: Publish a message and subscribe to a topic" +linkTitle: "How to: Publish & subscribe to topics" weight: 2000 description: "Learn how to send messages to a topic with one service and subscribe to that topic in another service" --- -## Introduction +Now that you've learned what the Dapr pub/sub building block provides, learn how it can work in your service. The below code example loosely describes an application that processes orders with two services, each with Dapr sidecars: -Pub/Sub is a common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. -Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers. +- A checkout service using Dapr to subscribe to the topic in the message queue. +- An order processing service using Dapr to publish a message to RabbitMQ. -Dapr provides an extensible Pub/Sub system with At-Least-Once guarantees, allowing developers to publish and subscribe to topics. -Dapr provides components for pub/sub, that enable operators to use their preferred infrastructure, for example Redis Streams, Kafka, etc. -## Content Types +Diagram showing state management of example service -When publishing a message, it's important to specify the content type of the data being sent. -Unless specified, Dapr will assume `text/plain`. When using Dapr's HTTP API, the content type can be set in a `Content-Type` header. -gRPC clients and SDKs have a dedicated content type parameter. +Dapr automatically wraps the user payload in a CloudEvents v1.0 compliant envelope, using `Content-Type` header value for `datacontenttype` attribute. [Learn more about messages with CloudEvents.]({{< ref pubsub-cloudevents.md >}}) -## Example: +The following example demonstrates how your applications publish and subscribe to a topic called `orders`. -The below code example loosely describes an application that processes orders. In the example, there are two services - an order processing service and a checkout service. Both services have Dapr sidecars. The order processing service uses Dapr to publish a message to RabbitMQ and the checkout service subscribes to the topic in the message queue. +{{% alert title="Note" color="primary" %}} + If you haven't already, [try out the pub/sub quickstart]({{< ref pubsub-quickstart.md >}}) for a quick walk-through on how to use pub/sub. -Diagram showing state management of example service +{{% /alert %}} -## Step 1: Setup the Pub/Sub component -The following example creates applications to publish and subscribe to a topic called `orders`. +## Set up the Pub/Sub component -The first step is to setup the Pub/Sub component: +The first step is to set up the pub/sub component: {{< tabs "Self-Hosted (CLI)" Kubernetes >}} {{% codetab %}} -The pubsub.yaml is created by default on your local machine when running `dapr init`. Verify by opening your components file under `%UserProfile%\.dapr\components\pubsub.yaml` on Windows or `~/.dapr/components/pubsub.yaml` on Linux/MacOS. +When you run `dapr init`, Dapr creates a default Redis `pubsub.yaml` and runs a Redis container on your local machine, located: + +- On Windows, under `%UserProfile%\.dapr\components\pubsub.yaml` +- On Linux/MacOS, under `~/.dapr/components/pubsub.yaml` -In this example, RabbitMQ is used for publish and subscribe. Replace `pubsub.yaml` file contents with the below contents. +With the `pubsub.yaml` component, you can easily swap out underlying components without application code changes. In this example, RabbitMQ is used. ```yaml apiVersion: dapr.io/v1alpha1 @@ -64,11 +63,55 @@ scopes: - checkout ``` -You can override this file with another Redis instance or another [pubsub component]({{< ref setup-pubsub >}}) by creating a `components` directory containing the file and using the flag `--components-path` with the `dapr run` CLI command. +You can override this file with another [pubsub component]({{< ref setup-pubsub >}}) by creating a components directory (in this example, `myComponents`) containing the file and using the flag `--components-path` with the `dapr run` CLI command. + +{{< tabs Dotnet Java Python Go Javascript >}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- dotnet run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- python3 app.py +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- go run app.go +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- npm start +``` +{{% /codetab %}} + +{{< /tabs >}} + {{% /codetab %}} {{% codetab %}} -To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of your [desired pubsub component]({{< ref setup-pubsub >}}) in the yaml below, save as `pubsub.yaml`, and run `kubectl apply -f pubsub.yaml`. +To deploy this into a Kubernetes cluster, fill in the `metadata` connection details of the [pub/sub component]({{< ref setup-pubsub >}}) in the YAML below, save as `pubsub.yaml`, and run `kubectl apply -f pubsub.yaml`. ```yaml apiVersion: dapr.io/v1alpha1 @@ -96,26 +139,21 @@ scopes: - orderprocessing - checkout ``` + {{% /codetab %}} {{< /tabs >}} +## Subscribe to topics -## Step 2: Subscribe to topics - -Dapr allows two methods by which you can subscribe to topics: +Dapr provides two methods by which you can subscribe to topics: - **Declaratively**, where subscriptions are defined in an external file. - **Programmatically**, where subscriptions are defined in user code. -{{% alert title="Note" color="primary" %}} - Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependency from your code and allows, for example, existing applications to subscribe to topics, without having to change code. The programmatic approach implements the subscription in your code. - -{{% /alert %}} - -### Declarative subscriptions +Learn more in the [declarative and programmatic subscriptions doc]({{< ref subscription-methods.md >}}). This example demonstrates a **declarative** subscription. -You can subscribe to a topic using the following Custom Resources Definition (CRD). Create a file named `subscription.yaml` and paste the following: +Create a file named `subscription.yaml` and paste the following: ```yaml apiVersion: dapr.io/v1alpha1 @@ -132,69 +170,13 @@ scopes: ``` The example above shows an event subscription to topic `orders`, for the pubsub component `order-pub-sub`. + - The `route` field tells Dapr to send all topic messages to the `/checkout` endpoint in the app. - The `scopes` field enables this subscription for apps with IDs `orderprocessing` and `checkout`. -Set the component with: - -Place the CRD in your `./components` directory. When Dapr starts up, it loads subscriptions along with components. - -Note: By default, Dapr loads components from `$HOME/.dapr/components` on MacOS/Linux and `%USERPROFILE%\.dapr\components` on Windows. - -You can also override the default directory by pointing the Dapr CLI to a components path: - -{{< tabs Dotnet Java Python Go Javascript Kubernetes>}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- dotnet run -``` - -{{% /codetab %}} +Place `subscription.yaml` in the same directory as your `pubsub.yaml` component. When Dapr starts up, it loads subscriptions along with the components. -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- python3 app.py -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- go run app.go -``` - -{{% /codetab %}} - -{{% codetab %}} - -```bash -dapr run --app-id myapp --components-path ./myComponents -- npm start -``` - -{{% /codetab %}} - -{{% codetab %}} -In Kubernetes, save the CRD to a file and apply it to the cluster: -```bash -kubectl apply -f subscription.yaml -``` -{{% /codetab %}} - -{{< /tabs >}} - -Below are code examples that leverage Dapr SDKs to subscribe to a topic. +Below are code examples that leverage Dapr SDKs to subscribe to the topic you defined in `subscription.yaml`. {{< tabs Dotnet Java Python Go Javascript>}} @@ -226,7 +208,7 @@ namespace CheckoutService.controller } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 --app-ssl dotnet run @@ -266,7 +248,7 @@ public class CheckoutServiceController { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 mvn spring-boot:run @@ -295,7 +277,7 @@ def mytopic(event: v1.Event) -> None: app.run(6002) ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --app-protocol grpc -- python3 CheckoutService.py @@ -340,7 +322,7 @@ func eventHandler(ctx context.Context, e *common.TopicEvent) (retry bool, err er } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 go run CheckoutService.go @@ -380,7 +362,7 @@ async function start(orderId) { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the subscriber application: ```bash dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-port 60002 npm start @@ -390,44 +372,44 @@ dapr run --app-id checkout --app-port 6002 --dapr-http-port 3602 --dapr-grpc-por {{< /tabs >}} -The `/checkout` endpoint matches the `route` defined in the subscriptions and this is where Dapr will send all topic messages to. - -## Step 3: Publish a topic +## Publish a message Start an instance of Dapr with an app-id called `orderprocessing`: ```bash dapr run --app-id orderprocessing --dapr-http-port 3601 ``` + +Then publish a message to the `orders` topic: + {{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} {{% codetab %}} -Then publish a message to the `orders` topic: - ```bash dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"orderId": "100"}' ``` + {{% /codetab %}} {{% codetab %}} -Then publish a message to the `orders` topic: + ```bash curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/json" -d '{"orderId": "100"}' ``` + {{% /codetab %}} {{% codetab %}} -Then publish a message to the `orders` topic: + ```powershell Invoke-RestMethod -Method Post -ContentType 'application/json' -Body '{"orderId": "100"}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' ``` + {{% /codetab %}} {{< /tabs >}} -Dapr automatically wraps the user payload in a Cloud Events v1.0 compliant envelope, using `Content-Type` header value for `datacontenttype` attribute. - Below are code examples that leverage Dapr SDKs to publish a topic. {{< tabs Dotnet Java Python Go Javascript>}} @@ -470,7 +452,7 @@ namespace EventService } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --app-ssl dotnet run @@ -520,7 +502,7 @@ public class OrderProcessingServiceApplication { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 mvn spring-boot:run @@ -557,7 +539,7 @@ while True: logging.info('Published data: ' + str(orderId)) ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --app-protocol grpc python3 OrderProcessingService.py @@ -605,7 +587,7 @@ func main() { } ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 go run OrderProcessingService.go @@ -648,7 +630,7 @@ function sleep(ms) { main(); ``` -Navigate to the directory containing the above code, then run the following command to launch a Dapr sidecar and run the application: +Navigate to the directory containing the above code, then run the following command to launch both a Dapr sidecar and the publisher application: ```bash dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 npm start @@ -658,50 +640,13 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{< /tabs >}} -## Step 4: ACK-ing a message +## Message acknowledgement and retries In order to tell Dapr that a message was processed successfully, return a `200 OK` response. If Dapr receives any other return status code than `200`, or if your app crashes, Dapr will attempt to redeliver the message following at-least-once semantics. -## Sending a custom CloudEvent - -Dapr automatically takes the data sent on the publish request and wraps it in a CloudEvent 1.0 envelope. -If you want to use your own custom CloudEvent, make sure to specify the content type as `application/cloudevents+json`. - -Read about content types [here](#content-types), and about the [Cloud Events message format]({{< ref "pubsub-overview.md#cloud-events-message-format" >}}). - -#### Example - -{{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```bash -dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -``` -{{% /codetab %}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```bash -curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/cloudevents+json" -d '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -``` -{{% /codetab %}} - -{{% codetab %}} -Publish a custom CloudEvent to the `orders` topic: -```powershell -Invoke-RestMethod -Method Post -ContentType 'application/cloudevents+json' -Body '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' -``` -{{% /codetab %}} - -{{< /tabs >}} - ## Next steps -- Try the [Pub/Sub quickstart sample](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) -- Learn about [PubSub routing]({{< ref howto-route-messages >}}) -- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -- List of [pub/sub components]({{< ref setup-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Try the [pub/sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub). +- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md index 5fac6c647b2..be9c37f7966 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/howto-route-messages.md @@ -2,23 +2,23 @@ type: docs title: "How-To: Route messages to different event handlers" linkTitle: "How-To: Route events" -weight: 2100 +weight: 2300 description: "Learn how to route messages from a topic to different event handlers based on CloudEvent fields" --- {{% alert title="Preview feature" color="warning" %}} -Pub/Sub message routing is currently in [preview]({{< ref preview-features.md >}}). +Pub/sub message routing is currently in [preview]({{< ref preview-features.md >}}). {{% /alert %}} -## Introduction +Pub/sub routing is an implementation of [content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html), a messaging pattern that utilizes a DSL instead of imperative application code. With pub/sub routing, you use expressions to route [CloudEvents](https://cloudevents.io) (based on their contents) to different URIs/paths and event handlers in your application. If no route matches, then an optional default route is used. This proves useful as your applications expand to support multiple event versions or special cases. -[Content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html) is a messaging pattern that utilizes a DSL instead of imperative application code. PubSub routing is an implementation of this pattern that allows developers to use expressions to route [CloudEvents](https://cloudevents.io) based on their contents to different URIs/paths and event handlers in your application. If no route matches, then an optional default route is used. This becomes useful as your applications expands to support multiple event versions, or special cases. Routing can be implemented with code; however, keeping routing rules external from the application can improve portability. +While routing can be implemented with code, keeping routing rules external from the application can improve portability. -This feature is available to both the declarative and programmatic subscription approaches. +This feature is available to both the [declarative and programmatic subscription approaches]({{< ref subscription-methods.md >}}). ## Enable message routing -This is a preview feature. To enable it, add the `PubSub.Routing` feature entry to your application configuration like so: +To enable this preview feature, add the `PubSub.Routing` feature entry to your application configuration: ```yaml apiVersion: dapr.io/v1alpha1 @@ -30,10 +30,12 @@ spec: - name: PubSub.Routing enabled: true ``` -Learn more about enabling [preview features]({{}}). + +Learn more about enabling [preview features]({{< ref preview-features >}}). + ## Declarative subscription -For declarative subscriptions, you must use `dapr.io/v2alpha1` as the `apiVersion`. Here is an example of `subscriptions.yaml` using routing. +For declarative subscriptions, use `dapr.io/v2alpha1` as the `apiVersion`. Here is an example of `subscriptions.yaml` using routing: ```yaml apiVersion: dapr.io/v2alpha1 @@ -57,7 +59,7 @@ scopes: ## Programmatic subscription -Alternatively, the programattic approach varies slightly in that the `routes` structure is returned instead of `route`. The JSON structure matches the declarative YAML. +In the programmatic approach, the `routes` structure is returned instead of `route`. The JSON structure matches the declarative YAML: {{< tabs Python Node "C#" Go PHP>}} @@ -268,27 +270,34 @@ $app->start(); ## Common Expression Language (CEL) -In these examples, depending on the type of the event (`event.type`), the application will be called on `/widgets`, `/gadgets` or `/products`. The expressions are written as [Common Expression Language (CEL)](https://github.com/google/cel-spec) where `event` represents the cloud event. Any of the attributes from the [CloudEvents core specification](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) can be referenced in the expression. +In these examples, depending on the `event.type`, the application will be called on: + +- `/widgets` +- `/gadgets` +- `/products` + +The expressions are written as [Common Expression Language (CEL)](https://github.com/google/cel-spec) where `event` represents the cloud event. Any of the attributes from the [CloudEvents core specification](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) can be referenced in the expression. ### Example expressions -Match "important" messages +Match "important" messages: ```javascript has(event.data.important) && event.data.important == true ``` -Match deposits greater than $10000 +Match deposits greater than $10,000: ```javascript event.type == "deposit" && event.data.amount > 10000 ``` -Match multiple versions of a message +Match multiple versions of a message: ```javascript event.type == "mymessage.v1" ``` + ```javascript event.type == "mymessage.v2" ``` @@ -301,201 +310,154 @@ For reference, the following attributes are from the CloudEvents specification. #### data -As defined by the term Data, CloudEvents MAY include domain-specific information about the occurrence. When present, this information will be encapsulated within `data`. +As defined by the term **data**, CloudEvents _may_ include domain-specific information about the occurrence. When present, this information will be encapsulated within `data`. -- Description: The event payload. This specification does not place any restriction on the type of this information. It is encoded into a media format which is specified by the `datacontenttype` attribute (e.g. application/json), and adheres to the `dataschema` format when those respective attributes are present. -- Constraints: +- **Description:** The event payload. This specification places no restriction on the information type. It is encoded into a media format, specified by the `datacontenttype` attribute (e.g. application/json), and adheres to the `dataschema` format when those respective attributes are present. +- **Constraints:** - OPTIONAL {{% alert title="Limitation" color="warning" %}} -Currently, it is only possible to access the attributes inside data if it is nested JSON values and not JSON escaped in a string. +Currently, you can only access the attributes inside data if it is nested JSON values and not JSON escaped in a string. {{% /alert %}} ### REQUIRED Attributes -The following attributes are REQUIRED to be present in all CloudEvents: +The following attributes are **required** in all CloudEvents: #### id -- Type: `String` -- Description: Identifies the event. Producers MUST ensure that `source` + `id` - is unique for each distinct event. If a duplicate event is re-sent (e.g. due - to a network error) it MAY have the same `id`. Consumers MAY assume that - Events with identical `source` and `id` are duplicates. -- Constraints: +- **Type:** `String` +- **Description:** Identifies the event. Producers _must_ ensure that `source` + `id` + are unique for each distinct event. If a duplicate event is re-sent (e.g. due + to a network error), it may have the same `id`. Consumers may assume that + events with identical `source` and `id` are duplicates. +- **Constraints:** - REQUIRED - - MUST be a non-empty string - - MUST be unique within the scope of the producer -- Examples: + - Must be a non-empty string + - Must be unique within the scope of the producer +- **Examples:** - An event counter maintained by the producer - A UUID #### source -- Type: `URI-reference` -- Description: Identifies the context in which an event happened. Often this - will include information such as the type of the event source, the - organization publishing the event or the process that produced the event. The - exact syntax and semantics behind the data encoded in the URI is defined by - the event producer. +- **Type:** `URI-reference` +- **Description:** Identifies the context in which an event happened. Often this includes information such as: + - The type of the event source + - The organization publishing the event + - The process that produced the event + + The exact syntax and semantics behind the data encoded in the URI is defined by the event producer. - Producers MUST ensure that `source` + `id` is unique for each distinct event. + Producers _must_ ensure that `source` + `id` are unique for each distinct event. - An application MAY assign a unique `source` to each distinct producer, which - makes it easy to produce unique IDs since no other producer will have the same - source. The application MAY use UUIDs, URNs, DNS authorities or an - application-specific scheme to create unique `source` identifiers. + An application may: + - Assign a unique `source` to each distinct producer, making it easier to produce unique IDs and preventing other producers from having the same `source`. + - Use UUIDs, URNs, DNS authorities, or an application-specific scheme to create unique `source` identifiers. - A source MAY include more than one producer. In that case the producers MUST - collaborate to ensure that `source` + `id` is unique for each distinct event. + A source may include more than one producer. In this case, the producers _must_ collaborate to ensure that `source` + `id` are unique for each distinct event. -- Constraints: +- **Constraints:** - REQUIRED - - MUST be a non-empty URI-reference + - Must be a non-empty URI-reference - An absolute URI is RECOMMENDED -- Examples - - Internet-wide unique URI with a DNS authority. +- **Examples:** + - Internet-wide unique URI with a DNS authority: - https://github.com/cloudevents - mailto:cncf-wg-serverless@lists.cncf.io - Universally-unique URN with a UUID: - urn:uuid:6e8bc430-9c3a-11d9-9669-0800200c9a66 - - Application-specific identifiers + - Application-specific identifiers: - /cloudevents/spec/pull/123 - /sensors/tn-1234567/alerts - 1-555-123-4567 #### specversion -- Type: `String` -- Description: The version of the CloudEvents specification which the event - uses. This enables the interpretation of the context. Compliant event - producers MUST use a value of `1.0` when referring to this version of the - specification. +- **Type:** `String` +- **Description:** The version of the CloudEvents specification used by the event. This enables the interpretation of the context. Compliant event producers _must_ use a value of `1.0` when referring to this version of the specification. + + Currently, this attribute only includes the 'major' and 'minor' version numbers. This allows patch changes to the specification to be made without changing this property's value in the serialization. - Currently, this attribute will only have the 'major' and 'minor' version - numbers included in it. This allows for 'patch' changes to the specification - to be made without changing this property's value in the serialization. - Note: for 'release candidate' releases a suffix might be used for testing + Note: for 'release candidate' releases, a suffix might be used for testing purposes. -- Constraints: +- **Constraints:** - REQUIRED - - MUST be a non-empty string + - Must be a non-empty string #### type -- Type: `String` -- Description: This attribute contains a value describing the type of event - related to the originating occurrence. Often this attribute is used for - routing, observability, policy enforcement, etc. The format of this is - producer defined and might include information such as the version of the - `type` - see - [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) - for more information. -- Constraints: +- **Type:** `String` +- **Description:** Contains a value describing the event type related to the originating occurrence. Often, this attribute is used for routing, observability, policy enforcement, etc. The format is producer-defined and might include information like the version of the `type`. See [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) for more information. +- **Constraints:** - REQUIRED - - MUST be a non-empty string - - SHOULD be prefixed with a reverse-DNS name. The prefixed domain dictates the - organization which defines the semantics of this event type. -- Examples + - Must be a non-empty string + - Should be prefixed with a reverse-DNS name. The prefixed domain dictates the + organization, which defines the semantics of this event type. +- **Examples:** - com.github.pull_request.opened - com.example.object.deleted.v2 ### OPTIONAL Attributes -The following attributes are OPTIONAL to appear in CloudEvents. See the -[Notational Conventions](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#notational-conventions) section for more information -on the definition of OPTIONAL. +The following attributes are **optional** to appear in CloudEvents. See the [Notational Conventions](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#notational-conventions) section for more information on the definition of OPTIONAL. #### datacontenttype -- Type: `String` per [RFC 2046](https://tools.ietf.org/html/rfc2046) -- Description: Content type of `data` value. This attribute enables `data` to - carry any type of content, whereby format and encoding might differ from that - of the chosen event format. For example, an event rendered using the - [JSON envelope](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#3-envelope) format might carry an XML payload - in `data`, and the consumer is informed by this attribute being set to - "application/xml". The rules for how `data` content is rendered for different - `datacontenttype` values are defined in the event format specifications; for - example, the JSON event format defines the relationship in - [section 3.1](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#31-handling-of-data). - - For some binary mode protocol bindings, this field is directly mapped to the - respective protocol's content-type metadata property. Normative rules for the - binary mode and the content-type metadata mapping can be found in the - respective protocol. - - In some event formats the `datacontenttype` attribute MAY be omitted. For - example, if a JSON format event has no `datacontenttype` attribute, then it is - implied that the `data` is a JSON value conforming to the "application/json" - media type. In other words: a JSON-format event with no `datacontenttype` is - exactly equivalent to one with `datacontenttype="application/json"`. - - When translating an event message with no `datacontenttype` attribute to a - different format or protocol binding, the target `datacontenttype` SHOULD be - set explicitly to the implied `datacontenttype` of the source. - -- Constraints: +- **Type:** `String` per [RFC 2046](https://tools.ietf.org/html/rfc2046) +- **Description:** Content type of `data` value. This attribute enables `data` to carry any type of content, whereby format and encoding might differ from that of the chosen event format. + + For example, an event rendered using the [JSON envelope](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#3-envelope) format might carry an XML payload in `data`. The consumer is informed by this attribute being set to `"application/xml"`. + + The rules for how `data` content is rendered for different `datacontenttype` values are defined in the event format specifications. For example, the JSON event format defines the relationship in [section 3.1](https://github.com/cloudevents/spec/blob/v1.0.1/json-format.md#31-handling-of-data). + + For some binary mode protocol bindings, this field is directly mapped to the respective protocol's content-type metadata property. You can find normative rules for the binary mode and the content-type metadata mapping in the respective protocol. + + In some event formats, you may omit the `datacontenttype` attribute. For example, if a JSON format event has no `datacontenttype` attribute, it's implied that the `data` is a JSON value conforming to the `"application/json"` media type. In other words: a JSON-format event with no `datacontenttype` is exactly equivalent to one with `datacontenttype="application/json"`. + + When translating an event message with no `datacontenttype` attribute to a different format or protocol binding, the target `datacontenttype` should be set explicitly to the implied `datacontenttype` of the source. + +- **Constraints:** - OPTIONAL - - If present, MUST adhere to the format specified in - [RFC 2046](https://tools.ietf.org/html/rfc2046) -- For Media Type examples see - [IANA Media Types](http://www.iana.org/assignments/media-types/media-types.xhtml) + - If present, must adhere to the format specified in [RFC 2046](https://tools.ietf.org/html/rfc2046) +- For Media Type examples, see [IANA Media Types](http://www.iana.org/assignments/media-types/media-types.xhtml) #### dataschema -- Type: `URI` -- Description: Identifies the schema that `data` adheres to. Incompatible - changes to the schema SHOULD be reflected by a different URI. See - [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) - for more information. -- Constraints: +- **Type:** `URI` +- **Description:** Identifies the schema that `data` adheres to. Incompatible changes to the schema should be reflected by a different URI. See [Versioning of CloudEvents in the Primer](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#versioning-of-cloudevents) for more information. +- **Constraints:** - OPTIONAL - - If present, MUST be a non-empty URI + - If present, must be a non-empty URI #### subject -- Type: `String` -- Description: This describes the subject of the event in the context of the - event producer (identified by `source`). In publish-subscribe scenarios, a - subscriber will typically subscribe to events emitted by a `source`, but the - `source` identifier alone might not be sufficient as a qualifier for any - specific event if the `source` context has internal sub-structure. - - Identifying the subject of the event in context metadata (opposed to only in - the `data` payload) is particularly helpful in generic subscription filtering - scenarios where middleware is unable to interpret the `data` content. In the - above example, the subscriber might only be interested in blobs with names - ending with '.jpg' or '.jpeg' and the `subject` attribute allows for - constructing a simple and efficient string-suffix filter for that subset of - events. - -- Constraints: +- **Type:** `String` +- **Description:** This describes the event subject in the context of the event producer (identified by `source`). In publish-subscribe scenarios, a subscriber will typically subscribe to events emitted by a `source`. The `source` identifier alone might not be sufficient as a qualifier for any specific event if the `source` context has internal sub-structure. + + Identifying the subject of the event in context metadata (opposed to only in the `data` payload) is helpful in generic subscription filtering scenarios, where middleware is unable to interpret the `data` content. In the above example, the subscriber might only be interested in blobs with names ending with '.jpg' or '.jpeg'. With the `subject` attribute, you can construct a simple and efficient string-suffix filter for that subset of events. + +- **Constraints:** - OPTIONAL - - If present, MUST be a non-empty string -- Example: - - A subscriber might register interest for when new blobs are created inside a - blob-storage container. In this case, the event `source` identifies the - subscription scope (storage container), the `type` identifies the "blob - created" event, and the `id` uniquely identifies the event instance to - distinguish separate occurrences of a same-named blob having been created; - the name of the newly created blob is carried in `subject`: - - `source`: https://example.com/storage/tenant/container - - `subject`: mynewfile.jpg + - If present, must be a non-empty string +- **Example:** + A subscriber might register interest for when new blobs are created inside a blob-storage container. In this case: + - The event `source` identifies the subscription scope (storage container) + - The event `type` identifies the "blob created" event + - The event `id` uniquely identifies the event instance to distinguish separately created occurrences of a same-named blob. + + The name of the newly created blob is carried in `subject`: + - `source`: https://example.com/storage/tenant/container + - `subject`: mynewfile.jpg #### time -- Type: `Timestamp` -- Description: Timestamp of when the occurrence happened. If the time of the - occurrence cannot be determined then this attribute MAY be set to some other - time (such as the current time) by the CloudEvents producer, however all - producers for the same `source` MUST be consistent in this respect. In other - words, either they all use the actual time of the occurrence or they all use - the same algorithm to determine the value used. -- Constraints: +- **Type:** `Timestamp` +- **Description:** Timestamp of when the occurrence happened. If the time of the occurrence cannot be determined, then this attribute may be set to some other time (such as the current time) by the CloudEvents producer. However, all producers for the same `source` _must_ be consistent in this respect. In other words, either they all use the actual time of the occurrence or they all use the same algorithm to determine the value used. +- **Constraints:** - OPTIONAL - - If present, MUST adhere to the format specified in - [RFC 3339](https://tools.ietf.org/html/rfc3339) + - If present, must adhere to the format specified in [RFC 3339](https://tools.ietf.org/html/rfc3339) {{% alert title="Limitation" color="warning" %}} Currently, comparisons to time (e.g. before or after "now") are not supported. @@ -511,9 +473,8 @@ Watch [this video](https://www.youtube.com/watch?v=QqJgRmbH82I&t=1063s) on how t ## Next steps -- Try the [Pub/Sub routing sample](https://github.com/dapr/samples/tree/master/pub-sub-routing) -- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -- List of [pub/sub components]({{< ref setup-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Try the [pub/sub routing sample](https://github.com/dapr/samples/tree/master/pub-sub-routing). +- Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) and [message time-to-live]({{< ref pubsub-message-ttl.md >}}). +- [Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md new file mode 100644 index 00000000000..e2ab04d5709 --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-cloudevents.md @@ -0,0 +1,109 @@ +--- +type: docs +title: "Publishing & subscribing messages with Cloudevents" +linkTitle: "Messages with Cloudevents" +weight: 2100 +description: "Learn why Dapr uses CloudEvents, how they work in Dapr pub/sub, and how to create CloudEvents." +--- + +To enable message routing and provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a CloudEvents envelope, using the [`Content-Type` header value]({{< ref "pubsub-overview.md#content-types" >}}) for `datacontenttype` attribute. + +Dapr uses CloudEvents to provide additional context to the event payload, enabling features like: + +- Tracing +- Deduplication by message Id +- Content-type for proper deserialization of event data + +## CloudEvents example + +Dapr implements the following CloudEvents fields when creating a message topic. + +- `id` +- `source` +- `specversion` +- `type` +- `traceparent` +- `datacontenttype` (optional) + +The following example demonstrates an `orders` topic message sent by Dapr that includes a W3C `traceid` unique to the message, the `data` and the fields for the CloudEvent where the data content is serialized as JSON. + +```json +{ + "topic": "orders", + "pubsubname": "order_pub_sub", + "traceid": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01", + "tracestate": "", + "data": { + "orderId": 1 + }, + "id": "5929aaac-a5e2-4ca1-859c-edfe73f11565", + "specversion": "1.0", + "datacontenttype": "application/json; charset=utf-8", + "source": "checkout", + "type": "com.dapr.event.sent", + "traceparent": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01" +} +``` + +As another example of a v1.0 CloudEvent, the following shows data as XML content in a CloudEvent message serialized as JSON: + +```json +{ + "specversion" : "1.0", + "type" : "xml.message", + "source" : "https://example.com/message", + "subject" : "Test XML Message", + "id" : "id-1234-5678-9101", + "time" : "2020-09-23T06:23:21Z", + "datacontenttype" : "text/xml", + "data" : "User1user2hi" +} +``` + +## Publish your own CloudEvent + +If you want to use your own CloudEvent, make sure to specify the [`datacontenttype`]({{< ref "pubsub-overview.md#setting-message-content-types" >}}) as `application/cloudevents+json`. + +### Example + +{{< tabs "Dapr CLI" "HTTP API (Bash)" "HTTP API (PowerShell)">}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```bash +dapr publish --publish-app-id orderprocessing --pubsub order-pub-sub --topic orders --data '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' +``` + +{{% /codetab %}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```bash +curl -X POST http://localhost:3601/v1.0/publish/order-pub-sub/orders -H "Content-Type: application/cloudevents+json" -d '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' +``` + +{{% /codetab %}} + +{{% codetab %}} + +Publish a CloudEvent to the `orders` topic: + +```powershell +Invoke-RestMethod -Method Post -ContentType 'application/cloudevents+json' -Body '{"specversion" : "1.0", "type" : "com.dapr.cloudevent.sent", "source" : "testcloudeventspubsub", "subject" : "Cloud Events Test", "id" : "someCloudEventId", "time" : "2021-08-02T09:00:00Z", "datacontenttype" : "application/cloudevents+json", "data" : {"orderId": "100"}}' -Uri 'http://localhost:3601/v1.0/publish/order-pub-sub/orders' +``` + +{{% /codetab %}} + +{{< /tabs >}} + +## Next steps + +- Learn why you might [not want to use CloudEvents]({{< ref pubsub-raw.md >}}) +- Try out the [pub/sub Quickstart]({{< ref pubsub-quickstart.md >}}) +- List of [pub/sub components]({{< ref setup-pubsub >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) + diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md index a30acdc5253..d6029842c73 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-message-ttl.md @@ -3,7 +3,7 @@ type: docs title: "Message Time-to-Live (TTL)" linkTitle: "Message TTL" weight: 6000 -description: "Use time-to-live in Pub/Sub messages." +description: "Use time-to-live in pub/sub messages." --- ## Introduction @@ -18,6 +18,11 @@ In some components, such as Kafka, time-to-live can be configured in the topic v When message time-to-live has native support in the pub/sub component, Dapr simply forwards the time-to-live configuration without adding any extra logic, keeping predictable behavior. This is helpful when the expired messages are handled differently by the component. For example, with Azure Service Bus, where expired messages are stored in the dead letter queue and are not simply deleted. +{{% alert title="Note" color="primary" %}} + You can also set message TTL for a given message broker at creation. Look at the specific characteristic of the component that you are using to see if this is suitable. + +{{% /alert %}} + ### Supported components #### Azure Service Bus @@ -85,6 +90,6 @@ See [this guide]({{< ref pubsub_api.md >}}) for a reference on the pub/sub API. ## Next steps - Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +- Learn [how to configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) - Read the [API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md index f185f020304..98426226c88 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-overview.md @@ -3,80 +3,94 @@ type: docs title: "Publish and subscribe overview" linkTitle: "Overview" weight: 1000 -description: "Overview of the Pub/Sub API building block" +description: "Overview of the pub/sub API building block" --- -## Introduction +## Publish and subscribe pattern -The [publish/subscribe pattern](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) allows microservices to communicate with each other using messages. The **producer or publisher** sends messages to a **topic** without knowledge of what application will receive them. This involves writing them to an input channel. Similarly, a **consumer or subscriber** subscribes to the topic and receive its messages without any knowledge of what service produced these messages. This involves receiving messages from an output channel. An intermediary message broker is responsible for copying each message from an input channel to an output channels for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another. +The publish and subscribe pattern (pub/sub) enables microservices to communicate with each other using messages for event-driven architectures. -The publish/subscribe API in Dapr provides an at-least-once guarantee and integrates with various message brokers and queuing systems. The specific implementation used by your service is pluggable and configured as a Dapr pub/sub component at runtime. This approach removes the dependency from your service and, as a result, makes your service more portable and flexible to changes. +- The producer, or **publisher**, writes messages to an input channel and sends them to a topic, unaware which application will receive them. +- The consumer, or **subscriber**, subscribes to the topic and receives messages from an output channel, unaware which service produced these messages. -The complete list of Dapr pub/sub components is [here]({{< ref supported-pubsub >}}). +An intermediary message broker copies each message from a publisher's input channel to an output channel for all subscribers interested in that message. This pattern is especially useful when you need to decouple microservices from one another.

-The Dapr pub/sub building block provides a platform-agnostic API to send and receive messages. Your services publish messages to a named topic and also subscribe to a topic to consume the messages. +## Pub/sub API in Dapr -The service makes a network call to a Dapr pub/sub building block, exposed as a sidecar. This building block then makes calls into a Dapr pub/sub component that encapsulates a specific message broker product. To receive topics, Dapr subscribes to the Dapr pub/sub component on behalf of your service and delivers the messages to an endpoint when they arrive. +The pub/sub API in Dapr: +- Provides a platform-agnostic API to send and receive messages. +- Offers at-least-once message delivery guarantee. +- Integrates with various message brokers and queuing systems. -The diagram below shows an example of a "shipping" service and an "email" service that have both subscribed to topics that are published by the "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component, for example Redis Streams, NATS Streaming, Azure Service Bus, or GCP Pub/Sub. +The specific message broker used by your service is pluggable and configured as a Dapr pub/sub component at runtime. This removes the dependency from your service and makes your service more portable and flexible to changes. + +When using pub/sub in Dapr: + +1. Your service makes a network call to a Dapr pub/sub building block API. +1. The pub/sub building block makes calls into a Dapr pub/sub component that encapsulates a specific message broker. +1. To receive messages on a topic, Dapr subscribes to the pub/sub component on behalf of your service with a topic and delivers the messages to an endpoint on your service when they arrive. + +In the diagram below, a "shipping" service and an "email" service have both subscribed to topics published by a "cart" service. Each service loads pub/sub component configuration files that point to the same pub/sub message bus component; for example: Redis Streams, NATS Streaming, Azure Service Bus, or GCP pub/sub.

-The diagram below has the same services, however this time showing the Dapr publish API that sends an "order" topic and order endpoints on the subscribing services that these topic messages are delivered posted to by Dapr. +In the diagram below, the Dapr API posts an "order" topic from the publishing "cart" service to "order" endpoints on the "shipping" and "email" subscribing services.

-## Features -The pub/sub building block provides several features to your application. +[View the complete list of pub/sub components that Dapr supports]({{< ref supported-pubsub >}}). -### Cloud Events message format +## Dapr pub/sub API features -To enable message routing and to provide additional context with each message, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically "wrapped" in a Cloud Events envelope, using `Content-Type` header value for `datacontenttype` attribute. +The pub/sub building block brings several features to your application. -Dapr implements the following Cloud Events fields: +### Sending messages using Cloud Events -* `id` -* `source` -* `specversion` -* `type` -* `datacontenttype` (Optional) +To enable message routing and provide additional context with each message between services, Dapr uses the [CloudEvents 1.0 specification](https://github.com/cloudevents/spec/tree/v1.0) as its message format. Any message sent by an application to a topic using Dapr is automatically wrapped in a Cloud Events envelope, using [`Content-Type` header value]({{< ref "pubsub-overview.md#content-types" >}}) for `datacontenttype` attribute. -The following example shows an XML content in CloudEvent v1.0 serialized as JSON: +For more information, read about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}), or [sending raw messages without CloudEvents]({{< ref pubsub-raw.md >}}). -```json -{ - "specversion" : "1.0", - "type" : "xml.message", - "source" : "https://example.com/message", - "subject" : "Test XML Message", - "id" : "id-1234-5678-9101", - "time" : "2020-09-23T06:23:21Z", - "datacontenttype" : "text/xml", - "data" : "User1user2hi" -} -``` - -### Message subscription +### Communication with applications not using Dapr and CloudEvents -Dapr applications can subscribe to published topics. Dapr allows two methods by which your applications can subscribe to topics: +If one of your applications uses Dapr while another doesn't, you can disable the CloudEvent wrapping for a publisher or subscriber. This allows partial adoption of Dapr pub/sub in applications that cannot adopt Dapr all at once. - - **Declarative**, where a subscription is defined in an external file, - - **Programmatic**, where a subscription is defined in the user code. +For more information, read [how to use pub/sub without CloudEvents]({{< ref pubsub-raw.md >}}). - Both declarative and programmatic approaches support the same features. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. The programmatic approach implements the subscription in your code. +### Setting message content types - For more information read [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe >}}). +When publishing a message, it's important to specify the content type of the data being sent. Unless specified, Dapr will assume `text/plain`. +- HTTP client: the content type can be set in a `Content-Type` header +- gRPC client and SDK: have a dedicated content type parameter ### Message delivery -In principle, Dapr considers message successfully delivered when the subscriber responds with a non-error response after processing the message. For more granular control, Dapr's publish/subscribe API also provides explicit statuses, defined in the response payload, which the subscriber can use to indicate the specific handling instructions to Dapr (e.g. `RETRY` or `DROP`). For more information on message routing read [Dapr publish/subscribe API documentation]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}}) +In principle, Dapr considers a message successfully delivered once the subscriber processes the message and responds with a non-error response. For more granular control, Dapr's pub/sub API also provides explicit statuses, defined in the response payload, with which the subscriber indicates specific handling instructions to Dapr (for example, `RETRY` or `DROP`). + +### Receiving messages with topic subscriptions + +Dapr applications can subscribe to published topics via two methods that support the same features: declarative and programmatic. + +| Subscription method | Description | +| ------------------- | ----------- | +| **Declarative** | Subscription is defined in an **external file**. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. | +| **Programmatic** | Subscription is defined in the **user code**. The programmatic approach implements the subscription in your code. | + +For more information, read [about the subscriptions in Subscription Methods]({{< ref subscription-methods.md >}}). + +### Message routing + +Dapr provides [content-based routing](https://www.enterpriseintegrationpatterns.com/ContentBasedRouter.html) pattern. [Pub/sub routing]({{< ref howto-route-messages.md >}}) is an implementation of this pattern that allows developers to use expressions to route [CloudEvents](https://cloudevents.io) based on their contents to different URIs/paths and event handlers in your application. If no route matches, an optional default route is used. This is useful as your applications expands to support multiple event versions or special cases. + +This feature is available to both the declarative and programmatic subscription approaches. + +For more information on message routing, read [Dapr pub/sub API reference]({{< ref "pubsub_api.md#provide-routes-for-dapr-to-deliver-topic-events" >}}) ### Handling failed messages with dead letter topics @@ -84,43 +98,45 @@ Sometimes, messages can't be processed because of a variety of possible issues, ### At-least-once guarantee -Dapr guarantees "At-Least-Once" semantics for message delivery. That means that when an application publishes a message to a topic using the publish/subscribe API, Dapr ensures that this message will be delivered at least once to every subscriber. +Dapr guarantees at-least-once semantics for message delivery. When an application publishes a message to a topic using the pub/sub API, Dapr ensures the message is delivered *at least once* to every subscriber. ### Consumer groups and competing consumers pattern -The burden of dealing with concepts like consumer groups and multiple application instances using a single consumer group is all handled automatically by Dapr. When multiple instances of the same application (running same app-IDs) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This is commonly known as the competing consumers pattern and is illustrated in the diagram below. +Dapr automatically handles the burden of dealing with concepts like consumer groups and competing consumers pattern. The competing consumers pattern refers to multiple application instances using a single consumer group. When multiple instances of the same application (running same Dapr app ID) subscribe to a topic, Dapr delivers each message to *only one instance of **that** application*. This concept is illustrated in the diagram below.

-Similarly, if two different applications (different app-IDs) subscribe to the same topic, Dapr deliver each message to *only one instance of **each** application*. +Similarly, if two different applications (with different app-IDs) subscribe to the same topic, Dapr delivers each message to *only one instance of **each** application*. -### Topic scoping +### Scoping topics for added security -By default, all topics backing the Dapr pub/sub component (e.g. Kafka, Redis Stream, RabbitMQ) are available to every application configured with that component. To limit which application can publish or subscribe to topics, Dapr provides topic scoping. This enables you to say which topics an application is allowed to publish and which topics an application is allowed to subscribe to. For more information read [publish/subscribe topic scoping]({{< ref pubsub-scopes.md >}}). +By default, all topic messages associated with an instance of a pub/sub component are available to every application configured with that component. You can limit which application can publish or subscribe to topics with Dapr topic scoping. For more information, read: [pub/sub topic scoping]({{< ref pubsub-scopes.md >}}). ### Message Time-to-Live (TTL) -Dapr can set a timeout message on a per message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This is to prevent the build up of messages that are not read. A message that has been in the queue for longer than the configured TTL is said to be dead. For more information read [publish/subscribe message time-to-live]({{< ref pubsub-message-ttl.md >}}). -- Note: Message TTL can also be set for a given queue at the time of component creation. Look at the specific characteristic of the component that you are using. +Dapr can set a timeout message on a per-message basis, meaning that if the message is not read from the pub/sub component, then the message is discarded. This timeout message prevents a build up of unread messages. If a message has been in the queue longer than the configured TTL, it is marked as dead. For more information, read [pub/sub message TTL]({{< ref pubsub-message-ttl.md >}}). -### Communication with applications not using Dapr and CloudEvents -For scenarios where one application uses Dapr but another doesn't, CloudEvent wrapping can be disabled for a publisher or subscriber. This allows partial adoption of Dapr pubsub in applications that cannot adopt Dapr all at once. For more information read [how to use pubsub without CloudEvent]({{< ref pubsub-raw.md >}}). +## Try out pub/sub + +### Quickstarts and tutorials + +Want to put the Dapr pub/sub API to the test? Walk through the following quickstart and tutorials to see pub/sub in action: + +| Quickstart/tutorial | Description | +| ------------------- | ----------- | +| [Pub/sub quickstart]({{< ref pubsub-quickstart.md >}}) | Send and receive messages using the publish and subscribe API. | +| [Pub/sub tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) | Demonstrates how to use Dapr to enable pub-sub applications. Uses Redis as a pub-sub component. | -### Publish/Subscribe API +### Start using pub/sub directly in your app -The publish/subscribe API is located in the [API reference]({{< ref pubsub_api.md >}}). +Want to skip the quickstarts? Not a problem. You can try out the pub/sub building block directly in your application to publish messages and subscribe to a topic. After [Dapr is installed]({{< ref "getting-started/_index.md" >}}), you can begin using the pub/sub API starting with [the pub/sub how-to guide]({{< ref howto-publish-subscribe.md >}}). ## Next steps -* Follow these guides on: - * [How-To: Publish a message and subscribe to a topic]({{< ref howto-publish-subscribe.md >}}) - * [How-To: Configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) -* Try out the [Pub/Sub quickstart sample](https://github.com/dapr/quickstarts/tree/master/tutorials/pub-sub) -* Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) -* Learn about [message time-to-live (TTL)]({{< ref pubsub-message-ttl.md >}}) -* Learn about [pubsub without CloudEvent]({{< ref pubsub-raw.md >}}) -* List of [pub/sub components]({{< ref supported-pubsub.md >}}) -* Read the [pub/sub API reference]({{< ref pubsub_api.md >}}) +- Learn about [messaging with CloudEvents]({{< ref pubsub-cloudevents.md >}}) and when you might want to [send messages without CloudEvents]({{< ref pubsub-raw.md >}}). +- Follow [How-To: Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}). +- Review the list of [pub/sub components]({{< ref setup-pubsub >}}). +- Read the [API reference]({{< ref pubsub_api.md >}}). diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md index f4d542f2daf..d5a0fbe61b0 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-raw.md @@ -1,32 +1,23 @@ --- type: docs -title: "Pub/Sub without CloudEvents" -linkTitle: "Pub/Sub without CloudEvents" -weight: 7000 -description: "Use Pub/Sub without CloudEvents." +title: "Publishing & subscribing messages without CloudEvents" +linkTitle: "Messages without CloudEvents" +weight: 2200 +description: "Learn when you might not use CloudEvents and how to disable them." --- -## Introduction +When adding Dapr to your application, some services may still need to communicate via pub/sub messages not encapsulated in CloudEvents, due to either compatibility reasons or some apps not using Dapr. These are referred to as "raw" pub/sub messages. Dapr enables apps to [publish and subscribe to raw events]({{< ref "pubsub-cloudevents.md#publishing-raw-messages" >}}) not wrapped in a CloudEvent for compatibility. -Dapr uses CloudEvents to provide additional context to the event payload, enabling features like: -* Tracing -* Deduplication by message Id -* Content-type for proper deserialization of event's data +## Publishing raw messages -For more information about CloudEvents, read the [CloudEvents specification](https://github.com/cloudevents/spec). +Dapr apps are able to publish raw events to pub/sub topics without CloudEvent encapsulation, for compatibility with non-Dapr apps. -When adding Dapr to your application, some services may still need to communicate via raw pub/sub messages not encapsulated in CloudEvents. This may be for compatibility reasons, or because some apps are not using Dapr. Dapr enables apps to publish and subscribe to raw events that are not wrapped in a CloudEvent. +Diagram showing how to publish with Dapr when subscriber does not use Dapr or CloudEvent {{% alert title="Warning" color="warning" %}} Not using CloudEvents disables support for tracing, event deduplication per messageId, content-type metadata, and any other features built using the CloudEvent schema. {{% /alert %}} -## Publishing raw messages - -Dapr apps are able to publish raw events to pub/sub topics without CloudEvent encapsulation, for compatibility with non-Dapr apps. - -Diagram showing how to publish with Dapr when subscriber does not use Dapr or CloudEvent - To disable CloudEvent wrapping, set the `rawPayload` metadata to `true` as part of the publishing request. This allows subscribers to receive these messages without having to parse the CloudEvent schema. {{< tabs curl "Python SDK" "PHP SDK">}} @@ -147,7 +138,7 @@ $app->start(); ## Declaratively subscribe to raw events -Similarly, you can subscribe to raw events declaratively by adding the `rawPayload` metadata entry to your Subscription Custom Resource Definition (CRD): +Similarly, you can subscribe to raw events declaratively by adding the `rawPayload` metadata entry to your subscription specification. ```yaml apiVersion: dapr.io/v1alpha1 @@ -167,6 +158,6 @@ scopes: ## Next steps -- Learn more about [how to publish and subscribe]({{< ref howto-publish-subscribe.md >}}) +- Learn more about [publishing and subscribing messages]({{< ref pubsub-overview.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) -- Read the [API reference]({{< ref pubsub_api.md >}}) +- Read the [API reference]({{< ref pubsub_api.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md index 34ffa9d8879..df45ad8f376 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/pubsub-scopes.md @@ -1,9 +1,9 @@ --- type: docs -title: "Scope Pub/Sub topic access" +title: "Scope Pub/sub topic access" linkTitle: "Scope topic access" weight: 5000 -description: "Use scopes to limit Pub/Sub topics to specific applications" +description: "Use scopes to limit pub/sub topics to specific applications" --- ## Introduction @@ -164,7 +164,7 @@ The table below shows which application is allowed to subscribe to the topics: ## Next steps -- Learn [how to configure Pub/Sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +- Learn [how to configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) - Learn about [message time-to-live]({{< ref pubsub-message-ttl.md >}}) - List of [pub/sub components]({{< ref supported-pubsub >}}) - Read the [API reference]({{< ref pubsub_api.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md b/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md new file mode 100644 index 00000000000..e34a9dd8a3a --- /dev/null +++ b/daprdocs/content/en/developing-applications/building-blocks/pubsub/subscription-methods.md @@ -0,0 +1,366 @@ +--- +type: docs +title: "Declarative and programmatic subscription methods" +linkTitle: "Subscription methods" +weight: 3000 +description: "Learn more about the methods by which Dapr allows you to subscribe to topics." +--- + +## Pub/sub API subscription methods + +Dapr applications can subscribe to published topics via two methods that support the same features: declarative and programmatic. + +| Subscription method | Description | +| ------------------- | ----------- | +| [**Declarative**]({{< ref "subscription-methods.md#declarative-subscriptions" >}}) | Subscription is defined in an **external file**. The declarative approach removes the Dapr dependency from your code and allows for existing applications to subscribe to topics, without having to change code. | +| [**Programmatic**]({{< ref "subscription-methods.md#programmatic-subscriptions" >}}) | Subscription is defined in the **application code**. The programmatic approach implements the subscription in your code. | + +The examples below demonstrate pub/sub messaging between a `checkout` app and an `orderprocessing` app via the `orders` topic. The examples demonstrate the same Dapr pub/sub component used first declaratively, then programmatically. + +### Declarative subscriptions + +You can subscribe declaratively to a topic using an external component file. This example uses a YAML component file named `subscription.yaml`: + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Subscription +metadata: + name: order +spec: + topic: orders + route: /checkout + pubsubname: pubsub +scopes: +- orderprocessing +- checkout +``` + +Here the subscription called `order`: +- Uses the pub/sub component called `pubsub` to subscribes to the topic called `orders`. +- Sets the `route` field to send all topic messages to the `/checkout` endpoint in the app. +- Sets `scopes` field to scope this subscription for access only by apps with IDs `orderprocessing` and `checkout`. + +When running Dapr, set the YAML component file path to point Dapr to the component. + +{{< tabs ".NET" Java Python JavaScript Go Kubernetes>}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- dotnet run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- mvn spring-boot:run +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- python3 app.py +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- npm start +``` + +{{% /codetab %}} + +{{% codetab %}} + +```bash +dapr run --app-id myapp --components-path ./myComponents -- go run app.go +``` + +{{% /codetab %}} + +{{% codetab %}} + +In Kubernetes, apply the component to the cluster: + +```bash +kubectl apply -f subscription.yaml +``` + +{{% /codetab %}} + +{{< /tabs >}} + +In your application code, subscribe to the topic specified in the Dapr pub/sub component. + +{{< tabs ".NET" Java Python JavaScript Go >}} + +{{% codetab %}} + +```csharp + //Subscribe to a topic +[Topic("pubsub", "orders")] +[HttpPost("checkout")] +public void getCheckout([FromBody] int orderId) +{ + Console.WriteLine("Subscriber received : " + orderId); +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```java + //Subscribe to a topic +@Topic(name = "orders", pubsubName = "pubsub") +@PostMapping(path = "/checkout") +public Mono getCheckout(@RequestBody(required = false) CloudEvent cloudEvent) { + return Mono.fromRunnable(() -> { + try { + log.info("Subscriber received: " + cloudEvent.getData()); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```python +#Subscribe to a topic +@app.subscribe(pubsub_name='pubsub', topic='orders') +def mytopic(event: v1.Event) -> None: + data = json.loads(event.Data()) + logging.info('Subscriber received: ' + str(data)) + +app.run(6002) +``` + +{{% /codetab %}} + +{{% codetab %}} + +```javascript +//Subscribe to a topic +await server.pubsub.subscribe("pubsub", "orders", async (orderId) => { + console.log(`Subscriber received: ${JSON.stringify(orderId)}`) +}); +await server.startServer(); +``` + +{{% /codetab %}} + +{{% codetab %}} + +```go +//Subscribe to a topic +if err := s.AddTopicEventHandler(sub, eventHandler); err != nil { + log.Fatalf("error adding topic subscription: %v", err) +} +if err := s.Start(); err != nil && err != http.ErrServerClosed { + log.Fatalf("error listenning: %v", err) +} + +func eventHandler(ctx context.Context, e *common.TopicEvent) (retry bool, err error) { + log.Printf("Subscriber received: %s", e.Data) + return false, nil +} +``` + +{{% /codetab %}} + +{{< /tabs >}} + +The `/checkout` endpoint matches the `route` defined in the subscriptions and this is where Dapr sends all topic messages to. + +### Programmatic subscriptions + +The programmatic approach returns the `routes` JSON structure within the code, unlike the declarative approach's `route` YAML structure. In the example below, you define the values found in the [declarative YAML subscription](#declarative-subscriptions) above within the application code. + +{{< tabs ".NET" Java Python JavaScript Go>}} + +{{% codetab %}} + +```csharp +[Topic("pubsub", "checkout", event.type ==\"order\"")] +[HttpPost("orders")] +public async Task> HandleCheckout(Checkout checkout, [FromServices] DaprClient daprClient) +{ + // Logic + return stock; +} +``` + +{{% /codetab %}} + +{{% codetab %}} + +```java +private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + +@Topic(name = "checkout", pubsubName = "pubsub") +@PostMapping(path = "/orders") +public Mono handleMessage(@RequestBody(required = false) CloudEvent cloudEvent) { + return Mono.fromRunnable(() -> { + try { + System.out.println("Subscriber received: " + cloudEvent.getData()); + System.out.println("Subscriber received: " + OBJECT_MAPPER.writeValueAsString(cloudEvent)); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); +``` + +{{% /codetab %}} + +{{% codetab %}} + +```python +@app.route('/dapr/subscribe', methods=['GET']) +def subscribe(): + subscriptions = [ + { + 'pubsubname': 'pubsub', + 'topic': 'checkout', + 'routes': { + 'rules': [ + { + 'match': 'event.type == "order"', + 'path': '/orders' + }, + ], + 'default': '/orders' + } + }] + return jsonify(subscriptions) + +@app.route('/orders', methods=['POST']) +def ds_subscriber(): + print(request.json, flush=True) + return json.dumps({'success':True}), 200, {'ContentType':'application/json'} +app.run() +``` + +{{% /codetab %}} + +{{% codetab %}} + +```javascript +const express = require('express') +const bodyParser = require('body-parser') +const app = express() +app.use(bodyParser.json({ type: 'application/*+json' })); + +const port = 3000 + +app.get('/dapr/subscribe', (req, res) => { + res.json([ + { + pubsubname: "pubsub", + topic: "checkout", + routes: { + rules: [ + { + match: 'event.type == "order"', + path: '/orders' + }, + ], + default: '/products' + } + } + ]); +}) + +app.post('/orders', (req, res) => { + console.log(req.body); + res.sendStatus(200); +}); + +app.listen(port, () => console.log(`consumer app listening on port ${port}!`)) +``` + +{{% /codetab %}} + +{{% codetab %}} + +```go +package main + + "encoding/json" + "fmt" + "log" + "net/http" + + "github.com/gorilla/mux" +) + +const appPort = 3000 + +type subscription struct { + PubsubName string `json:"pubsubname"` + Topic string `json:"topic"` + Metadata map[string]string `json:"metadata,omitempty"` + Routes routes `json:"routes"` +} + +type routes struct { + Rules []rule `json:"rules,omitempty"` + Default string `json:"default,omitempty"` +} + +type rule struct { + Match string `json:"match"` + Path string `json:"path"` +} + +// This handles /dapr/subscribe +func configureSubscribeHandler(w http.ResponseWriter, _ *http.Request) { + t := []subscription{ + { + PubsubName: "pubsub", + Topic: "checkout", + Routes: routes{ + Rules: []rule{ + { + Match: `event.type == "order"`, + Path: "/orders", + }, + }, + Default: "/orders", + }, + }, + } + + w.WriteHeader(http.StatusOK) + json.NewEncoder(w).Encode(t) +} + +func main() { + router := mux.NewRouter().StrictSlash(true) + router.HandleFunc("/dapr/subscribe", configureSubscribeHandler).Methods("GET") + log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", appPort), router)) +} +``` +{{% /codetab %}} + +{{< /tabs >}} + +## Next Steps + +* Try out the [pub/sub Quickstart]({{< ref pubsub-quickstart.md >}}) +* Follow: [How-To: Configure pub/sub components with multiple namespaces]({{< ref pubsub-namespaces.md >}}) +* Learn more about [declarative and programmatic subscription methods]({{< ref subscription-methods >}}). +* Learn about [topic scoping]({{< ref pubsub-scopes.md >}}) +* Learn about [message TTL]({{< ref pubsub-message-ttl.md >}}) +* Learn more about [pub/sub with and without CloudEvent]({{< ref pubsub-cloudevents.md >}}) +* List of [pub/sub components]({{< ref supported-pubsub.md >}}) +* Read the [pub/sub API reference]({{< ref pubsub_api.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/secrets/howto-secrets.md b/daprdocs/content/en/developing-applications/building-blocks/secrets/howto-secrets.md index 015553e264b..8630094a64d 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/secrets/howto-secrets.md +++ b/daprdocs/content/en/developing-applications/building-blocks/secrets/howto-secrets.md @@ -6,20 +6,16 @@ weight: 2000 description: "Use the secret store building block to securely retrieve a secret" --- -This article provides guidance on using Dapr's secrets API in your code to leverage the [secrets store building block]({{}}). The secrets API allows you to easily retrieve secrets in your application code from a configured secret store. - -## Example - -The below code example loosely describes an application that processes orders. In the example, there is an order processing service, which has a Dapr sidecar. The order processing service uses Dapr to store a secret in a local secret store. +This guide demonstrates how to use Dapr's secrets API in your code to leverage the [secrets store building block]({{< ref secrets-overview >}}). With the secrets API, you easily retrieve secrets in your application code from a configured secret store. Diagram showing secrets management of example service ## Set up a secret store -Before retrieving secrets in your application's code, you must have a secret store component configured. For the purposes of this guide, as an example you will configure a local secret store which uses a local JSON file to store secrets. +Before retrieving secrets in your application's code, you must configure a secret store component. This example configures a local secret store which uses a local JSON file to store secrets. {{% alert title="Warning" color="warning" %}} -In a production-grade application, local secret stores are not recommended. You can find other alternatives [here]({{}}) to securely manage your secrets. +In a production-grade application, local secret stores are not recommended. [Find alternatives]({{< ref supported-secret-stores >}}) to securely manage your secrets. {{% /alert %}} Create a file named `secrets.json` with the following contents: @@ -48,50 +44,13 @@ spec: value: ":" ``` ->Note: the path to the secret store JSON is relative to where you call `dapr run` from. - -To configure a different kind of secret store see the guidance on [how to configure a secret store]({{}}) and review [supported secret stores]({{}}) to see specific details required for different secret store solutions. -## Get a secret - -Run the Dapr sidecar with the application. - -{{< tabs Dotnet Java Python Go Javascript>}} - -{{% codetab %}} -```bash -dapr run --app-id orderprocessingservice --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --components-path ./components dotnet run -``` -{{% /codetab %}} - - -{{% codetab %}} -```bash -dapr run --app-id orderprocessingservice --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --components-path ./components mvn spring-boot:run -``` -{{% /codetab %}} - - -{{% codetab %}} -```bash -dapr run --app-id orderprocessingservice --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --components-path ./components python3 OrderProcessingService.py -``` -{{% /codetab %}} - - -{{% codetab %}} -```bash -dapr run --app-id orderprocessingservice --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --components-path ./components go run OrderProcessingService.go -``` -{{% /codetab %}} - +{{% alert title="Warning" color="warning" %}} +The path to the secret store JSON is relative to where you call `dapr run` from. +{{% /alert %}} -{{% codetab %}} -```bash -dapr run --app-id orderprocessingservice --app-port 6001 --dapr-http-port 3601 --dapr-grpc-port 60001 --components-path ./components npm start -``` -{{% /codetab %}} +For more information, see how to [configure a different kind of secret store]({{< ref setup-secret-store >}}) and review [supported secret stores]({{< ref supported-secret-stores >}}) to see specific details required for different secret store solutions. -{{< /tabs >}} +## Get a secret Get the secret by calling the Dapr sidecar using the secrets API: @@ -99,7 +58,7 @@ Get the secret by calling the Dapr sidecar using the secrets API: curl http://localhost:3601/v1.0/secrets/localsecretstore/secret ``` -For a full API reference, go [here]({{< ref secrets_api.md >}}). +See a [full API reference]({{< ref secrets_api.md >}}). ## Calling the secrets API from your code @@ -108,6 +67,7 @@ Once you have a secret store, call Dapr to get the secrets from your application {{< tabs Dotnet Java Python Go Javascript>}} {{% codetab %}} + ```csharp //dependencies using System; @@ -136,6 +96,7 @@ namespace EventService } } ``` + {{% /codetab %}} {{% codetab %}} @@ -169,6 +130,7 @@ public class OrderProcessingServiceApplication { } } ``` + {{% /codetab %}} {{% codetab %}} @@ -198,6 +160,7 @@ with DaprClient() as client: logging.info('Result for bulk secret: ') logging.info(sorted(secret.secrets.items())) ``` + {{% /codetab %}} {{% codetab %}} @@ -235,6 +198,7 @@ func main() { } } ``` + {{% /codetab %}} {{% codetab %}} @@ -259,6 +223,7 @@ async function main() { main(); ``` + {{% /codetab %}} {{< /tabs >}} @@ -270,4 +235,4 @@ main(); - [Configure a secret store]({{}}) - [Supported secrets]({{}}) - [Using secrets in components]({{}}) -- [Secret stores quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/secretstore) +- [Secret stores tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/secretstore) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/secrets/secrets-overview.md b/daprdocs/content/en/developing-applications/building-blocks/secrets/secrets-overview.md index 044510bb558..afad09c85f4 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/secrets/secrets-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/secrets/secrets-overview.md @@ -54,4 +54,17 @@ For more information read [referencing secret stores in components]({{< ref comp To provide more granular control on access to secrets, Dapr provides the ability to define scopes and restricting access permissions. Learn more about [using secret scoping]({{}}) +## Try out secrets management +### Quickstarts and tutorials + +Want to put the Dapr secrets management API to the test? Walk through the following quickstart and tutorials to see Dapr secrets in action: + +| Quickstart/tutorial | Description | +| ------------------- | ----------- | +| [Secrets management quickstart]({{< ref secrets-quickstart.md >}}) | Retrieve secrets in the application code from a configured secret store using the secrets management API. | +| [Secret Store tutorial](https://github.com/dapr/quickstarts/tree/master/tutorials/secretstore) | Demonstrates the use of Dapr Secrets API to access secret stores. | + +### Start managing secrets directly in your app + +Want to skip the quickstarts? Not a problem. You can try out the secret management building block directly in your application to retrieve and manage secrets. After [Dapr is installed]({{< ref "getting-started/_index.md" >}}), you can begin using the secrets management API starting with [the secrets how-to guide]({{< ref howto-secrets.md >}}). \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/service-invocation/howto-invoke-discover-services.md b/daprdocs/content/en/developing-applications/building-blocks/service-invocation/howto-invoke-discover-services.md index 77dd0e38651..2af48cbf352 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/service-invocation/howto-invoke-discover-services.md +++ b/daprdocs/content/en/developing-applications/building-blocks/service-invocation/howto-invoke-discover-services.md @@ -6,22 +6,18 @@ description: "Call between services using service invocation" weight: 2000 --- -This article describe how to deploy services each with an unique application ID, so that other services can discover and call endpoints on them using service invocation API. - -## Example: - -The below code examples loosely describes an application that processes orders. In the examples, there are two services - an order processing service and a checkout service. Both services have Dapr sidecars and the order processing service uses Dapr to invoke the checkout method in the checkout service. +This article demonstrates how to deploy services each with an unique application ID for other services to discover and call endpoints on them using service invocation over HTTP. Diagram showing service invocation of example service -## Step 1: Choose an ID for your service +For a complete sample demonstrating service invocation, [walk through the service invocation quickstart](https://github.com/dapr/quickstarts/tree/master/service_invocation). -Dapr allows you to assign a global, unique ID for your app. This ID encapsulates the state for your application, regardless of the number of instances it may have. +## Choose an ID for your service +Dapr allows you to assign a global, unique ID for your app. This ID encapsulates the state for your application, regardless of the number of instances it may have. {{< tabs Dotnet Java Python Go Javascript Kubernetes>}} - {{% codetab %}} ```bash @@ -88,7 +84,6 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{% /codetab %}} - {{% codetab %}} ```bash @@ -111,7 +106,6 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{% /codetab %}} - {{% codetab %}} ```bash @@ -134,7 +128,6 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{% /codetab %}} - {{% codetab %}} ### Set an app-id when deploying to Kubernetes @@ -164,24 +157,23 @@ spec: dapr.io/app-port: "6001" ... ``` + *If your app uses an SSL connection, you can tell Dapr to invoke your app over an insecure SSL connection with the `app-ssl: "true"` annotation (full list [here]({{< ref arguments-annotations-overview.md >}}))* {{% /codetab %}} {{< /tabs >}} -## Step 2: Invoke the service - -To invoke an application using Dapr, you can use the `invoke` API on any Dapr instance. +## Invoke the service -The sidecar programming model encourages each application to interact with its own instance of Dapr. The Dapr sidecars discover and communicate with one another. +To invoke an application using Dapr, you can use the `invoke` API on any Dapr instance. The sidecar programming model encourages each application to interact with its own instance of Dapr. The Dapr sidecars discover and communicate with one another. Below are code examples that leverage Dapr SDKs for service invocation. {{< tabs Dotnet Java Python Go Javascript>}} - {{% codetab %}} + ```csharp //dependencies using System; @@ -217,10 +209,11 @@ namespace EventService } } ``` -{{% /codetab %}} +{{% /codetab %}} {{% codetab %}} + ```java //dependencies import io.dapr.client.DaprClient; @@ -258,9 +251,11 @@ public class OrderProcessingServiceApplication { } } ``` + {{% /codetab %}} {{% codetab %}} + ```python #dependencies import random @@ -285,9 +280,11 @@ while True: logging.info('Order requested: ' + str(orderId)) logging.info('Result: ' + str(result)) ``` + {{% /codetab %}} {{% codetab %}} + ```go //dependencies import ( @@ -324,9 +321,11 @@ func main() { } } ``` + {{% /codetab %}} {{% codetab %}} + ```javascript //dependencies import { DaprClient, HttpMethod, CommunicationProtocolEnum } from 'dapr-client'; @@ -359,6 +358,7 @@ function sleep(ms) { main(); ``` + {{% /codetab %}} {{< /tabs >}} @@ -366,17 +366,18 @@ main(); ### Additional URL formats To invoke a 'GET' endpoint: + ```bash curl http://localhost:3602/v1.0/invoke/checkout/method/checkout/100 ``` -In order to avoid changing URL paths as much as possible, Dapr provides the following ways to call the service invocation API: - +To avoid changing URL paths as much as possible, Dapr provides the following ways to call the service invocation API: 1. Change the address in the URL to `localhost:`. 2. Add a `dapr-app-id` header to specify the ID of the target service, or alternatively pass the ID via HTTP Basic Auth: `http://dapr-app-id:@localhost:3602/path`. -For example, the following command +For example, the following command: + ```bash curl http://localhost:3602/v1.0/invoke/checkout/method/checkout/100 ``` @@ -403,7 +404,7 @@ dapr invoke --app-id checkout --method checkout/100 When running on [namespace supported platforms]({{< ref "service_invocation_api.md#namespace-supported-platforms" >}}), you include the namespace of the target app in the app ID: `checkout.production` -For example, invoking the example service with a namespace would be: +For example, invoking the example service with a namespace would look like: ```bash curl http://localhost:3602/v1.0/invoke/checkout.production/method/checkout/100 -X POST @@ -411,13 +412,17 @@ curl http://localhost:3602/v1.0/invoke/checkout.production/method/checkout/100 - See the [Cross namespace API spec]({{< ref "service_invocation_api.md#cross-namespace-invocation" >}}) for more information on namespaces. -## Step 3: View traces and logs +## View traces and logs + +Our example above showed you how to directly invoke a different service running locally or in Kubernetes. Dapr: -The example above showed you how to directly invoke a different service running locally or in Kubernetes. Dapr outputs metrics, tracing and logging information allowing you to visualize a call graph between services, log errors and optionally log the payload body. +- Outputs metrics, tracing, and logging information, +- Allows you to visualize a call graph between services and log errors, and +- Optionally, log the payload body. -For more information on tracing and logs see the [observability]({{< ref observability-concept.md >}}) article. +For more information on tracing and logs, see the [observability]({{< ref observability-concept.md >}}) article. - ## Related Links +## Related Links -* [Service invocation overview]({{< ref service-invocation-overview.md >}}) -* [Service invocation API specification]({{< ref service_invocation_api.md >}}) +- [Service invocation overview]({{< ref service-invocation-overview.md >}}) +- [Service invocation API specification]({{< ref service_invocation_api.md >}}) \ No newline at end of file diff --git a/daprdocs/content/en/developing-applications/building-blocks/service-invocation/service-invocation-overview.md b/daprdocs/content/en/developing-applications/building-blocks/service-invocation/service-invocation-overview.md index 4236525d2b5..6517f8acd7b 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/service-invocation/service-invocation-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/service-invocation/service-invocation-overview.md @@ -10,18 +10,22 @@ description: "Overview of the service invocation API building block" Using service invocation, your application can reliably and securely communicate with other applications using the standard [gRPC](https://grpc.io) or [HTTP](https://www.w3.org/Protocols/) protocols. -In many environments with multiple services that need to communicate with each other, developers often ask themselves the following questions: +In many microservice-based applications multiple services need the ability to communicate with one another. This inter-service communication requires that application developers handle problems like: -- How do I discover and invoke methods on different services? -- How do I call other services securely with encryption and apply access control on the methods? -- How do I handle retries and transient errors? -- How do I use tracing to see a call graph with metrics to diagnose issues in production? +- **Service discovery.** How do I discover my different services? +- **Standardizing API calls between services.** How do I invoke methods between services? +- **Secure inter-service communication.** How do I call other services securely with encryption and apply access control on the methods? +- **Mitigating request timeouts or failures.** How do I handle retries and transient errors? +- **Implementing observability and tracing.** How do I use tracing to see a call graph with metrics to diagnose issues in production? -Dapr addresses these challenges by providing a service invocation API that acts as a combination of a reverse proxy with built-in service discovery, while leveraging built-in distributed tracing, metrics, error handling, encryption and more. +Dapr addresses these challenges by providing a service invocation API that acts similar to a reverse proxy with built-in service discovery, while leveraging built-in distributed tracing, metrics, error handling, encryption and more. -Dapr uses a sidecar architecture. To invoke an application using Dapr, you use the `invoke` API on any Dapr instance. The sidecar programming model encourages each applications to talk to its own instance of Dapr. The Dapr instances discover and communicate with one another. +Dapr uses a sidecar architecture. To invoke an application using Dapr: +- You use the `invoke` API on the Dapr instance. +- Each application communicates with its own instance of Dapr. +- The Dapr instances discover and communicate with each other. -### Service invocation +### Service invocation diagram The diagram below is an overview of how Dapr's service invocation works. @@ -30,7 +34,7 @@ The diagram below is an overview of how Dapr's service invocation works. 1. Service A makes an HTTP or gRPC call targeting Service B. The call goes to the local Dapr sidecar. 2. Dapr discovers Service B's location using the [name resolution component]({{< ref supported-name-resolution >}}) which is running on the given [hosting platform]({{< ref "hosting" >}}). 3. Dapr forwards the message to Service B's Dapr sidecar - - **Note**: All calls between Dapr sidecars go over gRPC for performance. Only calls between services and Dapr sidecars can be either HTTP or gRPC + - **Note**: All calls between Dapr sidecars go over gRPC for performance. Only calls between services and Dapr sidecars can be either HTTP or gRPC. 4. Service B's Dapr sidecar forwards the request to the specified endpoint (or method) on Service B. Service B then runs its business logic code. 5. Service B sends a response to Service A. The response goes to Service B's sidecar. 6. Dapr forwards the response to Service A's Dapr sidecar. @@ -39,37 +43,39 @@ The diagram below is an overview of how Dapr's service invocation works. ## Features Service invocation provides several features to make it easy for you to call methods between applications. -### Namespace scoping - -Applications can be scoped to namespaces for deployment and security, and you can call between services deployed to different namespaces. For more information, read the [Service invocation across namespaces]({{< ref "service-invocation-namespaces.md" >}}) article. +### HTTP and gRPC service invocation +- **HTTP**: If you're already using HTTP protocols in your application, using the Dapr HTTP header might be the easiest way to get started. You don't need to change your existing endpoint URLs; just add the `dapr-app-id` header and you're ready to go. For more information, see [Invoke Services using HTTP]({{< ref howto-invoke-discover-services.md >}}). +- **gRPC**: Dapr allows users to keep their own proto services and work natively with gRPC. This means that you can use service invocation to call your existing gRPC apps without having to include any Dapr SDKs or include custom gRPC services. For more information, see the [how-to tutorial for Dapr and gRPC]({{< ref howto-invoke-services-grpc.md >}}). ### Service-to-service security -All calls between Dapr applications can be made secure with mutual (mTLS) authentication on hosted platforms, including automatic certificate rollover, via the Dapr Sentry service. +With the Dapr Sentry service, all calls between Dapr applications can be made secure with mutual (mTLS) authentication on hosted platforms, including automatic certificate rollover. For more information read the [service-to-service security]({{< ref "security-concept.md#sidecar-to-sidecar-communication" >}}) article. -### Access control +### Resiliency including retries -Applications can control which other applications are allowed to call them and what they are authorized to do via access policies. This enables you to restrict sensitive applications, that say have personnel information, from being accessed by unauthorized applications, and combined with service-to-service secure communication, provides for soft multi-tenancy deployments. +In the event of call failures and transient errors, service invocation provides a resiliency feature that performs automatic retries with backoff time periods. To find out more, see the [Resiliency article here]({{< ref resiliency-overview.md >}}). -For more information read the [access control allow lists for service invocation]({{< ref invoke-allowlist.md >}}) article. +### Tracing and metrics with observability -### Retries +By default, all calls between applications are traced and metrics are gathered to provide insights and diagnostics for applications. This is especially important in production scenarios, providing call graphs and metrics on the calls between your services. For more information read about [observability]({{< ref observability-concept.md >}}). -Service invocation performs automatic retries with backoff time periods in the event of call failures and transient errors. -Errors that cause retries are: +### Access control -- Network errors including endpoint unavailability and refused connections. -- Authentication errors due to a renewing certificate on the calling/callee Dapr sidecars. +With access policies, applications can control: -Per call retries are performed with a backoff interval of 1 second up to a threshold of 3 times. -Connection establishment via gRPC to the target sidecar has a timeout of 5 seconds. +- Which applications are allowed to call them. +- What applications are authorized to do. -### Pluggable service discovery +For example, you can restrict sensitive applications with personnel information from being accessed by unauthorized applications. Combined with service-to-service secure communication, you can provide for soft multi-tenancy deployments. -Dapr can run on a variety of [hosting platforms]({{< ref hosting >}}). To enable service discovery and service invocation, Dapr uses pluggable [name resolution components]({{< ref supported-name-resolution >}}). For example, the Kubernetes name resolution component uses the Kubernetes DNS service to resolve the location of other applications running in the cluster. Self-hosted machines can use the mDNS name resolution component. The Consul name resolution component can be used in any hosting environment including Kubernetes or self-hosted. +For more information read the [access control allow lists for service invocation]({{< ref invoke-allowlist.md >}}) article. + +### Namespace scoping + +You can scope applications to namespaces for deployment and security and call between services deployed to different namespaces. For more information, read the [Service invocation across namespaces]({{< ref "service-invocation-namespaces.md" >}}) article. ### Round robin load balancing with mDNS @@ -79,23 +85,15 @@ The diagram below shows an example of how this works. If you have 1 instance of Diagram showing the steps of service invocation -**Note**: App ID is unique per application, not application instance. This means regardless of how many instances of that application exist (due to scaling), all of them will share the same app ID. - -### Tracing and metrics with observability +**Note**: App ID is unique per _application_, not application instance. Regardless how many instances of that application exist (due to scaling), all of them will share the same app ID. -By default, all calls between applications are traced and metrics are gathered to provide insights and diagnostics for applications, which is especially important in production scenarios. This gives you call graphs and metrics on the calls between your services. For more information read about [observability]({{< ref observability-concept.md >}}). - -### Service invocation API - -The API for service invocation can be found in the [service invocation API reference]({{< ref service_invocation_api.md >}}) which describes how to invoke a method on another service. - -### gRPC proxying +### Pluggable service discovery -Dapr allows users to keep their own proto services and work natively with gRPC. This means that you can use service invocation to call your existing gRPC apps without having to include any Dapr SDKs or include custom gRPC services. For more information, see the [how-to tutorial for Dapr and gRPC]({{< ref howto-invoke-services-grpc.md >}}). +Dapr can run on a variety of [hosting platforms]({{< ref hosting >}}). To enable service discovery and service invocation, Dapr uses pluggable [name resolution components]({{< ref supported-name-resolution >}}). For example, the Kubernetes name resolution component uses the Kubernetes DNS service to resolve the location of other applications running in the cluster. Self-hosted machines can use the mDNS name resolution component. The Consul name resolution component can be used in any hosting environment, including Kubernetes or self-hosted. -## Example +## Example Architecture -Following the above call sequence, suppose you have the applications as described in the [hello world quickstart](https://github.com/dapr/quickstarts/blob/master/tutorials/hello-world/README.md), where a python app invokes a node.js app. In such a scenario, the python app would be "Service A" , and a Node.js app would be "Service B". +Following the above call sequence, suppose you have the applications as described in the [Hello World tutorial](https://github.com/dapr/quickstarts/blob/master/tutorials/hello-world/README.md), where a python app invokes a node.js app. In such a scenario, the python app would be "Service A" , and a Node.js app would be "Service B". The diagram below shows sequence 1-7 again on a local machine showing the API calls: @@ -104,17 +102,38 @@ The diagram below shows sequence 1-7 again on a local machine showing the API ca 1. The Node.js app has a Dapr app ID of `nodeapp`. The python app invokes the Node.js app's `neworder` method by POSTing `http://localhost:3500/v1.0/invoke/nodeapp/method/neworder`, which first goes to the python app's local Dapr sidecar. 2. Dapr discovers the Node.js app's location using name resolution component (in this case mDNS while self-hosted) which runs on your local machine. 3. Dapr forwards the request to the Node.js app's sidecar using the location it just received. -4. The Node.js app's sidecar forwards the request to the Node.js app. The Node.js app performs its business logic, logging the incoming message and then persist the order ID into Redis (not shown in the diagram) +4. The Node.js app's sidecar forwards the request to the Node.js app. The Node.js app performs its business logic, logging the incoming message and then persist the order ID into Redis (not shown in the diagram). 5. The Node.js app sends a response to the Python app through the Node.js sidecar. -6. Dapr forwards the response to the Python Dapr sidecar +6. Dapr forwards the response to the Python Dapr sidecar. 7. The Python app receives the response. -## Next steps +## Try out service invocation +### Quickstarts & tutorials +The Dapr docs contain multiple quickstarts that leverage the service invocation building block in different example architectures. To get a straight-forward understanding of the service invocation api and it's features we recommend starting with our quickstarts: + +| Quickstart/tutorial | Description | +| ------------------- | ----------- | +| [Service invocation quickstart]({{< ref serviceinvocation-quickstart.md >}}) | This quickstart gets you interacting directly with the service invocation building block. | +| [Hello world tutorial](https://github.com/dapr/quickstarts/blob/master/tutorials/hello-world/README.md) | This tutorial shows how to use both the service invocation and state management building blocks all running locally on your machine. | +| [Hello world kubernetes tutorial](https://github.com/dapr/quickstarts/blob/master/tutorials/hello-kubernetes/README.md) | This tutorial walks through using Dapr in kubernetes and covers both the service invocation and state management building blocks as well. | -- Follow these guides on: - - [How-to: Invoke services using HTTP]({{< ref howto-invoke-discover-services.md >}}) - - [How-To: Configure Dapr to use gRPC]({{< ref grpc >}}) - - [How-to: Invoke services using gRPC]({{< ref howto-invoke-services-grpc.md >}}) -- Try out the [hello world quickstart](https://github.com/dapr/quickstarts/blob/master/tutorials/hello-world/README.md) which shows how to use HTTP service invocation or try the samples in the [Dapr SDKs]({{< ref sdks >}}) -- Read the [service invocation API specification]({{< ref service_invocation_api.md >}}) -- Understand the [service invocation performance]({{< ref perf-service-invocation.md >}}) numbers + +### Start using service invocation directly in your app +Want to skip the quickstarts? Not a problem. You can try out the service invocation building block directly in your application to securely communicate with other services. After [Dapr is installed](https://docs.dapr.io/getting-started), you can begin using the service invocation API in the following ways. + +Invoke services using: +- **HTTP and gRPC service invocation** (recommended set up method) + - *HTTP* - Allows you to just add the `dapr-app-id` header and you're ready to get started. Read more on this here, [Invoke Services using HTTP.]({{< ref howto-invoke-discover-services.md >}}) + - *gRPC* - For gRPC based applications, the service invocation API is also available. Run the gRPC server, then invoke services using the Dapr CLI. Read more on this in [Configuring Dapr to use gRPC]({{< ref grpc >}}) and [Invoke services using gRPC]({{< ref howto-invoke-services-grpc.md >}}). +- **Direct call to the API** - In addition to proxying, there's also an option to directly call the service invocation API to invoke a GET endpoint. Just update your address URL to `localhost:` and you'll be able to directly call the API. You can also read more on this in the _Invoke Services using HTTP_ docs linked above under HTTP proxying. +- **SDKs** - If you're using a Dapr SDK, you can directly use service invocation through the SDK. Select the SDK you need and use the Dapr client to invoke a service. Read more on this in [Dapr SDKs]({{< ref sdks.md >}}). + + +For quick testing, try using the Dapr CLI for service invocation: +- **Dapr CLI command** - Once the Dapr CLI is set up, use `dapr invoke --method ` command along with the method flag and the method of interest. Read more on this in [Dapr CLI]({{< ref dapr-invoke.md >}}). + +## Next steps +- Read the [service invocation API specification]({{< ref service_invocation_api.md >}}). This reference guide for service invocation describes how to invoke methods on other services. +- Understand the [service invocation performance numbers]({{< ref perf-service-invocation.md >}}). +- Take a look at [observability]({{< ref monitoring.md >}}). Here you can dig into Dapr's monitoring tools like tracing, metrics and logging. +- Read up on our [security practices]({{< ref monitoring.md >}}) around mTLS encryption, token authentication, and endpoint authorization. diff --git a/daprdocs/content/en/developing-applications/building-blocks/state-management/howto-get-save-state.md b/daprdocs/content/en/developing-applications/building-blocks/state-management/howto-get-save-state.md index 48db160092e..fce19c3ff99 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/state-management/howto-get-save-state.md +++ b/daprdocs/content/en/developing-applications/building-blocks/state-management/howto-get-save-state.md @@ -210,38 +210,39 @@ dapr run --app-id orderprocessing --app-port 6001 --dapr-http-port 3601 --dapr-g {{% codetab %}} ```go -//dependencies +// dependencies import ( "context" "log" "math/rand" - "time" "strconv" + "time" + dapr "github.com/dapr/go-sdk/client" ) -//code +// code func main() { + const STATE_STORE_NAME = "statestore" + rand.Seed(time.Now().UnixMicro()) for i := 0; i < 10; i++ { - time.Sleep(5000) orderId := rand.Intn(1000-1) + 1 client, err := dapr.NewClient() - STATE_STORE_NAME := "statestore" if err != nil { panic(err) } defer client.Close() ctx := context.Background() - //Using Dapr SDK to save and get state - if err := client.SaveState(ctx, STATE_STORE_NAME, "order_1", []byte(strconv.Itoa(orderId))); err != nil { + err = client.SaveState(ctx, STATE_STORE_NAME, "order_1", []byte(strconv.Itoa(orderId)), nil) + if err != nil { panic(err) - } - result, err := client.GetState(ctx, STATE_STORE_NAME, "order_2") + } + result, err := client.GetState(ctx, STATE_STORE_NAME, "order_1", nil) if err != nil { panic(err) } - log.Println("Result after get: ") - log.Println(result) + log.Println("Result after get:", string(result.Value)) + time.Sleep(2 * time.Second) } } ``` @@ -932,4 +933,4 @@ Invoke-RestMethod -Method Post -ContentType 'application/json' -Body '{"keys":[" - Read the full [State API reference]({{< ref state_api.md >}}) - Try one of the [Dapr SDKs]({{< ref sdks >}}) -- Build a [stateful service]({{< ref howto-stateful-service.md >}}) \ No newline at end of file +- Build a [stateful service]({{< ref howto-stateful-service.md >}}) diff --git a/daprdocs/content/en/developing-applications/building-blocks/state-management/state-management-overview.md b/daprdocs/content/en/developing-applications/building-blocks/state-management/state-management-overview.md index 1bf78f979b5..e7dd518f12b 100644 --- a/daprdocs/content/en/developing-applications/building-blocks/state-management/state-management-overview.md +++ b/daprdocs/content/en/developing-applications/building-blocks/state-management/state-management-overview.md @@ -89,11 +89,11 @@ Transactional state stores can be used to store actor state. To specify which st ### Querying state There are two ways to query the state: - * Using the [state management query API]({{< ref "#state-query-api" >}}) provided in Dapr runtime. - * Querying state store [directly]({{< ref "#query-state-store-directly" >}}) with the store's native SDK. + * Using the state management query API provided in Dapr runtime. + * Querying state store directly with the store's native SDK. #### Query API -The query API provides a way of querying the key/value data saved using state management in state stores regardless of underlying database or storage technology. It is an optional state management API. Using the state management query API you can filter, sort and paginate the key/value data. For more details read [How-To: Query state]({{< ref howto-state-query-api.md >}}). +The [query API]({{< ref "reference/api/state_api.md#query-state" >}}) provides a way of querying the key/value data saved using state management in state stores regardless of underlying database or storage technology. It is an optional state management API. Using the state management query API you can filter, sort and paginate the key/value data. For more details read [How-To: Query state]({{< ref howto-state-query-api.md >}}). #### Querying state store directly Dapr saves and retrieves state values without any transformation. You can query and aggregate state directly from the [underlying state store]({{< ref query-state-store >}}). diff --git a/daprdocs/content/en/developing-applications/ides/vscode/vscode-how-to-debug-multiple-dapr-apps.md b/daprdocs/content/en/developing-applications/ides/vscode/vscode-how-to-debug-multiple-dapr-apps.md index 1b839d811c2..df5b1089fcf 100644 --- a/daprdocs/content/en/developing-applications/ides/vscode/vscode-how-to-debug-multiple-dapr-apps.md +++ b/daprdocs/content/en/developing-applications/ides/vscode/vscode-how-to-debug-multiple-dapr-apps.md @@ -65,7 +65,29 @@ In the case of the hello world quickstart, two applications are launched, each w } ``` -Each configuration requires a `request`, `type` and `name`. These parameters help VSCode identify the task configurations in the `.vscode/task.json` files. +If you're using ports other than the default ports baked into the code, set the `DAPR_HTTP_PORT` and `DAPR_GRPC_PORT` environment variables in the `launch.json` debug configuration. Match with the `httpPort` and `grpcPort` in the daprd `tasks.json`. For example, `launch.json`: + +```json +{ + // Set the non-default HTTP and gRPC ports + "env": { + "DAPR_HTTP_PORT": "3502", + "DAPR_GRPC_PORT": "50002" + }, +} +``` + +`tasks.json`: + +```json +{ + // Match with ports set in launch.json + "httpPort": 3502, + "grpcPort": 50002 +} +``` + +Each configuration requires a `request`, `type` and `name`. These parameters help VSCode identify the task configurations in the `.vscode/tasks.json` files. - `type` defines the language used. Depending on the language, it might require an extension found in the marketplace, such as the [Python Extension](https://marketplace.visualstudio.com/items?itemName=ms-python.python). - `name` is a unique name for the configuration. This is used for compound configurations when calling multiple configurations in your project. @@ -74,9 +96,9 @@ Each configuration requires a `request`, `type` and `name`. These parameters hel For more information on VSCode debugging parameters see [VS Code launch attributes](https://code.visualstudio.com/Docs/editor/debugging#_launchjson-attributes). -## Step 2: Configure task.json +## Step 2: Configure tasks.json -For each [task](https://code.visualstudio.com/docs/editor/tasks) defined in `.vscode/launch.json` , a corresponding task definition must exist in `.vscode/task.json`. +For each [task](https://code.visualstudio.com/docs/editor/tasks) defined in `.vscode/launch.json` , a corresponding task definition must exist in `.vscode/tasks.json`. For the quickstart, each service needs a task to launch a Dapr sidecar with the `daprd` type, and a task to stop the sidecar with `daprd-down`. The parameters `appId`, `httpPort`, `metricsPort`, `label` and `type` are required. Additional optional parameters are available, see the [reference table here](#daprd-parameter-table"). diff --git a/daprdocs/content/en/developing-applications/integrations/AWS/authenticating-aws.md b/daprdocs/content/en/developing-applications/integrations/AWS/authenticating-aws.md index fa75a2b94b2..2d0c1542259 100644 --- a/daprdocs/content/en/developing-applications/integrations/AWS/authenticating-aws.md +++ b/daprdocs/content/en/developing-applications/integrations/AWS/authenticating-aws.md @@ -8,19 +8,24 @@ aliases: - /developing-applications/integrations/authenticating/authenticating-aws/ --- -All Dapr components using various AWS services (DynamoDB, SQS, S3, etc) use a standardized set of attributes for configuration, these are described below. +All Dapr components using various AWS services (DynamoDB, SQS, S3, etc) use a standardized set of attributes for configuration. See [how the AWS SDK (which Dapr uses) handles credentials](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials). -[This article](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials) provides a good overview of how the AWS SDK (which Dapr uses) handles credentials +None of the following attributes are required, since you can configure the AWS SDK using the default provider chain, described in the link above. Test the component configuration and inspect the log output from the Dapr runtime to ensure that components initialize correctly. -None of the following attributes are required, since the AWS SDK may be configured using the default provider chain described in the link above. It's important to test the component configuration and inspect the log output from the Dapr runtime to ensure that components initialize correctly. +| Attribute | Description | +| --------- | ----------- | +| `region` | Which AWS region to connect to. In some situations (when running Dapr in self-hosted mode, for example) this flag can be provided by the environment variable `AWS_REGION`. Since Dapr sidecar injection doesn't allow configuring environment variables on the Dapr sidecar, it is recommended to always set the `region` attribute in the component spec. | +| `endpoint` | The endpoint is normally handled internally by the AWS SDK. However, in some situations it might make sense to set it locally - for example if developing against [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html). | +| `accessKey` | AWS Access key id. | +| `secretKey` | AWS Secret access key. Use together with `accessKey` to explicitly specify credentials. | +| `sessionToken` | AWS Session token. Used together with `accessKey` and `secretKey`. When using a regular IAM user's access key and secret, a session token is normally not required. | -- `region`: Which AWS region to connect to. In some situations (when running Dapr in self-hosted mode, for example) this flag can be provided by the environment variable `AWS_REGION`. Since Dapr sidecar injection doesn't allow configuring environment variables on the Dapr sidecar, it is recommended to always set the `region` attribute in the component spec. -- `endpoint`: The endpoint is normally handled internally by the AWS SDK. However, in some situations it might make sense to set it locally - for example if developing against [DynamoDB Local](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html). -- `accessKey`: AWS Access key id. -- `secretKey`: AWS Secret access key. Use together with `accessKey` to explicitly specify credentials. -- `sessionToken`: AWS Session token. Used together with `accessKey` and `secretKey`. When using a regular IAM user's access key and secret, a session token is normally not required. +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} ## Alternatives to explicitly specifying credentials in component manifest files + In production scenarios, it is recommended to use a solution such as [Kiam](https://github.com/uswitch/kiam) or [Kube2iam](https://github.com/jtblin/kube2iam). If running on AWS EKS, you can [link an IAM role to a Kubernetes service account](https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html), which your pod can use. All of these solutions solve the same problem: They allow the Dapr runtime process (or sidecar) to retrive credentials dynamically, so that explicit credentials aren't needed. This provides several benefits, such as automated key rotation, and avoiding having to manage secrets. @@ -28,37 +33,49 @@ All of these solutions solve the same problem: They allow the Dapr runtime proce Both Kiam and Kube2IAM work by intercepting calls to the [instance metadata service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html). ## Using instance role/profile when running in stand-alone mode on AWS EC2 + If running Dapr directly on an AWS EC2 instance in stand-alone mode, instance profiles can be used. Simply configure an iam role and [attach it to the instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html) for the ec2 instance, and Dapr should be able to authenticate to AWS without specifying credentials in the Dapr component manifest. ## Authenticating to AWS when running dapr locally in stand-alone mode + When running Dapr (or the Dapr runtime directly) in stand-alone mode, you have the option of injecting environment variables into the process like this (on Linux/MacOS: + ```bash FOO=bar daprd --app-id myapp ``` + If you have [configured named AWS profiles](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) locally , you can tell Dapr (or the Dapr runtime) which profile to use by specifying the "AWS_PROFILE" environment variable: ```bash AWS_PROFILE=myprofile dapr run... ``` + or + ```bash AWS_PROFILE=myprofile daprd... ``` + You can use any of the [supported environment variables](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html#envvars-list) to configure Dapr in this manner. On Windows, the environment variable needs to be set before starting the `dapr` or `daprd` command, doing it inline as shown above is not supported. ## Authenticating to AWS if using AWS SSO based profiles + If you authenticate to AWS using [AWS SSO](https://aws.amazon.com/single-sign-on/), some AWS SDKs (including the Go SDK) don't yet support this natively. There are several utilities you can use to "bridge the gap" between AWS SSO-based credentials, and "legacy" credentials, such as [AwsHelper](https://pypi.org/project/awshelper/) or [aws-sso-util](https://github.com/benkehoe/aws-sso-util). If using AwsHelper, start Dapr like this: + ```bash AWS_PROFILE=myprofile awshelper dapr run... ``` + or + ```bash AWS_PROFILE=myprofile awshelper daprd... ``` On Windows, the environment variable needs to be set before starting the `awshelper` command, doing it inline as shown above is not supported. +For more information, see [how the AWS SDK (which Dapr uses) handles credentials](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials). diff --git a/daprdocs/content/en/getting-started/install-dapr-cli.md b/daprdocs/content/en/getting-started/install-dapr-cli.md index 2bfab98c0e2..aef7cbca156 100644 --- a/daprdocs/content/en/getting-started/install-dapr-cli.md +++ b/daprdocs/content/en/getting-started/install-dapr-cli.md @@ -31,7 +31,7 @@ wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O #### Install without `sudo` -If you do not have access to the `sudo` command or your username is not in the `sudoers` file, you can install Dapr to an alternate directory via the `DAPR_INSTALL_DIR` environment variable. +If you do not have access to the `sudo` command or your username is not in the `sudoers` file, you can install Dapr to an alternate directory via the `DAPR_INSTALL_DIR` environment variable. This directory must already exist and be accessible by the current user. ```bash wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | DAPR_INSTALL_DIR="$HOME/dapr" /bin/bash @@ -96,7 +96,7 @@ arch -arm64 brew install dapr/tap/dapr-cli ``` #### Install without `sudo` -If you do not have access to the `sudo` command or your username is not in the `sudoers` file, you can install Dapr to an alternate directory via the `DAPR_INSTALL_DIR` environment variable. +If you do not have access to the `sudo` command or your username is not in the `sudoers` file, you can install Dapr to an alternate directory via the `DAPR_INSTALL_DIR` environment variable. This directory must already exist and be accessible by the current user. ```bash curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | DAPR_INSTALL_DIR="$HOME/dapr" /bin/bash diff --git a/daprdocs/content/en/getting-started/quickstarts/_index.md b/daprdocs/content/en/getting-started/quickstarts/_index.md index 39c106fcdd8..c8942f7c42f 100644 --- a/daprdocs/content/en/getting-started/quickstarts/_index.md +++ b/daprdocs/content/en/getting-started/quickstarts/_index.md @@ -23,10 +23,10 @@ Hit the ground running with our Dapr quickstarts, complete with code samples aim | Quickstarts | Description | | ----------- | ----------- | | [Service Invocation]({{< ref serviceinvocation-quickstart.md >}}) | Call a method on another service using the service invocation API. | -| [Publish and Subscribe]({{< ref pubsub-quickstart.md >}}) | Send and receive messages using the publish and subscribe API. | | [State Management]({{< ref statemanagement-quickstart.md >}}) | Create stateful applications using the state management API. | +| [Publish and Subscribe]({{< ref pubsub-quickstart.md >}}) | Send and receive messages using the publish and subscribe API. | +| [Secrets Management]({{< ref secrets-quickstart.md >}}) | Retrieve secrets in the application code from a configured secret store using the secrets management API. | | Bindings | Coming soon. | | Actors | Coming soon. | | Observability | Coming soon. | -| Secrets Management | Coming soon. | | Configuration | Coming soon. | \ No newline at end of file diff --git a/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md index 14edc3ac451..d7e25abbecc 100644 --- a/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/pubsub-quickstart.md @@ -2,7 +2,7 @@ type: docs title: "Quickstart: Publish and Subscribe" linkTitle: "Publish and Subscribe" -weight: 70 +weight: 73 description: "Get started with Dapr's Publish and Subscribe building block" --- diff --git a/daprdocs/content/en/getting-started/quickstarts/secrets-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/secrets-quickstart.md new file mode 100644 index 00000000000..fb02273fe01 --- /dev/null +++ b/daprdocs/content/en/getting-started/quickstarts/secrets-quickstart.md @@ -0,0 +1,578 @@ +--- +type: docs +title: "Quickstart: Secrets Management" +linkTitle: "Secrets Management" +weight: 74 +description: "Get started with Dapr's Secrets Management building block" +--- + +Dapr provides a dedicated secrets API that allows developers to retrieve secrets from a secrets store. In this quickstart, you: + +1. Run a microservice with a secret store component. +1. Retrieve secrets using the Dapr secrets API in the application code. + +Select your preferred language-specific Dapr SDK before proceeding with the Quickstart. + +{{< tabs "Python" "JavaScript" ".NET" "Java" "Go" >}} + +{{% codetab %}} + +### Pre-requisites + +For this example, you will need: + +- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started). +- [Python 3.7+ installed](https://www.python.org/downloads/). + +- [Docker Desktop](https://www.docker.com/products/docker-desktop) + + +### Step 1: Set up the environment + +Clone the [sample provided in the Quickstarts repo](https://github.com/dapr/quickstarts/tree/master/secrets_management). + +```bash +git clone https://github.com/dapr/quickstarts.git +``` + +### Step 2: Retrieve the secret + +In a terminal window, navigate to the `order-processor` directory. + +```bash +cd secrets_management/python/sdk/order-processor +``` + +Install the dependencies: + +```bash +pip3 install -r requirements.txt +``` + +Run the `order-processor` service alongside a Dapr sidecar. + +```bash +dapr run --app-id order-processor --components-path ../../../components/ -- python3 app.py +``` + +#### Behind the scenes + +**`order-processor` service** + +Notice how the `order-processor` service below points to: + +- The `DAPR_SECRET_STORE` defined in the `local-secret-store.yaml` component. +- The secret defined in `secrets.json`. + +```python +# app.py +DAPR_SECRET_STORE = 'localsecretstore' +SECRET_NAME = 'secret' +with DaprClient() as client: + secret = client.get_secret(store_name=DAPR_SECRET_STORE, key=SECRET_NAME) + logging.info('Fetched Secret: %s', secret.secret) +``` + +**`local-secret-store.yaml` component** + +`DAPR_SECRET_STORE` is defined in the `local-secret-store.yaml` component file, located in [secrets_management/components](https://github.com/dapr/quickstarts/blob/master/secrets_management/components/local-secret-store.yaml): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: localsecretstore + namespace: default +spec: + type: secretstores.local.file + version: v1 + metadata: + - name: secretsFile + value: secrets.json + - name: nestedSeparator + value: ":" +``` + +In the YAML file: + +- `metadata/name` is how your application references the component (called `DAPR_SECRET_STORE` in the code sample). +- `spec/metadata` defines the connection to the secret used by the component. + +**`secrets.json` file** + +`SECRET_NAME` is defined in the `secrets.json` file, located in [secrets_management/python/sdk/order-processor](https://github.com/dapr/quickstarts/blob/master/secrets_management/python/sdk/order-processor/secrets.json): + +```json +{ + "secret": "YourPasskeyHere" +} +``` + +### Step 3: View the order-processor outputs + +As specified in the application code above, the `order-processor` service retrieves the secret via the Dapr secret store and displays it in the console. + +Order-processor output: + +``` +== APP == INFO:root:Fetched Secret: {'secret': 'YourPasskeyHere'} +``` + +{{% /codetab %}} + + +{{% codetab %}} + +### Pre-requisites + +For this example, you will need: + +- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started). +- [Latest Node.js installed](https://nodejs.org/download/). + +- [Docker Desktop](https://www.docker.com/products/docker-desktop) + + +### Step 1: Set up the environment + +Clone the [sample provided in the Quickstarts repo](https://github.com/dapr/quickstarts/tree/master/secrets_management). + +```bash +git clone https://github.com/dapr/quickstarts.git +``` + +### Step 2: Retrieve the secret + +In a terminal window, navigate to the `order-processor` directory. + +```bash +cd secrets_management/javascript/sdk/order-processor +``` + +Install the dependencies: + +```bash +npm install +``` + +Run the `order-processor` service alongside a Dapr sidecar. + +```bash +dapr run --app-id order-processor --components-path ../../../components/ -- npm start +``` + +#### Behind the scenes + +**`order-processor` service** + +Notice how the `order-processor` service below points to: + +- The `DAPR_SECRET_STORE` defined in the `local-secret-store.yaml` component. +- The secret defined in `secrets.json`. + +```javascript +// index.js +const DAPR_SECRET_STORE = "localsecretstore"; +const SECRET_NAME = "secret"; + +async function main() { + // ... + const secret = await client.secret.get(DAPR_SECRET_STORE, SECRET_NAME); + console.log("Fetched Secret: " + JSON.stringify(secret)); +} +``` + +**`local-secret-store.yaml` component** + +`DAPR_SECRET_STORE` is defined in the `local-secret-store.yaml` component file, located in [secrets_management/components](https://github.com/dapr/quickstarts/blob/master/secrets_management/components/local-secret-store.yaml): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: localsecretstore + namespace: default +spec: + type: secretstores.local.file + version: v1 + metadata: + - name: secretsFile + value: secrets.json + - name: nestedSeparator + value: ":" +``` + +In the YAML file: + +- `metadata/name` is how your application references the component (called `DAPR_SECRET_STORE` in the code sample). +- `spec/metadata` defines the connection to the secret used by the component. + +**`secrets.json` file** + +`SECRET_NAME` is defined in the `secrets.json` file, located in [secrets_management/javascript/sdk/order-processor](https://github.com/dapr/quickstarts/blob/master/secrets_management/javascript/sdk/order-processor/secrets.json): + +```json +{ + "secret": "YourPasskeyHere" +} +``` + +### Step 3: View the order-processor outputs + +As specified in the application code above, the `order-processor` service retrieves the secret via the Dapr secret store and displays it in the console. + +Order-processor output: + +``` +== APP == +== APP == > order-processor@1.0.0 start +== APP == > node index.js +== APP == +== APP == Fetched Secret: {"secret":"YourPasskeyHere"} +``` + +{{% /codetab %}} + + +{{% codetab %}} + +### Pre-requisites + +For this example, you will need: + +- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started). +- [.NET SDK or .NET 6 SDK installed](https://dotnet.microsoft.com/download). + +- [Docker Desktop](https://www.docker.com/products/docker-desktop) + + +### Step 1: Set up the environment + +Clone the [sample provided in the Quickstarts repo](https://github.com/dapr/quickstarts/tree/master/secrets_management). + +```bash +git clone https://github.com/dapr/quickstarts.git +``` + +### Step 2: Retrieve the secret + +In a terminal window, navigate to the `order-processor` directory. + +```bash +cd secrets_management/csharp/sdk/order-processor +``` + +Install the dependencies: + +```bash +dotnet restore +dotnet build +``` + +Run the `order-processor` service alongside a Dapr sidecar. + +```bash +dapr run --app-id order-processor --components-path ../../../components/ -- dotnet run +``` + +#### Behind the scenes + +**`order-processor` service** + +Notice how the `order-processor` service below points to: + +- The `DAPR_SECRET_STORE` defined in the `local-secret-store.yaml` component. +- The secret defined in `secrets.json`. + +```csharp +// Program.cs +const string DAPR_SECRET_STORE = "localsecretstore"; +const string SECRET_NAME = "secret"; +var client = new DaprClientBuilder().Build(); + +var secret = await client.GetSecretAsync(DAPR_SECRET_STORE, SECRET_NAME); +var secretValue = string.Join(", ", secret); +Console.WriteLine($"Fetched Secret: {secretValue}"); +``` + +**`local-secret-store.yaml` component** + +`DAPR_SECRET_STORE` is defined in the `local-secret-store.yaml` component file, located in [secrets_management/components](https://github.com/dapr/quickstarts/blob/master/secrets_management/components/local-secret-store.yaml): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: localsecretstore + namespace: default +spec: + type: secretstores.local.file + version: v1 + metadata: + - name: secretsFile + value: secrets.json + - name: nestedSeparator + value: ":" +``` + +In the YAML file: + +- `metadata/name` is how your application references the component (called `DAPR_SECRET_NAME` in the code sample). +- `spec/metadata` defines the connection to the secret used by the component. + +**`secrets.json` file** + +`SECRET_NAME` is defined in the `secrets.json` file, located in [secrets_management/csharp/sdk/order-processor](https://github.com/dapr/quickstarts/blob/master/secrets_management/csharp/sdk/order-processor/secrets.json): + +```json +{ + "secret": "YourPasskeyHere" +} +``` + +### Step 3: View the order-processor outputs + +As specified in the application code above, the `order-processor` service retrieves the secret via the Dapr secret store and displays it in the console. + +Order-processor output: + +``` +== APP == Fetched Secret: [secret, YourPasskeyHere] +``` + +{{% /codetab %}} + + +{{% codetab %}} + +### Pre-requisites + +For this example, you will need: + +- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started). +- Java JDK 11 (or greater): + - [Oracle JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html#JDK11), or + - [OpenJDK](https://jdk.java.net/13/) +- [Apache Maven](https://maven.apache.org/install.html), version 3.x. + +- [Docker Desktop](https://www.docker.com/products/docker-desktop) + + +### Step 1: Set up the environment + +Clone the [sample provided in the Quickstarts repo](https://github.com/dapr/quickstarts/tree/master/secrets_management). + +```bash +git clone https://github.com/dapr/quickstarts.git +``` + +### Step 2: Retrieve the secret + +In a terminal window, navigate to the `order-processor` directory. + +```bash +cd secrets_management/java/sdk/order-processor +``` + +Install the dependencies: + +```bash +mvn clean install +``` + +Run the `order-processor` service alongside a Dapr sidecar. + +```bash +dapr run --app-id order-processor --components-path ../../../components/ -- java -jar target/order-processor-0.0.1-SNAPSHOT.jar +``` + +#### Behind the scenes + +**`order-processor` service** + +Notice how the `order-processor` service below points to: + +- The `DAPR_SECRET_STORE` defined in the `local-secret-store.yaml` component. +- The secret defined in `secrets.json`. + +```java +// OrderProcessingServiceApplication.java +private static final String SECRET_STORE_NAME = "localsecretstore"; +// ... + Map secret = client.getSecret(SECRET_STORE_NAME, "secret").block(); + System.out.println("Fetched Secret: " + secret); +``` + +**`local-secret-store.yaml` component** + +`DAPR_SECRET_STORE` is defined in the `local-secret-store.yaml` component file, located in [secrets_management/components](https://github.com/dapr/quickstarts/blob/master/secrets_management/components/local-secret-store.yaml): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: localsecretstore + namespace: default +spec: + type: secretstores.local.file + version: v1 + metadata: + - name: secretsFile + value: secrets.json + - name: nestedSeparator + value: ":" +``` + +In the YAML file: + +- `metadata/name` is how your application references the component (called `DAPR_SECRET_NAME` in the code sample). +- `spec/metadata` defines the connection to the secret used by the component. + +**`secrets.json` file** + +`SECRET_NAME` is defined in the `secrets.json` file, located in [secrets_management/python/sdk/order-processor](https://github.com/dapr/quickstarts/blob/master/secrets_management/java/sdk/order-processor/secrets.json): + +```json +{ + "secret": "YourPasskeyHere" +} +``` + +### Step 3: View the order-processor outputs + +As specified in the application code above, the `order-processor` service retrieves the secret via the Dapr secret store and displays it in the console. + +Order-processor output: + +``` +== APP == Fetched Secret: {secret=YourPasskeyHere} +``` + +{{% /codetab %}} + + +{{% codetab %}} + +### Pre-requisites + +For this example, you will need: + +- [Dapr CLI and initialized environment](https://docs.dapr.io/getting-started). +- [Latest version of Go](https://go.dev/dl/). + +- [Docker Desktop](https://www.docker.com/products/docker-desktop) + + +### Step 1: Set up the environment + +Clone the [sample provided in the Quickstarts repo](https://github.com/dapr/quickstarts/tree/master/secrets_management). + +```bash +git clone https://github.com/dapr/quickstarts.git +``` + +### Step 2: Retrieve the secret + +In a terminal window, navigate to the `order-processor` directory. + +```bash +cd secrets_management/go/sdk/order-processor +``` + +Install the dependencies: + +```bash +go build app.go +``` + +Run the `order-processor` service alongside a Dapr sidecar. + +```bash +dapr run --app-id order-processor --components-path ../../../components/ -- go run app.go +``` + +#### Behind the scenes + +**`order-processor` service** + +Notice how the `order-processor` service below points to: + +- The `DAPR_SECRET_STORE` defined in the `local-secret-store.yaml` component. +- The secret defined in `secrets.json`. + +```go +const DAPR_SECRET_STORE = "localsecretstore" + const SECRET_NAME = "secret" + // ... + secret, err := client.GetSecret(ctx, DAPR_SECRET_STORE, SECRET_NAME, nil) + if secret != nil { + fmt.Println("Fetched Secret: ", secret[SECRET_NAME]) + } +``` + +**`local-secret-store.yaml` component** + +`DAPR_SECRET_STORE` is defined in the `local-secret-store.yaml` component file, located in [secrets_management/components](https://github.com/dapr/quickstarts/blob/master/secrets_management/components/local-secret-store.yaml): + +```yaml +apiVersion: dapr.io/v1alpha1 +kind: Component +metadata: + name: localsecretstore + namespace: default +spec: + type: secretstores.local.file + version: v1 + metadata: + - name: secretsFile + value: secrets.json + - name: nestedSeparator + value: ":" +``` + +In the YAML file: + +- `metadata/name` is how your application references the component (called `DAPR_SECRET_NAME` in the code sample). +- `spec/metadata` defines the connection to the secret used by the component. + +**`secrets.json` file** + +`SECRET_NAME` is defined in the `secrets.json` file, located in [secrets_management/python/sdk/order-processor](https://github.com/dapr/quickstarts/blob/master/secrets_management/java/sdk/order-processor/secrets.json): + +```json +{ + "secret": "YourPasskeyHere" +} +``` + +### Step 3: View the order-processor outputs + +As specified in the application code above, the `order-processor` service retrieves the secret via the Dapr secret store and displays it in the console. + +Order-processor output: + +``` +== APP == Fetched Secret: YourPasskeyHere +``` + +{{% /codetab %}} + +{{< /tabs >}} + +## Tell us what you think! + +We're continuously working to improve our Quickstart examples and value your feedback. Did you find this Quickstart helpful? Do you have suggestions for improvement? + +Join the discussion in our [discord channel](https://discord.gg/22ZtJrNe). + +## Next steps + +- Use Dapr Secrets Management with HTTP instead of an SDK. + - [Python](https://github.com/dapr/quickstarts/tree/master/secrets_management/python/http) + - [JavaScript](https://github.com/dapr/quickstarts/tree/master/secrets_management/javascript/http) + - [.NET](https://github.com/dapr/quickstarts/tree/master/secrets_management/csharp/http) + - [Java](https://github.com/dapr/quickstarts/tree/master/secrets_management/java/http) + - [Go](https://github.com/dapr/quickstarts/tree/master/secrets_management/go/http) +- Learn more about the [Secrets Management building block]({{< ref secrets-overview >}}) + +{{< button text="Explore Dapr tutorials >>" page="getting-started/tutorials/_index.md" >}} diff --git a/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md index dc9bd48e348..af2066c16df 100644 --- a/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/serviceinvocation-quickstart.md @@ -2,7 +2,7 @@ type: docs title: "Quickstart: Service Invocation" linkTitle: "Service Invocation" -weight: 70 +weight: 71 description: "Get started with Dapr's Service Invocation building block" --- @@ -420,7 +420,7 @@ mvn clean install Run the `order-processor` service alongside a Dapr sidecar. ```bash -dapr run --app-id order-processor --app-port 6001 --app-protocol http --dapr-http-port 3501 -- java -jar target/OrderProcessingService-0.0.1-SNAPSHOT.jar +dapr run --app-id order-processor --app-port 9001 --app-protocol http --dapr-http-port 3501 -- java -jar target/OrderProcessingService-0.0.1-SNAPSHOT.jar ``` ```java diff --git a/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md b/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md index 8c1428d6cc0..b605bf3f07f 100644 --- a/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md +++ b/daprdocs/content/en/getting-started/quickstarts/statemanagement-quickstart.md @@ -2,8 +2,8 @@ type: docs title: "Quickstart: State Management" linkTitle: "State Management" -weight: 70 -description: "Get started with Dapr's State Store" +weight: 72 +description: "Get started with Dapr's State Management building block" --- Let's take a look at Dapr's [State Management building block]({{< ref state-management >}}). In this Quickstart, you will save, get, and delete state using a Redis state store, but you can swap this out for any one of the [supported state stores]({{< ref supported-state-stores.md >}}). diff --git a/daprdocs/content/en/operations/components/component-schema.md b/daprdocs/content/en/operations/components/component-schema.md index 09fd461ea29..0e4ab7822d7 100644 --- a/daprdocs/content/en/operations/components/component-schema.md +++ b/daprdocs/content/en/operations/components/component-schema.md @@ -67,6 +67,9 @@ spec: value: "false" ``` +The consumerID metadata values can also contain a `{podName}` tag that is replaced with the Kubernetes POD's name when the Dapr sidecar starts up. This can be used to have a persisted behavior where the ConsumerID does not change on restart when using StatefulSets in Kubernetes. + + ## Further reading - [Components concept]({{< ref components-concept.md >}}) - [Reference secrets in component definitions]({{< ref component-secrets.md >}}) diff --git a/daprdocs/content/en/operations/components/setup-secret-store.md b/daprdocs/content/en/operations/components/setup-secret-store.md index 13145d8b281..a95147af6e5 100644 --- a/daprdocs/content/en/operations/components/setup-secret-store.md +++ b/daprdocs/content/en/operations/components/setup-secret-store.md @@ -55,6 +55,10 @@ spec: value: "[aws_session_token]" ``` +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Apply the configuration Once you have created the component's YAML file, follow these instructions to apply it based on your hosting environment: diff --git a/daprdocs/content/en/operations/configuration/increase-request-size.md b/daprdocs/content/en/operations/configuration/increase-request-size.md index 118dfaa3a89..2faadecf085 100644 --- a/daprdocs/content/en/operations/configuration/increase-request-size.md +++ b/daprdocs/content/en/operations/configuration/increase-request-size.md @@ -1,7 +1,7 @@ --- type: docs title: "How-To: Handle large http body requests" -linkTitle: "Http request body size" +linkTitle: "HTTP request body size" weight: 6000 description: "Configure http requests that are bigger than 4 MB" --- diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-hybrid-clusters.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-hybrid-clusters.md index f74b5410bbe..ec57289a4f4 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-hybrid-clusters.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-hybrid-clusters.md @@ -3,10 +3,10 @@ type: docs title: "Deploy to hybrid Linux/Windows Kubernetes clusters" linkTitle: "Hybrid clusters" weight: 60000 -description: "How to run Dapr apps on Kubernetes clusters with windows nodes" +description: "How to run Dapr apps on Kubernetes clusters with Windows nodes" --- -Dapr supports running on kubernetes clusters with windows nodes. You can run your Dapr microservices exclusively on Windows, exclusively on Linux, or a combination of both. This is helpful to users who may be doing a piecemeal migration of a legacy application into a Dapr Kubernetes cluster. +Dapr supports running on Kubernetes clusters with Windows nodes. You can run your Dapr microservices exclusively on Windows, exclusively on Linux, or a combination of both. This is helpful to users who may be doing a piecemeal migration of a legacy application into a Dapr Kubernetes cluster. Kubernetes uses a concept called node affinity so that you can denote whether you want your application to be launched on a Linux node or a Windows node. When deploying to a cluster which has both Windows and Linux nodes, you must provide affinity rules for your applications, otherwise the Kubernetes scheduler might launch your application on the wrong type of node. @@ -38,7 +38,7 @@ If you are installing using the Dapr CLI or via a helm chart, simply follow the Affinity will be automatically set for `kubernetes.io/os=linux`. This will be sufficient for most users, as Kubernetes requires at least one Linux node pool. -> **Note:** Dapr control plane containers are built and tested for both windows and linux, however, we generally recommend using the linux control plane containers. They tend to be smaller and have a much larger user base. +> **Note:** Dapr control plane containers are built and tested for both Windows and Linux, however, we generally recommend using the Linux control plane containers. They tend to be smaller and have a much larger user base. If you understand the above, but want to deploy the Dapr control plane to Windows, you can do so by setting: @@ -101,7 +101,8 @@ In order to launch a Dapr application on Windows, you'll first need to create a ``` ### Linux applications -If you have already got a dapr application with runs on Linux, you'll still need to add affinity rules as above, but choose linux affinity instead. + +If you already have a Dapr application that runs on Linux, you'll still need to add affinity rules as above, but choose Linux affinity instead. 1. Create a deployment YAML diff --git a/daprdocs/content/en/operations/support/support-release-policy.md b/daprdocs/content/en/operations/support/support-release-policy.md index fe096916f65..48a647af68c 100644 --- a/daprdocs/content/en/operations/support/support-release-policy.md +++ b/daprdocs/content/en/operations/support/support-release-policy.md @@ -47,7 +47,8 @@ The table below shows the versions of Dapr releases that have been tested togeth | Apr 7th 2022 | 1.7.0
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported | | Apr 20th 2022 | 1.7.1
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported | | Apr 22th 2022 | 1.7.2
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported | -| May 17th 2022 | 1.7.3
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported (current) | +| May 17th 2022 | 1.7.3
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported | +| May 31st 2022 | 1.7.4
| 1.7.0 | Java 1.5.0
Go 1.4.0
PHP 1.1.0
Python 1.6.0
.NET 1.7.0
JS 2.1.0 | 0.10.0 | Supported (current) | ## Upgrade paths After the 1.0 release of the runtime there may be situations where it is necessary to explicitly upgrade through an additional release to reach the desired target. For example an upgrade from v1.0 to v1.2 may need go pass through v1.1 @@ -62,12 +63,12 @@ General guidance on upgrading can be found for [self hosted mode]({{}}). Only required if `authRequired` is set to `"true"`. | `""`, `"KeFg23!"` | -| initialOffset | N | The initial offset to use if no offset was previously committed. Should be "newest" or "oldest". Defaults to "newest". | `"oldest"` | +| initialOffset | N | Input | The initial offset to use if no offset was previously committed. Should be "newest" or "oldest". Defaults to "newest". | `"oldest"` | | maxMessageBytes | N | Input/Output | The maximum size in bytes allowed for a single Kafka message. Defaults to 1024. | `2048` | -| version | N | Kafka cluster version. Defaults to 1.0.0 | `1.0.0` +| version | N | Input/Output | Kafka cluster version. Defaults to 1.0.0 | `1.0.0` ## Binding support @@ -88,6 +90,10 @@ curl -X POST http://localhost:3500/v1.0/bindings/myKafka \ }' ``` +### Response + +An HTTP 204 (No Content) and empty body will be returned if successful. + ## Related links - [Basic schema for a Dapr component]({{< ref component-schema >}}) diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/kinesis.md b/daprdocs/content/en/reference/components-reference/supported-bindings/kinesis.md index 1e5aa5ceeb0..f2367429f7a 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/kinesis.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/kinesis.md @@ -55,6 +55,10 @@ The above example uses secrets as plain strings. It is recommended to use a secr | secretKey | Y | Output | The AWS Secret Access Key to access this resource | `"secretAccessKey"` | | sessionToken | N | Output | The AWS session token to use | `"sessionToken"` | +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Binding support This component supports both **input and output** binding interfaces. diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/s3.md b/daprdocs/content/en/reference/components-reference/supported-bindings/s3.md index 07079b7897d..ec5d0fe7b31 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/s3.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/s3.md @@ -67,6 +67,10 @@ The above example uses secrets as plain strings. It is recommended to use a secr | disableSSL | N | Output | Allows to connect to non `https://` endpoints. Defaults to `false` | `true`, `false` | | insecureSSL | N | Output | When connecting to `https://` endpoints, accepts invalid or self-signed certificates. Defaults to `false` | `true`, `false` | +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ### Using with Minio [Minio](https://min.io/) is a service that exposes local storage as S3-compatible block storage, and it's a popular alternative to S3 especially in development environments. You can use the S3 binding with Minio too, with some configuration tweaks: diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/ses.md b/daprdocs/content/en/reference/components-reference/supported-bindings/ses.md index 95a8386b00f..cec0010a685 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/ses.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/ses.md @@ -61,7 +61,9 @@ The above example uses secrets as plain strings. It is recommended to use a secr | emailBcc | N | Output | If set, this specifies email address to BCC in. See [also](#example-request) | `"me@example.com"` | | subject | N | Output | If set, this specifies the subject of the email message. See [also](#example-request) | `"subject of mail"` | - +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} ## Binding support diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/sns.md b/daprdocs/content/en/reference/components-reference/supported-bindings/sns.md index fe062fff475..948da3643f0 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/sns.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/sns.md @@ -50,6 +50,10 @@ The above example uses secrets as plain strings. It is recommended to use a secr | secretKey | Y | Output | The AWS Secret Access Key to access this resource | `"secretAccessKey"` | | sessionToken | N | Output | The AWS session token to use | `"sessionToken"` | +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Binding support This component supports **output binding** with the following operations: diff --git a/daprdocs/content/en/reference/components-reference/supported-bindings/sqs.md b/daprdocs/content/en/reference/components-reference/supported-bindings/sqs.md index 3625025f371..039374561fb 100644 --- a/daprdocs/content/en/reference/components-reference/supported-bindings/sqs.md +++ b/daprdocs/content/en/reference/components-reference/supported-bindings/sqs.md @@ -50,6 +50,9 @@ The above example uses secrets as plain strings. It is recommended to use a secr | secretKey | Y | Input/Output | The AWS Secret Access Key to access this resource | `"secretAccessKey"` | | sessionToken | N | Input/Output | The AWS session token to use | `"sessionToken"` | +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} ## Binding support diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-aws-snssqs.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-aws-snssqs.md index 91db8d928d5..05fdb2c4712 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-aws-snssqs.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-aws-snssqs.md @@ -90,6 +90,9 @@ The above example uses secrets as plain strings. It is recommended to use a secr * Be aware that specifying `fifoMessageGroupID` limits the number of concurrent consumers of the FIFO queue used to only one but guarantees global ordering of messages published by the app's Dapr sidecars. See [this](https://aws.amazon.com/blogs/compute/solving-complex-ordering-challenges-with-amazon-sqs-fifo-queues/) post to better understand the topic of Message Group IDs and FIFO queues. * Since v1.8.0, the component supports the `"parallel"` `concurrencyMode` as its default mode. In prior versions, the component default behavior was calling the subscriber a single message at a time and waiting for its response. +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} ## Create an SNS/SQS instance diff --git a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-inmemory.md b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-inmemory.md index 6ce033b3e24..add1f9aa3eb 100644 --- a/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-inmemory.md +++ b/daprdocs/content/en/reference/components-reference/supported-pubsub/setup-inmemory.md @@ -20,8 +20,11 @@ metadata: spec: type: pubsub.in-memory version: v1 + metadata: [] ``` +> Note: in-memory does not require any specific metadata for the component to work, however spec.metadata is a required field. + ## Related links - [Basic schema for a Dapr component]({{< ref component-schema >}}) in the Related links section - Read [this guide]({{< ref "howto-publish-subscribe.md#step-2-publish-a-topic" >}}) for instructions on configuring pub/sub components diff --git a/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-parameter-store.md b/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-parameter-store.md index a241391ce0c..8532851ef8e 100644 --- a/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-parameter-store.md +++ b/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-parameter-store.md @@ -44,6 +44,11 @@ The above example uses secrets as plain strings. It is recommended to use a loca | accessKey | Y | The AWS Access Key to access this resource | `"key"` | | secretKey | Y | The AWS Secret Access Key to access this resource | `"secretAccessKey"` | | sessionToken | N | The AWS session token to use | `"sessionToken"` | + +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Create an AWS SSM Parameter Store instance Setup AWS SSM Parameter Store using the AWS documentation: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html. diff --git a/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-secret-manager.md b/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-secret-manager.md index 16c6ebd21bc..375c51f6444 100644 --- a/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-secret-manager.md +++ b/daprdocs/content/en/reference/components-reference/supported-secret-stores/aws-secret-manager.md @@ -44,6 +44,11 @@ The above example uses secrets as plain strings. It is recommended to use a loca | accessKey | Y | The AWS Access Key to access this resource | `"key"` | | secretKey | Y | The AWS Secret Access Key to access this resource | `"secretAccessKey"` | | sessionToken | N | The AWS session token to use | `"sessionToken"` | + +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Create an AWS Secrets Manager instance Setup AWS Secrets Manager using the AWS documentation: https://docs.aws.amazon.com/secretsmanager/latest/userguide/tutorials_basic.html. diff --git a/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-dynamodb.md b/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-dynamodb.md index f18a6481c28..9ec0f02b465 100644 --- a/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-dynamodb.md +++ b/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-dynamodb.md @@ -57,6 +57,10 @@ In order to use DynamoDB as a Dapr state store, the table must have a primary ke | sessionToken | N |AWS session token to use. A session token is only required if you are using temporary security credentials. | `"TOKEN"` | ttlAttributeName | N |The table attribute name which should be used for TTL. | `"expiresAt"` +{{% alert title="Important" color="warning" %}} +When running the Dapr sidecar (daprd) with your application on EKS (AWS Kubernetes), if you're using a node/pod that has already been attached to an IAM policy defining access to AWS resources, you **must not** provide AWS access-key, secret-key, and tokens in the definition of the component spec you're using. +{{% /alert %}} + ## Setup AWS DynamoDB See [Authenticating to AWS]({{< ref authenticating-aws.md >}}) for information about authentication-related attributes diff --git a/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-redis.md b/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-redis.md index b2c352ea8ee..d4a91c10588 100644 --- a/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-redis.md +++ b/daprdocs/content/en/reference/components-reference/supported-state-stores/setup-redis.md @@ -150,9 +150,10 @@ We can use [Helm](https://helm.sh/) to quickly create a Redis instance in our Ku {{< /tabs >}} -### Querying JSON objects +## Querying JSON objects (optional) + +In addition to supporting storing and querying state data as key/value pairs, the Redis state store optionally supports querying of JSON objects to meet more complex querying or filtering requirements. To enable this feature, the following steps are required: -The Redis state store supports querying of JSON objects. To enable this feature, the following steps are required: 1. The Redis store must support Redis modules and specifically both Redisearch and RedisJson. If you are deploying and running Redis then load [redisearch](https://oss.redis.com/redisearch/) and [redisjson](https://oss.redis.com/redisjson/) modules when deploying the Redis service. `` 2. Specify `queryIndexes` entry in the metadata of the component config. The value of the `queryIndexes` is a JSON array of the following format: diff --git a/daprdocs/layouts/shortcodes/dapr-latest-version.html b/daprdocs/layouts/shortcodes/dapr-latest-version.html index 3bc389cd215..10881d1e8bd 100644 --- a/daprdocs/layouts/shortcodes/dapr-latest-version.html +++ b/daprdocs/layouts/shortcodes/dapr-latest-version.html @@ -1 +1 @@ -{{- if .Get "short" }}1.7{{ else if .Get "long" }}1.7.3{{ else if .Get "cli" }}1.7.0{{ else }}1.7.3{{ end -}} +{{- if .Get "short" }}1.7{{ else if .Get "long" }}1.7.4{{ else if .Get "cli" }}1.7.0{{ else }}1.7.4{{ end -}} diff --git a/daprdocs/static/images/observability-opentelemetry-collector.png b/daprdocs/static/images/observability-opentelemetry-collector.png index 855e51974c8..6e137ad80bc 100644 Binary files a/daprdocs/static/images/observability-opentelemetry-collector.png and b/daprdocs/static/images/observability-opentelemetry-collector.png differ diff --git a/daprdocs/static/images/observability-sidecar.png b/daprdocs/static/images/observability-sidecar.png index f37926aa52d..0cd3f9edaf7 100644 Binary files a/daprdocs/static/images/observability-sidecar.png and b/daprdocs/static/images/observability-sidecar.png differ diff --git a/daprdocs/static/images/observability-tracing.png b/daprdocs/static/images/observability-tracing.png index 553d4d8b93e..44e7bdafa89 100644 Binary files a/daprdocs/static/images/observability-tracing.png and b/daprdocs/static/images/observability-tracing.png differ diff --git a/daprdocs/static/images/overview-sidecar-apis.png b/daprdocs/static/images/overview-sidecar-apis.png new file mode 100644 index 00000000000..417d381b275 Binary files /dev/null and b/daprdocs/static/images/overview-sidecar-apis.png differ diff --git a/daprdocs/static/images/pubsub-overview-components.png b/daprdocs/static/images/pubsub-overview-components.png index fb95c647dc2..cc05cec2c9c 100644 Binary files a/daprdocs/static/images/pubsub-overview-components.png and b/daprdocs/static/images/pubsub-overview-components.png differ diff --git a/daprdocs/static/images/security-overview-capabilities-example.png b/daprdocs/static/images/security-overview-capabilities-example.png index 927ed29a9f5..0dbde1fc306 100644 Binary files a/daprdocs/static/images/security-overview-capabilities-example.png and b/daprdocs/static/images/security-overview-capabilities-example.png differ diff --git a/daprdocs/static/images/service-invocation-overview-example.png b/daprdocs/static/images/service-invocation-overview-example.png index 594b7875f56..28e2d79f78f 100644 Binary files a/daprdocs/static/images/service-invocation-overview-example.png and b/daprdocs/static/images/service-invocation-overview-example.png differ diff --git a/daprdocs/static/presentations/Dapr-Diagrams.pptx.zip b/daprdocs/static/presentations/Dapr-Diagrams.pptx.zip index bf380ae44ce..ce99f37ac19 100644 Binary files a/daprdocs/static/presentations/Dapr-Diagrams.pptx.zip and b/daprdocs/static/presentations/Dapr-Diagrams.pptx.zip differ