diff --git a/docs/develop/python/error-handling.mdx b/docs/develop/python/error-handling.mdx new file mode 100644 index 0000000000..952258680a --- /dev/null +++ b/docs/develop/python/error-handling.mdx @@ -0,0 +1,576 @@ +--- +id: error-handling +title: Error handling - Python SDK +sidebar_label: Error handling +description: Learn how to handle errors in Temporal Python applications with retry policies, idempotent Activities, and recovery patterns. +slug: /develop/python/error-handling +toc_max_heading_level: 2 +keywords: + - error handling + - exceptions + - failures + - retry policy + - idempotence + - saga pattern +tags: + - Activities + - Workflows + - Errors + - Python SDK +--- + +Temporal automatically handles many types of failures through retries and Durable Execution. +This page shows you how to build on these capabilities to create robust error handling for your applications. + +**Key concepts:** + +Not all failures should be handled the same way. +**Transient failures** (like brief network hiccups) resolve on their own and should be retried immediately. +**Intermittent failures** (like rate limiting) need increasing delays between retries. +**Permanent failures** (like invalid input) won't resolve through retries and need different data or code changes. + +Temporal distinguishes between **Workflow Task failures** (bugs that can be fixed with redeployment) and **Workflow Execution failures** (business logic failures that should stop the Workflow). +Task failures retry automatically so you can fix and redeploy without losing state. +Execution failures require you to explicitly raise an `ApplicationError`. + +This page shows how to: + +- [Make Activities idempotent](#make-activities-idempotent) +- [Raise exceptions from Activities](#raise-exceptions-from-activities) +- [Raise exceptions from Workflows](#raise-exceptions-from-workflows) +- [Handle exceptions in Workflows](#handle-exceptions-in-workflows) +- [Configure custom Retry Policies](#configure-custom-retry-policies) +- [Mark specific errors as non-retryable](#mark-errors-as-non-retryable) +- [Specify non-retryable error types in Retry Policies](#specify-non-retryable-error-types) +- [Implement rollback logic with the Saga pattern](#implement-saga-pattern) +- [Understand Temporal's failure types](#understand-failure-types) + +## Make Activities idempotent {#make-activities-idempotent} + +**How to make Activities idempotent using the Temporal Python SDK** + +Because Activities may be retried due to failures, it's strongly recommended to make them idempotent. +An idempotent operation produces the same result whether executed once or multiple times. + +Activities follow an at-least-once execution model. +If a Worker executes an Activity successfully but crashes before notifying the Temporal Service, the Activity will be retried. +Without idempotence, this could cause duplicate charges in payment processing or create duplicate resources in infrastructure provisioning. + +### Use idempotency keys + +Most external services support idempotency keys—unique identifiers that prevent duplicate operations. +When the service receives a request with a key it has already processed, it returns the original result instead of performing the operation again. + +Create an idempotency key by combining the Workflow Run ID and Activity ID: + +```python +from temporalio import activity + +@activity.defn +async def process_payment(amount: float, account: str): + info = activity.info() + idempotency_key = f"{info.workflow_run_id}-{info.activity_id}" + + # Pass idempotency_key to your payment service + result = await payment_service.charge( + amount=amount, + account=account, + idempotency_key=idempotency_key + ) + return result +``` + +This value remains constant across Activity retries but is unique among all Workflow Executions. + +### Design Activities to be atomic + +Activities are atomic—they either complete successfully or not. +If an Activity performs multiple steps and the last step fails, the entire Activity is retried. + +Consider this Activity: +1. Look up data in database +2. Call microservice with the data +3. Write result to filesystem + +If step 3 fails, all three steps execute again on retry. +You might split this into three separate Activities so only the failed step retries, but balance this against having a larger Event History with more Activity Executions. + +## Raise exceptions from Activities {#raise-exceptions-from-activities} + +**How to raise exceptions from Activities using the Temporal Python SDK** + +Use `ApplicationError` to communicate application-specific failures from Activities. +Temporal converts any Python exception raised in an Activity to an `ApplicationError`, but raising it explicitly gives you more control. + +```python +from temporalio import activity +from temporalio.exceptions import ApplicationError + +@activity.defn +async def validate_charge(credit_card_number: str, amount: float): + if not is_valid_card(credit_card_number): + raise ApplicationError( + f"Invalid credit card number: {credit_card_number}", + type="InvalidCreditCard", + ) + + if amount <= 0: + raise ApplicationError( + f"Amount must be positive, got {amount}", + type="InvalidAmount", + ) + + return True +``` + +When raising an `ApplicationError`: +- Provide a descriptive `message` +- Optionally provide a `type` string to categorize the failure +- The error appears in the Event History as an `ActivityTaskFailed` event + +When an Activity fails, Temporal wraps the exception in an `ActivityError` before surfacing it to the Workflow. +The `ActivityError` provides context including: +- Activity type that failed +- Number of retry attempts +- Original cause (the `ApplicationError` you raised, or `TimeoutError`, `CancelledError`, etc.) + +## Raise exceptions from Workflows {#raise-exceptions-from-workflows} + +**How to raise exceptions from Workflows using the Temporal Python SDK** + +The behavior depends on what exception you raise: + +### Fail a Workflow Execution + +To deliberately fail a Workflow Execution, raise an `ApplicationError`: + +```python +from temporalio import workflow +from temporalio.exceptions import ApplicationError + +@workflow.defn +class PizzaDeliveryWorkflow: + @workflow.run + async def run(self, order): + distance = await workflow.execute_activity( + calculate_distance, + order.address, + start_to_close_timeout=timedelta(seconds=10) + ) + + if order.is_delivery and distance.kilometers > 25: + workflow.logger.error("Customer outside service area") + raise ApplicationError( + "Customer lives outside the service area", + type="CustomerOutsideServiceArea" + ) + + # Continue with order... +``` + +This puts the Workflow Execution in "Failed" state with no automatic retries. +Use this for permanent failures where retrying won't help—like the customer being too far away. + +### Trigger a Workflow Task retry + +Raising any other Python exception (like `ValueError` or `TypeError`) causes a Workflow Task failure, which retries automatically: + +```python +# This causes a Workflow Task failure (retries automatically) +raise ValueError("Unexpected condition") +``` + +This is intentional. +Regular Python exceptions are treated as bugs that can be fixed with a code deployment, not business logic failures. +The Workflow Task retries indefinitely, letting you fix the bug and redeploy without losing Workflow state. + +## Handle exceptions in Workflows {#handle-exceptions-in-workflows} + +**How to handle exceptions in Workflows using the Temporal Python SDK** + +Use Python's `try/except` blocks to handle Activity failures in your Workflow: + +```python +from temporalio import workflow +from temporalio.exceptions import ActivityError, ApplicationError +from datetime import timedelta + +@workflow.defn +class MoneyTransferWorkflow: + @workflow.run + async def run(self, details): + # Withdraw money + try: + withdraw_result = await workflow.execute_activity( + withdraw, + details, + start_to_close_timeout=timedelta(seconds=10) + ) + except ActivityError as e: + raise ApplicationError( + f"Withdrawal failed: {e.cause}", + type="WithdrawalError" + ) + + # Deposit money + try: + deposit_result = await workflow.execute_activity( + deposit, + details, + start_to_close_timeout=timedelta(seconds=10) + ) + except ActivityError as e: + # Deposit failed - attempt refund + try: + await workflow.execute_activity( + refund, + withdraw_result, + start_to_close_timeout=timedelta(seconds=10) + ) + raise ApplicationError( + f"Deposit failed but money refunded to source account", + type="DepositError" + ) + except ActivityError as refund_err: + raise ApplicationError( + f"Deposit failed and refund also failed: {refund_err.cause}", + type="CriticalTransferError" + ) + + return f"Transfer complete: {withdraw_result}, {deposit_result}" +``` + +Common Temporal exceptions you can catch in Workflows: +- `ActivityError` - Activity failed after exhausting retries +- `ChildWorkflowError` - Child Workflow failed +- `CancelledError` - Workflow, Activity, or Timer was canceled +- `TimeoutError` - Operation exceeded timeout + +If these exceptions propagate unhandled, the Workflow Execution fails (or enters "Canceled" state for `CancelledError`). + +## Configure custom Retry Policies {#configure-custom-retry-policies} + +**How to configure custom Retry Policies using the Temporal Python SDK** + +Activities have a default Retry Policy with unlimited attempts and exponential backoff. +Customize this to match your expected failure patterns. + +```python +from temporalio import workflow +from temporalio.common import RetryPolicy +from datetime import timedelta + +@workflow.defn +class OrderWorkflow: + @workflow.run + async def run(self, order): + # Custom retry for rate-limited service + retry_policy = RetryPolicy( + initial_interval=timedelta(seconds=10), + backoff_coefficient=3.0, + maximum_interval=timedelta(minutes=5), + maximum_attempts=20, + ) + + result = await workflow.execute_activity( + call_external_service, + order, + start_to_close_timeout=timedelta(seconds=30), + retry_policy=retry_policy, + ) + return result +``` + +Retry Policy attributes: +- **`initial_interval`**: Delay before first retry (default: 1 second) +- **`backoff_coefficient`**: Multiplier for subsequent delays (default: 2.0) +- **`maximum_interval`**: Cap on retry delay (default: 100× initial interval) +- **`maximum_attempts`**: Maximum retry attempts (default: unlimited) +- **`non_retryable_error_types`**: Error types that shouldn't retry (default: empty) + +### Match your Retry Policy to failure types + +**For transient failures** (brief network issues): Use the defaults or a low `initial_interval` and `backoff_coefficient`. + +**For intermittent failures** (rate limiting): Increase `initial_interval` and `backoff_coefficient` to space out retries and let the condition resolve. + +**For cost-sensitive APIs**: Set `maximum_attempts` to limit retries (rare—usually prefer timeouts). + +### Use different policies for different Activities + +You can use different Retry Policies for different Activities, or even multiple policies for the same Activity: + +```python +fast_retry = RetryPolicy( + initial_interval=timedelta(seconds=1), + backoff_coefficient=1.5, +) + +slow_retry = RetryPolicy( + initial_interval=timedelta(seconds=30), + backoff_coefficient=3.0, +) + +# Same Activity, different policies +await workflow.execute_activity( + process_order, + order, + start_to_close_timeout=timedelta(seconds=10), + retry_policy=fast_retry, +) + +# Later, with different circumstances... +await workflow.execute_activity( + process_order, + order, + start_to_close_timeout=timedelta(seconds=10), + retry_policy=slow_retry, +) +``` + +### Don't use Workflow Retry Policies + +Unlike Activities, Workflows don't retry by default, and you usually shouldn't add a Retry Policy. +Workflows are deterministic and not designed for failure-prone operations. +A Workflow failure typically indicates a code bug or bad input data—retrying the entire Workflow repeats the same logic without fixing the underlying issue. + +If you need retry logic for specific Workflow operations, implement it in your Workflow code rather than using a Workflow Retry Policy. + +## Mark specific errors as non-retryable {#mark-errors-as-non-retryable} + +**How to mark specific errors as non-retryable using the Temporal Python SDK** + +Some failures are permanent and won't resolve through retries. +Mark these as non-retryable to fail fast instead of waiting for timeouts. + +Set the `non_retryable` flag when raising an `ApplicationError`: + +```python +from temporalio import activity +from temporalio.exceptions import ApplicationError + +@activity.defn +async def process_payment(card_number: str, amount: float): + if not is_valid_card_format(card_number): + # Invalid format will never become valid through retries + raise ApplicationError( + f"Invalid credit card format: {card_number}", + type="InvalidCardFormat", + non_retryable=True, + ) + + if amount <= 0: + # Invalid amount won't be fixed by retrying + raise ApplicationError( + f"Amount must be positive: {amount}", + type="InvalidAmount", + non_retryable=True, + ) + + # Process payment... +``` + +An `ApplicationError` with `non_retryable=True` will never retry, regardless of the Retry Policy. + +Use non-retryable errors for: +- Invalid input data that prevents the Activity from proceeding +- Business rule violations +- Authorization failures + +**Use this sparingly.** +In most cases, it's better to let the Retry Policy handle when to stop retrying based on time or attempts. + +## Specify non-retryable error types {#specify-non-retryable-error-types} + +**How to specify non-retryable error types in Retry Policies using the Temporal Python SDK** + +Sometimes you want the Workflow (caller) to decide which error types shouldn't retry, rather than the Activity (implementer). + +List error types that shouldn't retry in your Retry Policy: + +```python +from temporalio import workflow +from temporalio.common import RetryPolicy +from datetime import timedelta + +@workflow.defn +class CheckoutWorkflow: + @workflow.run + async def run(self, payment_details): + retry_policy = RetryPolicy( + non_retryable_error_types=[ + "InvalidCardFormat", + "InsufficientFunds", + "AccountClosed", + ] + ) + + try: + result = await workflow.execute_activity( + process_payment, + payment_details, + start_to_close_timeout=timedelta(seconds=30), + retry_policy=retry_policy, + ) + return result + except ActivityError as e: + workflow.logger.error(f"Payment failed: {e.cause}") + # Handle the non-retryable error... +``` + +When an Activity raises an `ApplicationError`, Temporal checks if its `type` is in `non_retryable_error_types`. +If it matches, the Activity fails immediately without retries. + +### When to use each approach + +**`non_retryable=True` in the Activity**: Use when the Activity implementer knows the error is permanently unrecoverable. +This enforces the constraint for all callers. + +**`non_retryable_error_types` in the Retry Policy**: Use when the caller wants to decide which errors are unrecoverable based on their business logic. +This lets different Workflows make different decisions about the same Activity. + +## Implement rollback logic with the Saga pattern {#implement-saga-pattern} + +**How to implement the Saga pattern using the Temporal Python SDK** + +The Saga pattern coordinates a sequence of operations where each operation has a compensating action to undo its effects. +If any operation fails, execute compensating actions in reverse order to roll back previous operations. + +Use this for multi-step processes like: +- E-commerce checkout (payment, inventory, shipping) +- Distributed transactions across services +- Multi-stage data updates + +```python +from temporalio import workflow +from temporalio.exceptions import ActivityError +from datetime import timedelta + +@workflow.defn +class OrderWorkflow: + @workflow.run + async def run(self, order): + compensations = [] + + try: + # Reserve inventory + compensations.append({ + "activity": revert_inventory, + "input": order + }) + await workflow.execute_activity( + reserve_inventory, + order, + start_to_close_timeout=timedelta(seconds=10), + ) + + # Charge payment + compensations.append({ + "activity": refund_payment, + "input": order + }) + payment_id = await workflow.execute_activity( + charge_payment, + order, + start_to_close_timeout=timedelta(seconds=10), + ) + + # Create shipment + compensations.append({ + "activity": cancel_shipment, + "input": payment_id + }) + shipment_id = await workflow.execute_activity( + create_shipment, + order, + start_to_close_timeout=timedelta(seconds=10), + ) + + return {"payment_id": payment_id, "shipment_id": shipment_id} + + except ActivityError as e: + workflow.logger.error(f"Order failed: {e.cause}, rolling back...") + + # Execute compensations in reverse order + for compensation in reversed(compensations): + try: + await workflow.execute_activity( + compensation["activity"], + compensation["input"], + start_to_close_timeout=timedelta(seconds=10), + ) + except ActivityError as comp_err: + # Log compensation failure but continue with others + workflow.logger.error(f"Compensation failed: {comp_err.cause}") + + # Re-raise the original error + raise ApplicationError( + f"Order failed: {e.cause}", + type="OrderFailed" + ) +``` + +Key points: +- Add compensating actions to a list **before** executing each Activity +- Use `reversed(compensations)` to undo operations in the correct order +- Handle compensation failures gracefully (they might fail too) +- Temporal manages all state and retry logic, making Saga implementation straightforward + +## Understand Temporal's failure types {#understand-failure-types} + +Temporal uses specialized exception types to represent different failure scenarios. +All exceptions inherit from [`TemporalError`](https://python.temporal.io/temporalio.exceptions.TemporalError.html). + +**Do not extend `TemporalError` or its children.** +Use the provided exception types to ensure: +- Consistent behavior across process and language boundaries +- Compatibility with the Temporal Service +- Proper serialization via Protocol Buffers + +### Common failure types + +**`ApplicationError`**: Raised by your code to indicate application-specific failures. +This is the only Temporal exception you should raise manually. +When you raise an `ApplicationError`, you can optionally provide a `type` string and mark it as `non_retryable`. + +**`ActivityError`**: Wraps exceptions raised from Activities. +The `cause` field contains the original error (`ApplicationError`, `TimeoutError`, `CancelledError`, etc.). +Catch this in Workflows to handle Activity failures. + +**`TimeoutError`**: Occurs when an Activity or Workflow exceeds its configured timeout. + +**`CancelledError`**: Results from cancellation of a Workflow, Activity, or Timer. +You can catch and ignore this to continue execution despite cancellation. + +**`TerminatedError`**: Occurs when a Workflow Execution is forcefully terminated. + +**`ChildWorkflowError`**: Raised when a Child Workflow Execution fails. + +**`WorkflowAlreadyStartedError`**: Raised when attempting to start a Workflow with an ID that's already running. + +**`ServerError`**: Used for exceptions from the Temporal Service itself (like database failures). + +### Workflow Task vs Workflow Execution failures + +**Workflow Task failures** occur when Workflow code raises a non-Temporal exception (like `ValueError`, `TypeError`, or non-determinism errors). +These retry automatically, letting you fix bugs and redeploy without losing Workflow state. + +**Workflow Execution failures** occur when Workflow code raises a Temporal exception like `ApplicationError`. +These put the Workflow in "Failed" state with no automatic retries. + +Example of a permanent failure that should fail the Workflow: + +```python +if distance.kilometers > MAX_DELIVERY_DISTANCE: + # Retrying won't change the distance - this is permanent + raise ApplicationError( + "Customer lives outside service area", + type="OutsideServiceArea" + ) +``` + +### Protecting sensitive information + +The default Failure Converter copies exception messages and stack traces as plain text visible in the Web UI. +If your exceptions might contain sensitive information, configure a custom Failure Converter to encrypt this data. +See the [Securing Application Data course](https://learn.temporal.io/courses/appdatasec/) for details. diff --git a/docs/production-deployment/cloud/get-started/api-keys.mdx b/docs/production-deployment/cloud/get-started/api-keys.mdx index 35e157084a..960c1fcb3e 100644 --- a/docs/production-deployment/cloud/get-started/api-keys.mdx +++ b/docs/production-deployment/cloud/get-started/api-keys.mdx @@ -341,21 +341,28 @@ Workflow access secure. ## API keys for Namespace authentication {#namespace-authentication} Create a Namespace with API key authentication as an alternative to mTLS-based authentication by selecting "Allow API -key authentication" during setup. The gRPC endpoint format for the Namespace depends on the authentication method: +key authentication" during setup. The gRPC endpoint format for the Namespace depends on the authentication method and +whether or not High Availability features are enabled. -- For API key connections, use the gRPC regional endpoint `..api.temporal.io:7233`. - -Use this gRPC endpoint in the Temporal CLI or SDK to connect to Temporal Cloud with an API key. +See the following documentation for [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information. :::info -For [Namespaces with High Availability features](/cloud/high-availability) with API key authentication enabled, use the -gRPC Namespace endpoint: `..tmprl.cloud:7233`. This allows automated failover without needing to -switch endpoints. +When switching on or off High Availability features for a Namespace, you may need to update the gRPC endpoint used by +your Workers and Clients, because the Namespace endpoint changes based on whether High Availability features are +enabled. See [Disable High Availability](/cloud/high-availability/enable#disable) for more information. ::: -See the following documentation for [accessing Namespaces](/cloud/namespaces#access-namespaces) for more information. +### Without High Availability features + +Use the gRPC regional endpoint `..api.temporal.io:7233`. + +### With High Availability features + +Use the gRPC Namespace endpoint: `..tmprl.cloud:7233`. This allows Workers and Clients to always +connect to the active region of the Namespace. In a failover event, Temporal Cloud changes the Namespace's active region +and points this Namespace endpoint to the new active region. ## Use API keys to authenticate {#using-apikeys} diff --git a/docs/production-deployment/cloud/get-started/namespaces.mdx b/docs/production-deployment/cloud/get-started/namespaces.mdx index 4887c77824..d22358e6e2 100644 --- a/docs/production-deployment/cloud/get-started/namespaces.mdx +++ b/docs/production-deployment/cloud/get-started/namespaces.mdx @@ -2,7 +2,9 @@ id: namespaces title: Namespaces sidebar_label: Namespaces -description: A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique identifiers, and gRPC endpoints in Temporal Cloud. +description: + A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique + identifiers, and gRPC endpoints in Temporal Cloud. slug: /cloud/namespaces toc_max_heading_level: 4 keywords: @@ -19,7 +21,8 @@ tags: import { CaptionedImage, ZoomingImage } from '@site/src/components'; -A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique identifiers, and gRPC endpoints in Temporal Cloud. +A Namespace is a unit of isolation within Temporal Cloud, providing security boundaries, Workflow management, unique +identifiers, and gRPC endpoints in Temporal Cloud. - [Create a Namespace](#create-a-namespace) - [Access a Namespace](#access-namespaces) @@ -29,30 +32,31 @@ A Namespace is a unit of isolation within Temporal Cloud, providing security bou ## What is a Cloud Namespace Name? {#temporal-cloud-namespace-name} -A Cloud Namespace Name is a customer-supplied name for a [Namespace](/namespaces) in Temporal Cloud. -Each Namespace Name, such as `accounting-production`, is unique within the scope of a customer's account. -It cannot be changed after the Namespace is provisioned. +A Cloud Namespace Name is a customer-supplied name for a [Namespace](/namespaces) in Temporal Cloud. Each Namespace +Name, such as `accounting-production`, is unique within the scope of a customer's account. It cannot be changed after +the Namespace is provisioned. Each Namespace Name must conform to the following rules: - A Namespace Name must contain at least 2 characters and no more than 39 characters. -- A Namespace Name must begin with a letter, end with a letter or number, and contain only letters, numbers, and the hyphen (-) character. -- Each hyphen (-) character must be immediately preceded _and_ followed by a letter or number; consecutive hyphens are not permitted. +- A Namespace Name must begin with a letter, end with a letter or number, and contain only letters, numbers, and the + hyphen (-) character. +- Each hyphen (-) character must be immediately preceded _and_ followed by a letter or number; consecutive hyphens are + not permitted. - All letters in a Namespace Name must be lowercase. ## What is a Temporal Cloud Account ID? {#temporal-cloud-account-id} -A Temporal Cloud Account ID is a unique customer identifier assigned by Temporal Technologies. -Each Id is a short string of numbers and letters like `f45a2`, at least five characters long. -This account identifier is retained throughout the time each customer uses Temporal Cloud. +A Temporal Cloud Account ID is a unique customer identifier assigned by Temporal Technologies. Each Id is a short string +of numbers and letters like `f45a2`, at least five characters long. This account identifier is retained throughout the +time each customer uses Temporal Cloud. -At times you may need to know your customer Account ID. -Accessing the account's Namespaces provides an easy way to capture this information. -Each Temporal Namespace use an Account ID suffix. -This is the alphanumeric character string found after the period in any Temporal Cloud Namespace name. +At times you may need to know your customer Account ID. Accessing the account's Namespaces provides an easy way to +capture this information. Each Temporal Namespace use an Account ID suffix. This is the alphanumeric character string +found after the period in any Temporal Cloud Namespace name. -You can retrieve an Account ID from the [Temporal Cloud](https://cloud.temporal.io) Web UI or by using the `tcld` utility at a command line interface (CLI). -Follow these steps. +You can retrieve an Account ID from the [Temporal Cloud](https://cloud.temporal.io) Web UI or by using the `tcld` +utility at a command line interface (CLI). Follow these steps. @@ -123,26 +127,26 @@ Follow these steps. ## What is a Cloud Namespace Id? {#temporal-cloud-namespace-id} -A Cloud Namespace Id is a globally unique identifier for a [Namespace](/namespaces) in Temporal Cloud. -A Namespace Id is formed by concatenating the following: +A Cloud Namespace Id is a globally unique identifier for a [Namespace](/namespaces) in Temporal Cloud. A Namespace Id is +formed by concatenating the following: 1. A [Namespace Name](#temporal-cloud-namespace-name) 1. A period (.) 1. The [Account ID](#temporal-cloud-account-id) to which the Namespace belongs -For example, for the Account ID `123de` and Namespace Name `accounting-production`, the Namespace Id is `accounting-production.123de`. +For example, for the Account ID `123de` and Namespace Name `accounting-production`, the Namespace Id is +`accounting-production.123de`. ## What is a Cloud gRPC Endpoint? {#temporal-cloud-grpc-endpoint} -Temporal Clients communicate between application code and a Temporal Server by sending and receiving messages via the gRPC protocol. -gRPC is a Remote Procedure Call framework featuring low latency and high performance. -gRPC provides Temporal with an efficient, language-agnostic communication framework. +Temporal Clients communicate between application code and a Temporal Server by sending and receiving messages via the +gRPC protocol. gRPC is a Remote Procedure Call framework featuring low latency and high performance. gRPC provides +Temporal with an efficient, language-agnostic communication framework. -Every Temporal Namespace uses a gRPC endpoint for communication. -When migrating to Temporal Cloud, you'll need to switch the gRPC endpoint in your code from your current hosting, whether self-hosted or locally-hosted, to Temporal Cloud. +Every Temporal Namespace uses a gRPC endpoint for communication. When migrating to Temporal Cloud, you'll need to switch +the gRPC endpoint in your code from your current hosting, whether self-hosted or locally-hosted, to Temporal Cloud. -A gRPC endpoint appears on the detail page for each Cloud Namespace. -Follow these steps to find it: +A gRPC endpoint appears on the detail page for each Cloud Namespace. Follow these steps to find it: 1. Log into your account on [cloud.temporal.io](https://cloud.temporal.io/namespaces). 2. Navigate to the Namespace list page from the left-side vertical navigation. @@ -150,24 +154,27 @@ Follow these steps to find it: 4. On the Namespace detail page, click on the "Connect" button in the top right corner of the page. 5. Click the copy icon next to the gRPC address to copy it to your clipboard. -See [How to access a Namespace in Temporal Cloud](/cloud/namespaces/#access-namespaces) for more information on different gRPC endpoint types and how to access them. +See [How to access a Namespace in Temporal Cloud](/cloud/namespaces/#access-namespaces) for more information on +different gRPC endpoint types and how to access them. ## How to create a Namespace in Temporal Cloud {#create-a-namespace} :::info -The user who creates a [Namespace](/namespaces) is automatically granted [Namespace Admin](/cloud/users#namespace-level-permissions) permission for that Namespace. +The user who creates a [Namespace](/namespaces) is automatically granted +[Namespace Admin](/cloud/users#namespace-level-permissions) permission for that Namespace. -To create a Namespace, a user must have the Developer, Account Owner, or Global Admin account-level [Role](/cloud/users#account-level-roles). +To create a Namespace, a user must have the Developer, Account Owner, or Global Admin account-level +[Role](/cloud/users#account-level-roles). ::: :::tip -By default, each account is allocated with a limit of ten Namespaces. -As you start using Namespaces by scheduling Workflows, Temporal Cloud automatically raises your allowance. -This automatic adjustment happens whenever all your Namespaces are in use, up to a maximum of 100 Namespaces. -You can request further increases beyond the 100 Namespace limit by opening a [support ticket](/cloud/support#support-ticket). +By default, each account is allocated with a limit of ten Namespaces. As you start using Namespaces by scheduling +Workflows, Temporal Cloud automatically raises your allowance. This automatic adjustment happens whenever all your +Namespaces are in use, up to a maximum of 100 Namespaces. You can request further increases beyond the 100 Namespace +limit by opening a [support ticket](/cloud/support#support-ticket). ::: @@ -176,9 +183,13 @@ You can request further increases beyond the 100 Namespace limit by opening a [s To create a Namespace in Temporal Cloud, gather the following information: - [Namespace Name](/cloud/namespaces#temporal-cloud-namespace-name), region, and Cloud Provider -- [Retention Period](/temporal-service/temporal-server#retention-period) for the [Event History](/workflow-execution/event#event-history) of closed [Workflow Executions](/workflow-execution). -- [CA certificate](/cloud/certificates#certificate-requirements) for the Namespace, if you are using mTLS authentication. -- [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) to show decoded payloads to users in the Event History for Workflow Executions in the Namespace. For details, see [Securing your data](/production-deployment/data-encryption). +- [Retention Period](/temporal-service/temporal-server#retention-period) for the + [Event History](/workflow-execution/event#event-history) of closed [Workflow Executions](/workflow-execution). +- [CA certificate](/cloud/certificates#certificate-requirements) for the Namespace, if you are using mTLS + authentication. +- [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) to + show decoded payloads to users in the Event History for Workflow Executions in the Namespace. For details, see + [Securing your data](/production-deployment/data-encryption). - [Permissions](/cloud/users#namespace-level-permissions) for each user. @@ -187,22 +198,23 @@ To create a Namespace in Temporal Cloud, gather the following information: ### Create a Namespace using Temporal Cloud UI -1. Gather the information listed earlier in [Information needed to create a Namespace](#information-needed-to-create-a-namespace). +1. Gather the information listed earlier in + [Information needed to create a Namespace](#information-needed-to-create-a-namespace). 1. Go to the Temporal Cloud UI and log in. 1. On the left side of the window, click **Namespaces**. 1. On the **Namespaces** page, click **Create Namespace** in the upper-right portion of the window. 1. On the **Create Namespace** page in **Name**, enter the Namespace Name. 1. In **Cloud Provider**, select the cloud provider in which to host this Namespace. 1. In **Region**, select the region in which to host this Namespace. -1. In **Retention Period**, specify a value from 1 to 90 days. - When choosing this value, consider your needs for Event History versus the cost of maintaining that Event History. - Typically, a development Namespace has a short retention period and a production Namespace has a longer retention period. - (If you need to change this value later, contact [Temporal Support](/cloud/support#support-ticket).) +1. In **Retention Period**, specify a value from 1 to 90 days. When choosing this value, consider your needs for Event + History versus the cost of maintaining that Event History. Typically, a development Namespace has a short retention + period and a production Namespace has a longer retention period. (If you need to change this value later, contact + [Temporal Support](/cloud/support#support-ticket).) 1. Select your authentication method: [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates). 1. If using mTLS authentication, paste the CA certificate for this Namespace. -1. Optional: In **Codec Server**, enter the HTTPS URL (including the port number) of your Codec Server endpoint. - You may also enable "Pass the user access token with your endpoint" and "Include cross-origin credentials." - For details, see [Hosting your Codec Server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli). +1. Optional: In **Codec Server**, enter the HTTPS URL (including the port number) of your Codec Server endpoint. You may + also enable "Pass the user access token with your endpoint" and "Include cross-origin credentials." For details, see + [Hosting your Codec Server](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli). 1. Click **Create Namespace**. @@ -217,38 +229,39 @@ See the [`tcld` namespace create](/cloud/tcld/namespace/#create) command referen ## What are some Namespace best practices? {#best-practices} -This section provides general guidance for organizing [Namespaces](/namespaces) across use cases, services, applications, or domains. -Temporal Cloud provides Namespace–as-a-service, so the Namespace is the endpoint. -Customers should consider not only a Namespace naming convention but also how to group or isolate workloads using the Namespace as a boundary. +This section provides general guidance for organizing [Namespaces](/namespaces) across use cases, services, +applications, or domains. Temporal Cloud provides Namespace–as-a-service, so the Namespace is the endpoint. Customers +should consider not only a Namespace naming convention but also how to group or isolate workloads using the Namespace as +a boundary. -Each team can have their own Namespace for improved modularity, security, debugging, and fault isolation. -Namespaces contain the blast radius of misbehaving Workers that may exhaust rate limits. -Sensitive Workflow state (PCI data) can be secured with per-Namespace permissions and encrypted with a separate encryption key. +Each team can have their own Namespace for improved modularity, security, debugging, and fault isolation. Namespaces +contain the blast radius of misbehaving Workers that may exhaust rate limits. Sensitive Workflow state (PCI data) can be +secured with per-Namespace permissions and encrypted with a separate encryption key. -Temporal Applications in different Namespaces may be connected with [Nexus](/cloud/nexus) by exposing a clean service contract for others to use with built-in [Nexus access controls](/cloud/nexus/security). -Nexus supports cross-team, cross-domain, multi-region, and multi-cloud use cases. +Temporal Applications in different Namespaces may be connected with [Nexus](/cloud/nexus) by exposing a clean service +contract for others to use with built-in [Nexus access controls](/cloud/nexus/security). Nexus supports cross-team, +cross-domain, multi-region, and multi-cloud use cases. ### Constraints and limitations Before considering an appropriate Namespace configuration, you should be aware of the following constraints: -- By default, each account is allocated with a limit of ten Namespaces. - As you create and use your Namespaces, for example by scheduling Workflows, Temporal Cloud automatically raises your limit. - Our service identifies your usage patterns. - It responds by slowly increasing your allowance, up to 100 Namespaces. - You can request further increases beyond the 100 Namespace limit by opening a [support ticket](/cloud/support#support-ticket). -- Each Namespace has a rate limit, which is measured in Actions per second (APS). - A namespace may be throttled when its throughput becomes too high. - Throttling means limiting the rate at which actions are performed to prevent overloading the system. - A Namespace's default limit is set at 400 APS and automatically adjusts based on recent usage (over the prior 7 days). - Your APS limit will never fall below this default value. +- By default, each account is allocated with a limit of ten Namespaces. As you create and use your Namespaces, for + example by scheduling Workflows, Temporal Cloud automatically raises your limit. Our service identifies your usage + patterns. It responds by slowly increasing your allowance, up to 100 Namespaces. You can request further increases + beyond the 100 Namespace limit by opening a [support ticket](/cloud/support#support-ticket). +- Each Namespace has a rate limit, which is measured in Actions per second (APS). A namespace may be throttled when its + throughput becomes too high. Throttling means limiting the rate at which actions are performed to prevent overloading + the system. A Namespace's default limit is set at 400 APS and automatically adjusts based on recent usage (over the + prior 7 days). Your APS limit will never fall below this default value. - Each Namespace has a default service-level agreement (SLA) of 99.9% uptime. -- You can opt-in to using [High Availability features](https://docs.temporal.io/cloud/high-availability) with a 99.99% contractual SLA. -- A Namespace is a security isolation boundary. - Access to Temporal by [Worker Processes](/workers#worker-process) is permitted at the Namespace level. - Isolating applications or environments (development, test, staging, production) should take this into consideration. -- A Namespace is provisioned with an endpoint for executing your Workflows. - Accessing a Namespace from a Temporal Client requires [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication. +- You can opt-in to using [High Availability features](https://docs.temporal.io/cloud/high-availability) with a 99.99% + contractual SLA. +- A Namespace is a security isolation boundary. Access to Temporal by [Worker Processes](/workers#worker-process) is + permitted at the Namespace level. Isolating applications or environments (development, test, staging, production) + should take this into consideration. +- A Namespace is provisioned with an endpoint for executing your Workflows. Accessing a Namespace from a Temporal Client + requires [API keys](/cloud/api-keys) or [mTLS](/cloud/certificates) authentication. - [Workflow Id](/workflow-execution/workflowid-runid#workflow-id)uniqueness is per Namespace. - [Task Queue](/task-queue) names are unique per Namespace. - Closed Workflow retention is per Namespace. @@ -256,14 +269,13 @@ Before considering an appropriate Namespace configuration, you should be aware o ### General guidance -Namespace configuration requires some consideration. -Following are some general guidelines to consider. +Namespace configuration requires some consideration. Following are some general guidelines to consider. -- Namespaces are usually defined per use case. - A use case can encompass a broad range of Workflow types and a nearly unlimited scale of concurrent [Workflow Executions](/workflow-execution). +- Namespaces are usually defined per use case. A use case can encompass a broad range of Workflow types and a nearly + unlimited scale of concurrent [Workflow Executions](/workflow-execution). - Namespaces can be split along additional boundaries such as service, application, domain or even sub-domain. -- Environments such as production and development usually have requirements for isolation. - We recommend that each environment has its own Namespace. +- Environments such as production and development usually have requirements for isolation. We recommend that each + environment has its own Namespace. - Namespaces should be used to reduce the "blast radius" for mission-critical applications. - Workflows that need to communicate with each other should (for now) be in the same Namespace. - If you need to share Namespaces across team or domain boundaries, be sure to ensure the uniqueness of Workflow Ids. @@ -274,7 +286,8 @@ Following are some ideas about how to organize Namespaces. #### Example 1: Namespace per use case and environment -We recommend using one Namespace for each use case and environment combination for simple configurations in which multiple services and team or domain boundaries don't exist. +We recommend using one Namespace for each use case and environment combination for simple configurations in which +multiple services and team or domain boundaries don't exist. Sample naming convention: @@ -282,7 +295,8 @@ Sample naming convention: #### Example 2: Namespace per use case, service, and environment -We recommend using one Namespace for each use case, service, and environment combination when multiple services that are part of same use case communicate externally to Temporal via API (HTTP/gRPC). +We recommend using one Namespace for each use case, service, and environment combination when multiple services that are +part of same use case communicate externally to Temporal via API (HTTP/gRPC). Sample naming convention: @@ -290,10 +304,11 @@ Sample naming convention: #### Example 3: Namespace per use case, domain, and environment -We recommend using one namespace per use case, domain, and environment combination when multiple services that are part of the same use case need to communicate with each another via [Signals](/sending-messages#sending-signals) or by starting [Child Workflows](/child-workflows). -In this case, though, you must be mindful about Workflow Id uniqueness by prefixing each Workflow Id with a service-specific string. -The name of each Task Queue must also be unique. -If multiple teams are involved, the domain could also represent a team boundary. +We recommend using one namespace per use case, domain, and environment combination when multiple services that are part +of the same use case need to communicate with each another via [Signals](/sending-messages#sending-signals) or by +starting [Child Workflows](/child-workflows). In this case, though, you must be mindful about Workflow Id uniqueness by +prefixing each Workflow Id with a service-specific string. The name of each Task Queue must also be unique. If multiple +teams are involved, the domain could also represent a team boundary. Sample naming convention: @@ -307,31 +322,62 @@ Sample workflowId convention: {/* How to access a Namespace in Temporal Cloud */} -Temporal Cloud normally supports authentication to Namespaces using [API keys](/cloud/api-keys) _or_ [mTLS](/cloud/certificates). -If you need to migrate from one authentication method to another, or you require both API key and mTLS authentication to be enabled on your Namespace, please contact [Support](https://docs.temporal.io/cloud/support#support-ticket). +Temporal Cloud normally supports authentication to Namespaces using [API keys](/cloud/api-keys) _or_ +[mTLS](/cloud/certificates). If you need to migrate from one authentication method to another, or you require both API +key and mTLS authentication to be enabled on your Namespace, please contact +[Support](https://docs.temporal.io/cloud/support#support-ticket). -See the documentation for [API keys](/cloud/api-keys) and [mTLS certificates](/cloud/certificates) for more information on how to create and manage your credentials. +:::info + +Namespace authentication requiring both API key and mTLS is in +[pre-release](/evaluate/development-production-features/release-stages), and doesn't support +[High Availability features](/cloud/high-availability). + +::: + +See the documentation for [API keys](/cloud/api-keys) and [mTLS certificates](/cloud/certificates) for more information +on how to create and manage your credentials. + +Programmatically accessing your Namespace requires specific endpoints based on your authentication method. There are two +types of gRPC endpoints for accessing a Namespace in Temporal Cloud: + +- A namespace endpoint (`..tmprl.cloud:7233`) +- A regional endpoint (`..api.temporal.io:7233`). + +Which one to use depends on your authentication method and whether your Namespace has +[High Availability features](/cloud/high-availability) enabled, as shown in the table below. -Authentication methods require specific endpoints in order to programmatically access your Namespace. +| | Not High Availability | High Availability | +| ---------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| mTLS Authentication | Namespace | Namespace | +| API Key Authentication | Regional | Both work, but we reommend using the Namespace endpoint because it reduces the unavailability window during a failover event | -- For the API key authentication method, use the gRPC regional endpoint `..api.temporal.io:7233`. -- For the mTLS authentication method, use the gRPC Namespace endpoint `..tmprl.cloud:7233`. -- For [Namespaces with High Availability features](/cloud/high-availability) use the gRPC Namespace endpoint `..tmprl.cloud:7233`, regardless of your authentication method. - This allows automated failover without needing to switch your endpoint. +:::info + +When switching on or off High Availability features for a Namespace, you may need to update the gRPC endpoint used by +your clients and Workers, because the Namespace endpoint changes based on whether High Availability features are +enabled. See [Disable High Availability](/cloud/high-availability/enable#disable) for more information. + +::: For information on how to connect to Clients using a specific authentication method see the following documentation. -- To use API keys to connect with the [Temporal CLI](/cli), [Client SDK](/develop), [tcld](/cloud/tcld), [Cloud Ops API](/ops), and [Terraform](/production-deployment/cloud/terraform-provider), see [Use API keys to authenticate](/cloud/api-keys#using-apikeys). -- To use mTLS to connect with the [Temporal CLI](/cli) and [Client SDK](/develop), see [Configure Clients to use Client certificates](/cloud/certificates#configure-clients-to-use-client-certificates). +- To use API keys to connect with the [Temporal CLI](/cli), [Client SDK](/develop), [tcld](/cloud/tcld), + [Cloud Ops API](/ops), and [Terraform](/production-deployment/cloud/terraform-provider), see + [Use API keys to authenticate](/cloud/api-keys#using-apikeys). +- To use mTLS to connect with the [Temporal CLI](/cli) and [Client SDK](/develop), see + [Configure Clients to use Client certificates](/cloud/certificates#configure-clients-to-use-client-certificates). -For accessing the Temporal Web UI, use the HTTPS endpoint in the form: `https://cloud.temporal.io/namespaces/.`. -For example: `https://cloud.temporal.io/namespaces/accounting-production.f45a2`. +For accessing the Temporal Web UI, use the HTTPS endpoint in the form: +`https://cloud.temporal.io/namespaces/.`. For example: +`https://cloud.temporal.io/namespaces/accounting-production.f45a2`. -To ensure the security of your data, all traffic to and from your Namespace is encrypted. -However, for enhanced protection, you have additional options: +To ensure the security of your data, all traffic to and from your Namespace is encrypted. However, for enhanced +protection, you have additional options: -- (recommended) Set up private connectivity by [creating a ticket for Temporal Support](/cloud/support#support-ticket). -- Set up your allow list for outgoing network requests from your Clients and Workers with the IP address ranges of the Cloud Provider region in which your Namespace is located: +- (Recommended) Set up private connectivity by [creating a ticket for Temporal Support](/cloud/support#support-ticket). +- Set up your allow list for outgoing network requests from your Clients and Workers with the IP address ranges of the + Cloud Provider region in which your Namespace is located: - [AWS IP address ranges](https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html) - [GCP IP address ranges](https://cloud.google.com/compute/docs/faq#find_ip_range) @@ -345,7 +391,8 @@ To list Namespaces: - On the left side of the window, select **Namespaces**. -To edit a Namespace (including custom Search Attributes, certificates, certificate filters, Codec Server endpoint, permissions, and users), find the Namespace and do either of the following: +To edit a Namespace (including custom Search Attributes, certificates, certificate filters, Codec Server endpoint, +permissions, and users), find the Namespace and do either of the following: - On the right end of the Namespace row, select the three vertical dots (⋮). Click **Edit**. - Select the Namespace name. In the top-right portion of the page, select **Edit**. @@ -355,8 +402,10 @@ On the **Edit** page, you can do the following: - Add a [custom Search Attribute](/search-attribute#custom-search-attribute). - [Manage CA certificates](/cloud/certificates). - [Manage certificate filters](/cloud/certificates#manage-certificate-filters-using-temporal-cloud-ui). -- Set [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) for all users on the Namespace. - Each user on the Namespace has the option to [override this setting](/production-deployment/data-encryption#web-ui) in their browser. +- Set + [Codec Server endpoint](/production-deployment/data-encryption#set-your-codec-server-endpoints-with-web-ui-and-cli) + for all users on the Namespace. Each user on the Namespace has the option to + [override this setting](/production-deployment/data-encryption#web-ui) in their browser. - Manage [Namespace-level permissions](/cloud/users#namespace-level-permissions). - Add users. @@ -376,14 +425,16 @@ To list Namespaces and get information about them, use the following [tcld](/clo To manage certificates, use the [tcld namespace accepted-client-ca](/cloud/tcld/namespace/#accepted-client-ca) commands. For more information, see [How to manage certificates in Temporal Cloud](/cloud/certificates). -To manage certificate filters, use the [tcld namespace certificate-filters](/cloud/tcld/namespace/#certificate-filters) commands. -For more information, see [How to manage certificate filters in Temporal Cloud](/cloud/certificates#manage-certificate-filters). +To manage certificate filters, use the [tcld namespace certificate-filters](/cloud/tcld/namespace/#certificate-filters) +commands. For more information, see +[How to manage certificate filters in Temporal Cloud](/cloud/certificates#manage-certificate-filters). ## How to delete a Namespace in Temporal Cloud {#delete-a-namespace} :::info -To delete a Namespace, a user must have Namespace Admin [permission](/cloud/users#namespace-level-permissions) for that Namespace. +To delete a Namespace, a user must have Namespace Admin [permission](/cloud/users#namespace-level-permissions) for that +Namespace. ::: @@ -397,12 +448,12 @@ To delete a Namespace, a user must have Namespace Admin [permission](/cloud/user 1. In the **Delete Namespace** dialog, type `DELETE` to confirm the deletion of that Namespace. 1. Select **Delete**. -After deleting a Temporal Cloud Namespace, the Temporal Service immediately removes the Namespace's Workflow Executions and Task Queues. -Make sure all Workflows have been completed, canceled, or terminated before removing a Namespace. -The Namespace removal is permanent. +After deleting a Temporal Cloud Namespace, the Temporal Service immediately removes the Namespace's Workflow Executions +and Task Queues. Make sure all Workflows have been completed, canceled, or terminated before removing a Namespace. The +Namespace removal is permanent. -Closed Workflow Histories remain in Temporal storage until the user-defined retention period expires. -This period reflects the policy in effect when the Workflow Execution was closed. +Closed Workflow Histories remain in Temporal storage until the user-defined retention period expires. This period +reflects the policy in effect when the Workflow Execution was closed. For further questions or concerns, contact [Support](https://docs.temporal.io/cloud/support#support-ticket). @@ -412,8 +463,8 @@ See the [tcld namespace delete](/cloud/tcld/namespace/#delete) command reference ### Namespace deletion protection {#delete-protection} -To prevent accidental Namespace deletion, Temporal Cloud provides a protection feature. -When you enable Deletion Protection for your production environment Namespace, you ensure that critical data won't be deleted unintentionally. +To prevent accidental Namespace deletion, Temporal Cloud provides a protection feature. When you enable Deletion +Protection for your production environment Namespace, you ensure that critical data won't be deleted unintentionally. Follow these steps: @@ -424,12 +475,12 @@ Follow these steps: - Enable **Deletion Protection** -To enable or disable this feature using [`tcld`](/cloud/tcld), use the following command. -Set the value to `true` to enable or `false` to disable: +To enable or disable this feature using [`tcld`](/cloud/tcld), use the following command. Set the value to `true` to +enable or `false` to disable: ``` tcld namespace lifecycle set \ @@ -439,14 +490,16 @@ tcld namespace lifecycle set \ ## How to tag a Namespace in Temporal Cloud {#tag-a-namespace} -Tags are key-value metadata pairs that can be attached to namespaces in Temporal Cloud to help operators organize, track, and manage namespaces more easily. +Tags are key-value metadata pairs that can be attached to namespaces in Temporal Cloud to help operators organize, +track, and manage namespaces more easily. ### Tag Structure and Limits - Each namespace can have a maximum of 10 tags - Each key must be unique for a given namespace (e.g., a namespace cannot have both `team:foo` and `team:bar` tags) - Keys and values must be 1-63 characters in length -- Allowed characters: lowercase letters (`a-z`), numbers (`0-9`), periods (`.`), underscores (`_`), hyphens (`-`), and at signs (`@`) +- Allowed characters: lowercase letters (`a-z`), numbers (`0-9`), periods (`.`), underscores (`_`), hyphens (`-`), and + at signs (`@`) - Tags are not a secure storage mechanism and should not store PII or PHI - Tags will not change the behavior of the tagged resource - There is a soft limit of 1000 unique tag keys per account @@ -462,28 +515,25 @@ See the [tcld namespace tags](/cloud/tcld/namespace/#tags) command reference for ### Terraform -See the [Terraform provider](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/docs/resources/namespace_tags.md) for details. +See the +[Terraform provider](https://github.com/temporalio/terraform-provider-temporalcloud/blob/main/docs/resources/namespace_tags.md) +for details. ### Web UI -Tags can be viewed and managed through the Temporal Cloud web interface. When viewing a namespace, you'll see tags displayed and can add, edit, or remove them if you have the appropriate permissions. +Tags can be viewed and managed through the Temporal Cloud web interface. When viewing a namespace, you'll see tags +displayed and can add, edit, or remove them if you have the appropriate permissions. - + - + diff --git a/docs/production-deployment/cloud/high-availability/enable.mdx b/docs/production-deployment/cloud/high-availability/enable.mdx index a5dd6307c2..8701690d41 100644 --- a/docs/production-deployment/cloud/high-availability/enable.mdx +++ b/docs/production-deployment/cloud/high-availability/enable.mdx @@ -10,25 +10,34 @@ import { ToolTipTerm } from '@site/src/components'; :::tip Support, stability, and dependency info -Same-region Replication and Multi-cloud Replication are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview). +Same-region Replication and Multi-cloud Replication are in +[Public Preview](/evaluate/development-production-features/release-stages#public-preview). -Multi-region Replication is in [General Availability](/evaluate/development-production-features/release-stages#general-availability) +Multi-region Replication is in +[General Availability](/evaluate/development-production-features/release-stages#general-availability) ::: -You can enable High Availability features ([Single-region Replication](/cloud/high-availability#same-region-replication), [Multi-region Replication](/cloud/high-availability#multi-region-replication), or [Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication)) for a new or existing Namespace by adding a replica. -When you add a replica, Temporal Cloud begins asynchronously replicating ongoing and existing Workflow Executions. +You can enable High Availability features +([Single-region Replication](/cloud/high-availability#same-region-replication), +[Multi-region Replication](/cloud/high-availability#multi-region-replication), or +[Multi-cloud Replication](/cloud/high-availability#multi-cloud-replication)) for a new or existing Namespace by adding a +replica. When you add a replica, Temporal Cloud begins asynchronously replicating ongoing and existing Workflow +Executions. -Not all replication options are available in all regions. See the [region documentation](/cloud/regions) for the replication options available in each region. +Not all replication options are available in all regions. See the [region documentation](/cloud/regions) for the +replication options available in each region. -Using private network connectivity with a HA namespace requires extra setup. See [Connectivity for HA](/cloud/high-availability/ha-connectivity). +Using private network connectivity with a HA namespace requires extra setup. See +[Connectivity for HA](/cloud/high-availability/ha-connectivity). -There are charges associated with Replication and enabling High Availability features. -For pricing details, visit Temporal Cloud's [Pricing](/cloud/pricing) page. +There are charges associated with Replication and enabling High Availability features. For pricing details, visit +Temporal Cloud's [Pricing](/cloud/pricing) page. ## Create a Namespace with High Availability features {#create} -To create a new Namespace with High Availability features, you can use the Temporal Cloud UI or the tcld command line utility. +To create a new Namespace with High Availability features, you can use the Temporal Cloud UI or the tcld command line +utility. @@ -60,7 +69,8 @@ Specify the [region codes](/cloud/regions) as arguments to the two `--region` fl - Using the same region replicates to an isolation domain within that region. - Using a different region replicates across regions. -If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace create`. +If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before +`namespace create`. @@ -81,8 +91,8 @@ A replica can be added after a namespace has already been created. 1. Select the “Add a replica” button. 1. Choose the [region](/cloud/regions) for the replica. -The web interface will present an estimated time for replication to complete. -This time is based on your selection and the size and scale of the Workflows in your Namespace. +The web interface will present an estimated time for replication to complete. This time is based on your selection and +the size and scale of the Workflows in your Namespace. Temporal Cloud sends an email alert to all Namespace Admins once your Namespace replica is ready for use. @@ -98,9 +108,11 @@ tcld namespace add-region \ --region ``` -Specify the [region code](/cloud/regions) of the region where you want to create the replica as an argument to the `--region` flag. +Specify the [region code](/cloud/regions) of the region where you want to create the replica as an argument to the +`--region` flag. -If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace add-region`. +If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before +`namespace add-region`. Temporal Cloud sends an email alert once your Namespace is ready for use. @@ -113,30 +125,49 @@ Temporal Cloud sends an email alert once your Namespace is ready for use. :::caution We discourage changing the location of your replica for deployed applications, except under exceptional circumstances. -Changing the location of your Namespace replica will result in a mandatory 7-day waiting period before you can re-enable High Availability Namespace features. +Changing the location of your Namespace replica will result in a mandatory 7-day waiting period before you can re-enable +High Availability Namespace features. ::: -Temporal Cloud can't change replica locations directly. -To update the location, you need to remove the current replica and add a new one. -Follow these steps to change the replica location: +Temporal Cloud can't change replica locations directly. To update the location, you need to remove the current replica +and add a new one. Follow these steps to change the replica location: -1. [Remove your replica](#discontinuing). - This disables High Availability for your Namespace. +1. [Remove your replica](#disable). This disables High Availability for your Namespace. 2. Wait through the required 7-day waiting period. 3. [Add a new replica](#upgrade) to your Namespace. You will receive an email alert once your Namespace is ready for use. -## Discontinue High Availability features {#discontinuing} +## Disable High Availability (remove a replica) {#disable} -Removing a Namespace replica disables High Availability and automatic failover features. -Follow these steps to disable these features and end High Availability charges: +To disable High Availability features on a Namespace, remove the replica from that Namespace. Removing a replica +disables all High Availability features: + +- Discontinues replication of the Workflows in the Namespace. +- Disables the Namespace's ability to trigger a failover to a different region or cloud. +- For Workers and Clients that use API keys, removing a replica requires connecting to the Namespace using the published + [regional endpoint](/cloud/regions) for the Namespace's region. + - Disables connecting to the Namespace with API keys and the Namespace's endpoint or the replica region's regional + endpoint. +- Ends High Availability charges. + +:::caution + +After removing a Namespace's replica, you cannot re-enable replication on that same Temporal Cloud Namespace for seven +days. + +::: + +Follow these steps to remove a replica from a Namespace: +1. If you are using API keys for authentication on this Namespace, configure your Workers and Clients that use API keys + to [connect with the regional Temporal Cloud endpoint](/cloud/api-keys#namespace-authentication) for the Namespace's + primary region. 1. Navigate to the Namespace details page in Temporal Cloud 1. Select the option to "Remove Replica" on the "Region" card. @@ -144,26 +175,32 @@ Follow these steps to disable these features and end High Availability charges: -At the command line, enter: +First, if you are using API keys for authentication on this Namespace, configure your Workers and Clients that use API +keys to [connect with the regional Temporal Cloud endpoint](/cloud/api-keys#namespace-authentication) for the +Namespace's primary region. + +Then, run the following command to remove the replica: ``` tcld namespace delete-region \ + --api-key \ --namespace . \ --region ``` -If using API key authentication with the `--api-key` flag, you must add it directly after the tcld command and before `namespace delete-region` - -After following these instructions, Temporal Cloud deletes the replica. -Your Namespace will no longer use High Availability features and you will no longer be charged for this feature. +:::important -:::note +To remove a replica from a Namespace with API keys enabled, you need assistance from Temporal Support. Please +[contact support](/cloud/support#support-ticket) with the Namespace ID of the Namespace where you want to remove the +replica. You must confirm that Workers and Clients with API keys have been configured to connect to the Namespace using +the published [regional endpoint](/cloud/regions). -After removing a replica, Temporal Cloud can't re-enable replication for a given Namespace for seven days. +This safeguard ensures that Workers and Clients continue running uninterrupted once Temporal Support removes the +replica. After the replica is removed, if Workers and Clients with API keys attempt to use the Namespace endpoint or the +former replica's regional endpoint, their requests will fail. ::: -