Releases: temporalio/temporal
v1.27.1
Schema Changes
Before upgrading your Temporal Cluster to v1.27.1, you must upgrade your core schema if you are using MySQL or PostgreSQL, and your visibility schema to the following:
- Core:
- MySQL schema v1.16
- PostgreSQL schema v1.16
- Visibility:
- Elasticsearch schema v9
- MySQL schema 1.9
- PostgreSQL schema v1.9
Please see our upgrade documentation for the necessary steps to upgrade your schemas.
NOTE: The upgrade to MySQL and PostgreSQL Visibility schemas may come with temporary performance degradation because of creation of a new column _version
which has default values. Consider performing the schema upgrades when load is low. There are protective mechanisms in place to account for timeouts from any VisibilityStore.
Deprecation of Visibility Scan APIs
Visibility Scan APIs have been deprecated in favor of List Workflow APIs. Visibility Scan APIs will be removed in a future version. Migration to List Workflow APIs will be required in future versions.
Nexus GA
Nexus is now GA with a stable server API.
Read more here on how to disable Nexus or how to operate it here.
Notable features and bug fixes since v1.26.2
:
- The server now allows a maximum of 30 pending Nexus operations per workflow by default, as opposed to the previous limit being 30 total.
- Add support for attaching callbacks, request IDs, and links to a workflow via
StartWorkflowExecutionRequest.OnConflictOptions
withWorkflowIdConflictPolicy
ofUSE_EXISTING
. This allows multiple operations to be backed by the same workflow. - Misc small features and tests.
Safe Deploys
The following APIs are added for Worker Versioning. All APIs are experimental and not yet recommended for production usage. You need to set the dynamic configs system.enableDeployments
and system.enableDeploymentVersions
in order to use them.
- ListWorkerDeployments
- DescribeWorkerDeployment
- DescribeWorkerDeploymentVersion
- SetWorkerDeploymentCurrentVersion
- SetWorkerDeploymentRampingVersion
- UpdateWorkerVersionMetadata
- DeleteWorkerDeployment
- DeleteWorkerDeploymentVersion
The following APIs are now deprecated and replaced by above:
DescribeDeploymentListDeploymentsGetDeploymentReachabilityGetCurrentDeploymentSetCurrentDeployment
Activity Commands (pre-release)
Changes to the Activity Commands — a set of APIs designed to resolve issues related to activity execution. The following APIs where updated:
UpdateActivityOptionsById
was renamed toUpdateActivityOptions
. This API can be used by the client to update the options of an activity while activity is running.PauseActivityById
was renamed toPauseActivity
. With this API, If the Activity is not currently running (e.g. because it previously failed and is waiting for the next retry), it will not be run again until it is unpaused.activity_type
parameter was added. If this parameter is set - all running activities of this type will be paused.
UnpauseActivityById
was renamed toUnpauseActivity
. With this API clients can re-schedule a previously-paused Activity for execution.no_wait
parameter was removedactivity_type
parameter was added. If this parameter is set - all paused activities of this type will be unpaused.match-all
parameter was added. If this parameter is set - all paused activities will be unpaused.jitter
parameter was added. If set, the activity will start at a random time within the specified jitter duration.
ResetActivityById
was renamed toResetActivity
. With this API clients can reset the execution of an activity, specified by its ID or type.no_wait
parameter was removedactivity_type
parameter was added. If this parameter is provided - all paused activities of this type will be unpaused.keep_paused
parameter was added. If this parameter is provided - all paused activities will stay paused. Otherwise they will be unpaused.jitter
parameter was added. If set, the activity will start at a random time within the specified jitter duration.
- New batch operation was introduced -
BatchOperationUnpauseActivities
.
Workflow Reset with children
In this release we have added the functionality to reset a workflow with a pending child.
Prior to this release reseting to a point between child workflow initiated and child workflow completed was not supported (the reset operation would fail). In the current release the reset operation will allow this case and the behavior of the parent after reset is to reconnect to the running child. The new run of the parent will receive the child’s completion event and result (if any) from the child.
The feature is gated behind a per namespace boolean dynamic configuration AllowResetWithPendingChildren
which is enabled by default for all namespaces.
Note: If you are using Go-SDK and are relying on the SDK to generate child workflow IDs then you need to update it to the latest version to be able to use this feature. Other SDKs don’t need any upgrade to use this feature.
Delete namespace improvement
-
Delete workflow executions RPS is now dynamic. Previously,
frontend.deleteNamespaceDeleteActivityRPS
was read only once when namespace deletion started, and subsequent changes to this dynamic config didn't affect the ongoing deletion. This was inconvenient for large namespaces since the default RPS is only100
. Now the RPS can be adjusted on the fly.Please note: Since deletion of Workflow Execution is an asynchronous process, this RPS controls the rate at which delete execution tasks are created. Decreasing this value (for example, from
1000
to10
) won't immediately slow down the process, as existing tasks in the transfer queue must be processed first. -
DeleteExecutionsWorkflow
now supports astats
query to track its progress. Since this Workflow can run for hours after a namespace is marked as deleted, it was previously difficult to monitor how many Workflow Executions remained. The new query handler provides current statistics about total and remaining executions. -
Metrics and logging have been enhanced for better actionability. Key improvements include:
- Monitor
ReclaimResources
workflow failures using the metricsreclaim_resources_namespace_delete_failure
andreclaim_resources_delete_executions_failure
. - Track
DeleteExecutionsWorkflow
progress using the metricsdelete_executions_success
anddelete_executions_failure
.
- Monitor
-
Business critical namespaces can be protected from deletion. Use dynamic config to list namespaces which are not deletable:
worker.protectedNamespaces: - value: - critical_namespace - just_very_important_namespace
-
Sleep duration in
ReclaimResourcesWorkflow
now supports dynamic changes. If a namespace delete delay was mistakenly set too long, you can now modify it after the Workflow has started. Use this command to update the delay to a new value (10 hours in this example):temporal workflow update --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"10h"'
Or use this command to remove the delay entirely:
temporal workflow update --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"0"'
Please note: The new delay starts from when it is set, not from when the original timer was created. For example, if the Workflow has already slept for 2 hours and the timer is updated to
10h
, it will sleep for another 10 hours, not 8.
Scheduled Actions
- Scheduler workflow version has been updated.
FutureActionTimes
now accounts for a schedule’s update time andRemainingActions
.ScheduleActionResult
now includes aWorkflowExecutionStatus
field, providing an eventually-consistent view of a workflow’s status within List results.
- Bugfix: Queries on schedules might have been delayed after dynamic config changes were applied to per-namespace workers. An anti-entropy mechanism has been added to the per-namespace worker component.
Fix out-of-order Visibility tasks with SQL database
All SQL stores used for Visibility had the rare possibility to perform updates to a workflow's visibility state out-of-order. This could result in Workflows occasionally appearing to have state that is out of date.
Implementation
This has been fixed by adding a new column _version
to all SQL store implementations. Queries to update Visibility data now ensure the _version
advances before performing any writes.
Updates to Visibility data are prepared by checking actual Workflow state. Therefore when a write is rejected for being out-of-order, we know the VisibilityStore already contains a equal or more up-to-date state, so we drop out-of-order updates silently.
Important Notes
- Make sure to upgrade your Schemas before advancing your temporal version. See the Schema Changes section.
- This will likely reduce the performance of SQL VisibilityStores.
- We suggest ElasticSearch as the performant solution for VisibilityStore.
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release (use the...
v1.27.0
Caution
Please DO NOT use it if you are using SQL-based (PostgreSQL, MySQL, or sqlite) persistence. Update directly to v1.27.1.
This release made changes to the SQL schemas, and there's a bug when upgrading from an older version.
Schema Changes
Before upgrading your Temporal Cluster to v1.27.0, you must upgrade your core schema if you are using MySQL or PostgreSQL, and your visibility schema to the following:
- Core:
- MySQL schema v1.15
- PostgreSQL schema v1.15
- Visibility:
- Elasticsearch schema v9
- MySQL schema 1.9
- PostgreSQL schema v1.9
Please see our upgrade documentation for the necessary steps to upgrade your schemas.
NOTE: The upgrade to MySQL and PostgreSQL Visibility schemas may come with temporary performance degradation because of creation of a new column _version
which has default values. Consider performing the schema upgrades when load is low. There are protective mechanisms in place to account for timeouts from any VisibilityStore.
Deprecation of Visibility Scan APIs
Visibility Scan APIs have been deprecated in favor of List Workflow APIs. Visibility Scan APIs will be removed in a future version. Migration to List Workflow APIs will be required in future versions.
Nexus GA
Nexus is now GA with a stable server API.
Read more here on how to disable Nexus or how to operate it here.
Notable features and bug fixes since v1.26.2
:
- The server now allows a maximum of 30 pending Nexus operations per workflow by default, as opposed to the previous limit being 30 total.
- Add support for attaching callbacks, request IDs, and links to a workflow via
StartWorkflowExecutionRequest.OnConflictOptions
withWorkflowIdConflictPolicy
ofUSE_EXISTING
. This allows multiple operations to be backed by the same workflow. - Misc small features and tests.
Safe Deploys
The following APIs are added for Worker Versioning. All APIs are experimental and not yet recommended for production usage. You need to set the dynamic configs system.enableDeployments
and system.enableDeploymentVersions
in order to use them.
- ListWorkerDeployments
- DescribeWorkerDeployment
- DescribeWorkerDeploymentVersion
- SetWorkerDeploymentCurrentVersion
- SetWorkerDeploymentRampingVersion
- UpdateWorkerVersionMetadata
- DeleteWorkerDeployment
- DeleteWorkerDeploymentVersion
The following APIs are now deprecated and replaced by above:
DescribeDeploymentListDeploymentsGetDeploymentReachabilityGetCurrentDeploymentSetCurrentDeployment
Activity Commands (pre-release)
Changes to the Activity Commands — a set of APIs designed to resolve issues related to activity execution. The following APIs where updated:
UpdateActivityOptionsById
was renamed toUpdateActivityOptions
. This API can be used by the client to update the options of an activity while activity is running.PauseActivityById
was renamed toPauseActivity
. With this API, If the Activity is not currently running (e.g. because it previously failed and is waiting for the next retry), it will not be run again until it is unpaused.activity_type
parameter was added. If this parameter is set - all running activities of this type will be paused.
UnpauseActivityById
was renamed toUnpauseActivity
. With this API clients can re-schedule a previously-paused Activity for execution.no_wait
parameter was removedactivity_type
parameter was added. If this parameter is set - all paused activities of this type will be unpaused.match-all
parameter was added. If this parameter is set - all paused activities will be unpaused.jitter
parameter was added. If set, the activity will start at a random time within the specified jitter duration.
ResetActivityById
was renamed toResetActivity
. With this API clients can unpauses the execution of a previously paused activity, specified by its ID.no_wait
parameter was removedactivity_type
parameter was added. If this parameter is provided - all paused activities of this type will be unpaused.keep_paused
parameter was added. If this parameter is provided - all paused activities will stay paused. Otherwise they will be unpaused.jitter
parameter was added. If set, the activity will start at a random time within the specified jitter duration.
- New batch operation was introduced -
BatchOperationUnpauseActivities
.
Workflow Reset with children
In this release we have added the functionality to reset a workflow with a pending child.
Prior to this release reseting to a point between child workflow initiated and child workflow completed was not supported (the reset operation would fail). In the current release the reset operation will allow this case and the behavior of the parent after reset is to reconnect to the running child. The new run of the parent will receive the child’s completion event and result (if any) from the child.
The feature is gated behind a per namespace boolean dynamic configuration AllowResetWithPendingChildren
which is enabled by default for all namespaces.
Note: If you are using Go-SDK and are relying on the SDK to generate child workflow IDs then you need to update it to the latest version to be able to use this feature. Other SDKs don’t need any upgrade to use this feature.
Delete namespace improvement
-
Delete workflow executions RPS is now dynamic. Previously,
frontend.deleteNamespaceDeleteActivityRPS
was read only once when namespace deletion started, and subsequent changes to this dynamic config didn't affect the ongoing deletion. This was inconvenient for large namespaces since the default RPS is only100
. Now the RPS can be adjusted on the fly.Please note: Since deletion of Workflow Execution is an asynchronous process, this RPS controls the rate at which delete execution tasks are created. Decreasing this value (for example, from
1000
to10
) won't immediately slow down the process, as existing tasks in the transfer queue must be processed first. -
DeleteExecutionsWorkflow
now supports astats
query to track its progress. Since this Workflow can run for hours after a namespace is marked as deleted, it was previously difficult to monitor how many Workflow Executions remained. The new query handler provides current statistics about total and remaining executions. -
Metrics and logging have been enhanced for better actionability. Key improvements include:
- Monitor
ReclaimResources
workflow failures using the metricsreclaim_resources_namespace_delete_failure
andreclaim_resources_delete_executions_failure
. - Track
DeleteExecutionsWorkflow
progress using the metricsdelete_executions_success
anddelete_executions_failure
.
- Monitor
-
Business critical namespaces can be protected from deletion. Use dynamic config to list namespaces which are not deletable:
worker.protectedNamespaces: - value: - critical_namespace - just_very_important_namespace
-
Sleep duration in
ReclaimResourcesWorkflow
now supports dynamic changes. If a namespace delete delay was mistakenly set too long, you can now modify it after the Workflow has started. Use this command to update the delay to a new value (10 hours in this example):temporal workflow update --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"10h"'
Or use this command to remove the delay entirely:
temporal workflow update --namespace temporal-system --name update_namespace_delete_delay --workflow-id temporal-sys-reclaim-namespace-resources-workflow/default-deleted-93f5e --input '"0"'
Please note: The new delay starts from when it is set, not from when the original timer was created. For example, if the Workflow has already slept for 2 hours and the timer is updated to
10h
, it will sleep for another 10 hours, not 8.
Scheduled Actions
- Scheduler workflow version has been updated.
FutureActionTimes
now accounts for a schedule’s update time andRemainingActions
.ScheduleActionResult
now includes aWorkflowExecutionStatus
field, providing an eventually-consistent view of a workflow’s status within List results.
- Bugfix: Queries on schedules might have been delayed after dynamic config changes were applied to per-namespace workers. An anti-entropy mechanism has been added to the per-namespace worker component.
Fix out-of-order Visibility tasks with SQL database
All SQL stores used for Visibility had the rare possibility to perform updates to a workflow's visibility state out-of-order. This could result in Workflows occasionally appearing to have state that is out of date.
Implementation
This has been fixed by adding a new column _version
to all SQL store implementations. Queries to update Visibility data now ensure the _version
advances before performing any writes.
Updates to Visibility data are prepared by checking actual Workflow state. Therefore when a write is rejected for being out-of-order, we know the VisibilityStore already contains a equal or more up-to-date state, so we drop out-of-order updates silently.
Important Notes
- Make sure to upgrade your Schemas before advancing your temporal version. See the Schema Changes section.
- This will likely reduce the performance of SQL VisibilityStores.
- We suggest ElasticSearch as the performant solution for VisibilityStore.
...
v1.26.2
Visibility schema changes
TemporalPauseInfo
column was added to visibility. TemporalPauseInfo
contains search attribute related to paused entities in temporal workflows.
Before upgrading your Temporal Cluster to v1.26.2, you must upgrade your visibility schemas to the following:
- Visibility:
- Elasticsearch schema v8
- MySQL schema 1.7
- PostgreSQL schema v1.7
Batch metrics (wide events)
PR: #6655
Description: Extended metrics.Handler
interface with a new StartBatch
method. StartBatch returns a BatchHandler that can be used to send a sequence of metrics as a single event when Close() is called on the batch. All provided metric handlers have been updated with the new interface and simply send metrics individually.
💥 BREAKING CHANGE 💥 If you provide a custom metrics handler with temporal.WithCustomMetricsHandler(metricsHandler)
you will need to implement StartBatch()
on that handler. See the tally metrics handler for an example of this.
Versioning / safe deploy (pre-release)
The following EXPERIMENTAL Versioning APIs are added in this release:
- Deployment APIs, for managing worker deployments:
DescribeDeployment
ListDeployments
GetCurrentDeployment
SetCurrentDeployment
GetDeploymentReachability
UpdateWorkflowExecutionOptions
API (and its batch mode) for setting versioning override for executions.
Documentation is not available at this point. Do not use above APIs in production.
To enable these APIs the following configs should be enabled: system.enableDeployments
and frontend.workerVersioningWorkflowAPIs
.
Workflow Update GA
Description:
Workflow Update enables a gRPC client of a Workflow Execution to issue requests to that Workflow Execution and receive a response. These requests are delivered to and processed by a client-specified Workflow Execution. Updates are differentiated from Queries in that the processing of an Update is allowed to modify the state of a Workflow Execution. Updates are different from Signals in that an Update returns a response.
Any gRPC client can invoke Updates via the WorkflowService.UpdateWorkflowExecution
. Additionally, past Update requests can be observed via the WorkflowService.PollWorkflowExecutionUpdate
API. The wait stage option determines whether they respond once the Update was accepted or completed.
Note that an Update only becomes durable when it was accepted, until then, it will not appear in the Workflow history. SDKs will automatically retry to ensure Update requests complete.
The execution and retention of Updates is configured via two optional dynamic configuration values:
history.maxTotalUpdates
controls the total number of Updates that a single Workflow Execution can support. The default is 2000.history.maxInFlightUpdates
controls the number of Updates that can be “in-flight” (that is, concurrently executing, not having completed) for a given Workflow Execution. The default is 10.
Since the 1.25 release, several minor bugs have been fixed.
Workflow Update-With-Start (public preview)
Update-With-Start sends a Workflow Update that checks whether an already-running Workflow with that ID exists. If it does, the Update is processed. If not, it starts a new Workflow Execution with the supplied ID. When starting a new Workflow, it immediately processes the Update.
Update-With-Start is great for latency-sensitive use cases.
Activity API (pre-release)
Description:
We introduce the Activity API — a set of APIs designed to resolve issues related to activity execution. The following APIs where introduced:
- UpdateActivityOptionsById. This API can be used by the client to update the options of an activity while activity is running.
- The following options are supported:
- task-queue
- schedule-to-close-timeout
- schedule-to-start-timeout
- start-to-close-timeout
- heartbeat-timeout
- retry-initial-interval
- retry-maximum-interval
- retry-backoff-coefficient
- retry-maximum-attempts
- Partial updates are supported.
- Timeout updates are effective immediately.
- The following options are supported:
- PauseActivityById. With this API, If the Activity is not currently running (e.g. because it previously failed and is waiting for the next retry), it will not be run again until it is unpaused.
- However, if the Activity is currently running, it will run to completion. If the Activity is on its last retry attempt and fails, the failure will be returned to the caller, just as if the Activity had not been paused. Otherwise, if it fails, it will enter a paused state.
- UnpauseActivityById . With this API clients can re-schedule a previously-paused Activity for execution.
- If the Activity is not running and has exceeded its retry timeout, it will be scheduled immediately. Otherwise, it will be scheduled when its retry timeout expires. The Activity can be retried immediately using
--no-wait
.
- If the Activity is not running and has exceeded its retry timeout, it will be scheduled immediately. Otherwise, it will be scheduled when its retry timeout expires. The Activity can be retried immediately using
- ResetActivityById. With this API clients can unpauses the execution of a previously paused activity, specified by its ID.
- Resetting an activity means:
- number of attempts will be reset to 0.
- activity timeouts will be reset.
- Resetting an activity means:
Documentation is not available at this point. Do not use above APIs in production.
Nexus
Nexus is still in public preview for this release, but has now been enabled by default.
Read more here on how to disable Nexus or how to operate it here.
Notable features and bug fixes since v1.25.2
:
- Allow completing a Nexus operation after workflow reset (#6434).
- Fix a few issues around replication and cancelation.
- Record NexusOperationStarted event and link when async operation completion races with the sync response path.
- Support for full Temporal Failure rehydration and encryption. Nexus handlers and workflows backing operations will now expose the full error details by default. Caller workflows will also rehydrate the original error to allow for better E2E debugging experience.
- Allow skipping application of Nexus events when resetting a workflow.
Notable bug fixes
Primary engineer: @prathyushpv
Table not found bug in sqlite.
Durations with mismatched seconds and nanoseconds signs will now fail validation and return an InvalidArgument error.
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release (use the tag 1.26.2
)
Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools
Full Changelog: v1.25.2...v1.26.2
v1.25.2
Release Highlights
This patch release fixes few minor Nexus and Update-with-Start related bugs.
Full Changelog: v1.25.1...v1.25.2
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release (use the tag 1.25.2
)
Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools
v1.24.3
Release Highlights
This release fixes one bug:
- #6566: track_total_hits is disabled for Elasticsearch Scroll API
It also fixes a small bug in the Makefile which affects users building from source.
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release (use the tag 1.24.3
)
Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools
Full Changelog: v1.24.2...v1.24.3
v1.25.1
Release Highlights
This patch release fixes a couple bugs:
- #6566: track_total_hits is disabled for Elasticsearch Scroll API
- #6514: failure to reconnect to database
It also adds support for Nexus links.
Full Changelog: v1.25.0...v1.25.1
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release (use the tag 1.25.1
)
Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools
v1.25.0
Schema changes
Before upgrading your Temporal Cluster to v1.25.0, you must upgrade your core and visibility schemas to the following:
- Core:
- MySQL schema v1.14
- PostgreSQL schema v1.14
- Cassandra schema v1.11
- Visibility:
- Elasticsearch schema v7
- MySQL schema 1.6
- PostgreSQL schema v1.6
Please see our upgrade documentation for the necessary steps to upgrade your schemas.
Release Highlights
1. Nexus
Nexus RPC is an open-source service framework for arbitrary-length operations whose lifetime may extend beyond a traditional RPC. It is an underpinning connecting durable executions within and across namespaces, clusters and regions – with an API contract designed with multi-team collaboration in mind. A service can be exposed as a set of sync or async Nexus operations – the latter provides an operation identifier and a uniform interface to get the status of an operation or its result, receive a completion callback, or cancel the operation.
Temporal uses the Nexus RPC protocol to allow calling across namespace and cluster boundaries. The Go SDK Nexus proposal explains the user experience and shows sequence diagrams from an external perspective.
Read more here on how to enable Nexus and how to operate it here.
2. Workflow Update (public preview)
Workflow Update enables a gRPC client of a Workflow Execution to issue requests to that Workflow Execution and receive a response. These requests are delivered to and processed by a client-specified Workflow Execution. Updates are differentiated from Queries in that the processing of an Update is allowed to modify the state of a Workflow Execution. Updates are different from Signals in that an Update returns a response.
Any gRPC client can invoke Updates via the WorkflowService.UpdateWorkflowExecution
. Additionally, past Update requests can be observed via the WorkflowService.PollWorkflowExecutionUpdate
API. The wait stage option determines whether they respond once the Update was accepted or completed.
Note that an Update only becomes durable when it was accepted, until then, it will not appear in the Workflow history. SDKs will automatically retry to ensure Update requests complete.
The execution and retention of Updates is configured via two optional dynamic configuration values:
history.maxTotalUpdates
controls the total number of Updates that a single Workflow Execution can support. The default is 2000.history.maxInFlightUpdates
controls the number of Updates that can be “in-flight” (that is, concurrently executing, not having completed) for a given Workflow Execution. The default is 10.
Since the 1.21 release, the feature was heavily tested and several bug fixes as well as performance optimizations were made.
You can find more information at this link.
3. Host level MutableState cache
The MutableState
cache has been updated to operate as a host-level cache by default. Previously, this cache was managed at the shard level, with each shard cache holding 512 MutableState
entries. Now, the host-level cache, enabled by default (history.enableHostHistoryCache = true
), will be shared across all shards on a given host.
The size of the host-level cache is controlled by the history.hostLevelCacheMaxSize
configuration, which is set to 128,000 entries by default. This change may impact the memory usage of the history service, but it can be adjusted by modifying the history.hostLevelCacheMaxSize
value.
4. Visibility Enhancement
Enhanced the Nexus CLI to support query filtering for the schedule list
command. The --query
or -q
(string) option allows filtering of results using a specified list filter.
5. Task Queue Statistics
Provide stats for Task Queue backlogs to be used for worker scaling decisions.
User DescribeTaskQueue
API in enhanced mode (with report_stats=true
) to get the following info about the Task Queue:
- Approximate backlog count
- Approximate backlog age
- Approximate rate of adding tasks to the task queue
- Approximate rate of dispatching tasks from the task queue
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release
Server (use the tag 1.25.0
)
Server With Auto Setup ([what is Auto-Setup?] (https://docs.temporal.io/blog/auto-setup)) (use the tag 1.25.0
)
Admin-Tools (use the tag 1.25.0-tctl-1.18.1-cli-1.0.0
)
Full Changelog: v1.24.0...v1.25.0
v1.24.2
What's Changed
- Acquire workflow lock during backfill history by @yux0 in #6102
- Next retry delay should not be bounded by retry policy MaximumInterval by @gow in #6063
- Handle NextRetryDelay option in workflow failures by @gow in #5946
- Remove build id from sticky recordWorkflowTaskStarted by @ShahabT in #6096
- Scheduler Bugfix: getFutureActionTimes should ignore actions prior to update time, respect RemainingActions counter by @lina-temporal in #6122
Full Changelog: v1.24.1...v1.24.2
v1.24.1
Schema changes
If you are using SQL based visibility, before upgrading your Temporal Cluster to v1.24.0, you must upgrade your visibility schemas to the following:
- MySQL schema 1.6
- PostgreSQL schema v1.6
Please see our upgrade documentation for the necessary steps to upgrade your schemas.
Release Highlights
This release contains schema fix for SQL based visibility introduced in 1.24.0
. For full set of new features please check v1.24.0 release notes.
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release
Server (use the tag 1.24.1
)
Server With Auto Setup (what is Auto-Setup?) (use the tag 1.24.1
)
Admin-Tools (use the tag 1.24.1-tctl-1.18.1-cli-0.12.0
)
Full Changelog: v1.24.0...v1.24.1
v1.24.0
Caution
This release introduces a bug in SQL visibility. Please DO NOT use it if you are using SQL-based (PostgreSQL, MySQL, or sqlite) visibility. Elasticsearch based visibility is not affected. Update directly to v1.24.1.
Schema changes
Before upgrading your Temporal Cluster to v1.24.0, you must upgrade your core and visibility schemas to the following:
- Core:
- MySQL schema v1.12
- PostgreSQL schema v1.12
- Cassandra schema v1.10
- Visibility:
- Elasticsearch schema v7
- MySQL schema 1.5
- PostgreSQL schema v1.5
Please see our upgrade documentation for the necessary steps to upgrade your schemas.
Breaking changes
Standard Visibility
As planned, standard visibility is no longer supported in this version. Please, upgrade to advanced visibility as well as the config keys to setup visibility before upgrading to this version. Refer to v1.20.0 release notes for upgrade instructions, and also check the v1.21.0 release notes for config key changes.
Note that you also need to update the plugin name for the main store, ie., if you are using mysql
plugin name for the main store, then you need to change to mysql8
. Similarly, if it's postgres
, then change it to postgres12
.
Deprecation Announcements
Worker Versioning APIs [Experimental]
The following changes were made to Worker Versioning APIs:
- Deprecated
UpdateWorkerBuildIdCompatibility
in favor of the newUpdateWorkerVersioningRules
API. - Deprecated
GetWorkerBuildIdCompatibility
in favor of the newGetWorkerVersioningRules
API. - Deprecated
GetWorkerTaskReachability
in favor ofDescribeTaskQueue
enhanced mode (api_mode=ENHANCED
)
Together with the old APIs, the Version Set concept is also deprecated and replaced with “versioning rules” which are more powerful and flexible. More details can be found in https://github.com/temporalio/api/blob/master/temporal/api/taskqueue/v1/message.proto#L153.
For using these experimental APIs you need to enable the following configs:
frontend.workerVersioningRuleAPIs
frontend.workerVersioningWorkflowAPIs
Release Highlights
Namespace Burst Ratio
Removing frontend.namespaceBurst
and adding frontend.namespaceBurstRatio
config. Similarly replacing frontend.namespaceBurst.visibility
and frontend.namespaceBurst.namespaceReplicationInducingAPIs
with frontend.namespaceBurstRatio.visibility
and frontend.namespaceBurstRatio.namespaceReplicationInducingAPIs
.
The old values are used to specify the burst rate as number of requests per second. New values will specify burst as a ratio of their respective RPS limit. This ratio will be applied to calculated RPS limit from global and per-instance rate limits.
Visibility: Parent workflow execution
We added two new system search attributes: RootWorkflowId
and RootRunId
. If you have previously created custom search attributes with one of these names, attempts to set them will start to fail. We suggest updating your workflows to not set those search attributes, delete those search attributes and then upgrade Temporal to this version. Alternatively, you can also set the dynamic config system.supressErrorSetSystemSearchAttribute: true
. When this dynamic config is set to true
, your workflow will not fail when trying to set a value on a system search attribute, and it will ignore your input for those system search attributes.
OpenAPI HTTP API Documentation
OpenAPI v2 docs are served at /api/v1/swagger.json
while v3 is at /api/v1/openapi.yaml
when our HTTP API is enabled.
Shard Info Update Optimizations
Operators can now configure how often we update shard info (tracking how many tasks have been acked, etc). This improves recovery speed by persisting shard data more frequently.
This can be configured through the following dynamic config values:
history.shardUpdateMinTasksCompleted
- the minimum number of tasks which must be completed (across all queues) before the shard info can be updatedhistory.shardUpdateMinInterval
- the minimum amount of time between shard info updates unlessshardUpdateMinTasksCompleted
tasks have been acked
Note that once history.shardUpdateMinInterval amount
of time has passed we'll update the shard info regardless of the number of tasks completed
Interpolate MySQL query parameters by default (#5428)
We now interpolate parameters into queries client-side for MySQL main databases but not visibility.
When interpolateParams
is false (the default) the driver will prepare parameterized statements before executing them, meaning we need two round-trips to the database for each query. By setting interpolateParams
to true the DB driver will handle interpolation and send the query just once to the database, halving the number of round trips necessary. This should improve the performance of all Temporal deploys using MySQL.
OpenTelemetry env variables
Support for enabling OpenTelemetry for tracing gRPC requests via environment variables. See develop/docs/tracing.md
for details.
Matching task queue handover
Various improvements were made to task queue handover when adding/removing/restarting matching nodes. This should improve tail latency for task dispatch during those situations. To enable the improvements, operators should set the dynamic config matching.alignMembershipChange
to a value like 10s
after fully deploying v1.24 to the entire cluster. This may become the default in future versions.
UTF-8 validation in protobuf messages
When we migrated Temporal from the deprecated gogoproto fork of Google’s protobuf library to the official version in v1.23, we disabled protobuf’s default utf-8 validation to ensure a smooth deployment, since gogoproto did not validate fields for utf-8, and turning on validation immediately would have broken applications that accidentally used invalid utf-8.
This was a temporary measure and we will eventually re-enable validation. As the first step, we’ve added tools to detect and warn about invalid utf-8 without breaking applications. There are two sets of dynamic config settings to use.
The sample
settings are set to a floating point value between 0.0 and 1.0 (default 0.0), and control what proportion of either RPC requests, responses, or data read from persistence, is validated for utf-8 in strings. If invalid utf-8 is found, warnings will be sent to logs, and the counter metric utf8_validation_errors
will be incremented.
The fail
settings (boolean, default false) control whether a validation error will be turned into a RPC failure or data corruption error.
system.validateUTF8.sample.rpcRequest
system.validateUTF8.sample.rpcResponse
system.validateUTF8.sample.persistence
system.validateUTF8.fail.rpcRequest
system.validateUTF8.fail.rpcResponse
system.validateUTF8.fail.persistence
If you think your application may be using invalid utf-8, we suggesting turning on the sample settings without the fail settings and running for a while. In a future version, validation and errors will be turned on by default (effectively sample set to 1.0 and fail set to true).
admin-tools
docker image versioning
We separated admin-tools
docker image release process. Version tag now includes versions of tctl
(deprecated but still supported CLI) and temporal
(modern CLI) binaries. This image is released every time whenever new version of any of these component is released. Current latest tag is 1.24.0-tctl-1.18.1-cli-0.12.0.
Helpful links to get you started with Temporal
Temporal Docs
Server
Docker Compose
Helm Chart
Docker images for this release
Server (use the tag 1.24.0
)
Server With Auto Setup (what is Auto-Setup?) (use the tag 1.24.0
)
Admin-Tools (use the tag 1.24.0-tctl-1.18.1-cli-0.12.0
)
Full Changelog: v1.23.1...v1.24.0