v1.21.1

IMPORTANT

• Do not upgrade to this version if you using advanced visibility with MySQL as storage. The release of v1.21.0 included an script to upgrade the schema, but it contains a bug. Please upgrade to v1.21.2 which fixes the bug.
• If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

See the 1.21.0 release notes for schema upgrade information for upgrading to the 1.21 series.

Release Highlights

This release fixes these bugs:

• Skip reading close event for retention task if not using Cassandra visibility store.
• Use lower priority for standby tasks.
• Added sanity check to protect against mutable state corruption.
• Fix replication ack aggregation logic.

All Changes

2023-06-30 - 873df34 - Improve build id scavenger (#4568)
2023-06-30 - ed7f94c - Fix migration in mem ack aggregation (#4569)
2023-06-29 - 0fcff51 - Measure backlog after completing a current task (#4562)
2023-06-29 - 2755679 - Add corruption protection to ms (#4560)
2023-06-29 - 8b3426b - Handle get namespace by id error (#4559)
2023-06-29 - 72fb120 - Use lowest persistence priority for standby tasks (#4551)
2023-06-29 - ca41bb0 - Consider non default sets reachable by new workflows for a while after they stop being queue default (#4545)
2023-06-29 - 0d7350d - Get execution start/close time when using Cassandra visibility store (#4549)
2023-06-29 - 0c441a7 - Add more metrics for stream replication (#4539)

Details about the v1.21.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use the tag 1.21.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.21.0

IMPORTANT

• Do not upgrade to this version if you using advanced visibility with MySQL as storage. The release of v1.21.0 included an script to upgrade the schema, but it contains a bug. Please upgrade to v1.21.2 which fixes the bug.
• If downgrading from this release to the 1.20 series, downgrade to 1.20.4 instead of 1.20.3. This addresses an issue where the new BuildIds search attribute was not understood and would cause errors in visibility processing.

Schema Upgrades

This release requires schema upgrades for both persistence and visibility. See the schema upgrade documentation for the required commands for performing the upgrade.

Release Highlights

Worker Versioning (preview)

Links to docs:

• https://docs.temporal.io/workers#worker-versioning
• https://docs.temporal.io/dev-guide/java/versioning#worker-versioning
• https://docs.temporal.io/dev-guide/golang/versioning#worker-versioning

Description: Worker Versioning simplifies the process of deploying changes to Workflow Definitions.

⚠️ This feature requires Advanced Visibility

In the SDK, a user can identify a Worker with a Build ID, which should be associated at build time, and configure that Worker to use Versioning.

An operator can use the UpdateWorkerBuildIdCompatibility gRPC API (and the high-level APIs in the SDKs) to define which builds are compatible with each other and control task routing for existing and new Workflows based on the compatibility constraints.

Additionally, operators and automated scripts can use the GetWorkerTaskReachability API to query for build IDs that are no longer reachable and whose associated workers can be retired.

⚠️ To enable worker versioning, turn on the frontend.workerVersioningDataAPIs, frontend.workerVersioningWorkflowAPIs, and worker.buildIdScavengerEnabled dynamic config flags (disabled by default in this release).

This feature introduces a couple of schema changes:

1. A new BuildIds Search Attribute that is used to track reachability. This new Search Attribute requires a Visibility schema update. See the schema upgrade documentation for the required commands for performing the upgrade.
2. A new task_queue_user_data table, which stores the Versioning data for the various Task Queues and is replicated using the Namespace replication queue. See the schema upgrade documentation for the required commands for performing the upgrade.

The Matching Service will now load Task Queue user data from persistence for the Task Queue root partition, and non-root partitions will try to obtain and maintain a fresh view of the user data by long-polling other Task Queue partitions for the same Task Queue.

If user data loading is suspected to be causing processing overhead, it can be completely disabled by using the matching.loadUserData dynamic config flag.

Note that disabling the loading of user data will essentially disable the Versioning feature and will cause tasks for versioned Workflows to be dropped by the Matching Service, causing versioned Workflows to be stuck. This approach was chosen over dispatching versioned Workflows to arbitrary Workers and breaking the user-defined compatibility semantics. Manual intervention will be required to regenerate Tasks for stuck versioned Workflows.

As part of this work, the UpdateWorkerBuildIdCompatiblity, UpdateNamespace, and RegisterNamespace APIs now have stricter rate limiting because they can produce replication messages for Global Namespaces. The Namespace replication queue is a critical resource that is not partitioned and serves as a mechanism for delivering failover messages between Clusters.

A few more dynamic configuration options were added for this feature:

• limit.workerBuildIdSize is the byte length limit for a Worker Build ID as used in the RPC methods for updating the version sets for a Task Queue. Do not set this option to a value greater than 255 for Clusters that use SQL-based persistence due to predefined VARCHAR column width.
• limit.versionCompatibleSetLimitPerQueue is the maximum number of compatible sets allowed in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.versionBuildIdLimitPerQueue is the maximum number of Build IDs allowed to be defined in the Versioning data for a Task Queue. Update requests that would cause the Versioning data to exceed this number will fail with a FailedPrecondition error.
• limit.taskQueuesPerBuildId limits the number of Task Queue names that can be mapped to a single Build ID.
• limit.reachabilityTaskQueueScan limits the number of Task Queues to scan when responding to a GetWorkerTaskReachability query.
• limit.reachabilityQueryBuildIds limits the number of Build IDs that can be requested in a single call to the GetWorkerTaskReachability API.
• worker.removableBuildIdDurationSinceDefault is the minimum duration since a Build ID was last set as default in its containing set for it to be considered for removal. The value is used by the Build ID scavenger.
• frontend.rps.namespaceReplicationInducingAPIs limits the requests-per-second (RPS) rate for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceRPS.namespaceReplicationInducingAPIs is a per-host and per-Namespace RPS limit for Namespace replication-including APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.namespaceBurst.namespaceReplicationInducingAPIs is a per-host and per-Namespace burst limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). This config is EXPERIMENTAL and might be changed or removed in a later release.
• frontend.globalNamespaceRPS.namespaceReplicationInducingAPIs is a Cluster global, per-Namespace RPS limit for Namespace replication-inducing APIs (such as RegisterNamespace, UpdateNamespace, and UpdateWorkerBuildIdCompatibility). The limit is evenly distributed among available Frontend Service instances. If this option is set, it overwrites the per-instance limit configured with frontend.namespaceRPS.namespaceReplicationInducingAPIs. This config is EXPERIMENTAL and might be changed or removed in a later release.

Synchronous Update (preview)

Synchronous Workflow Update enables a gRPC client of a Workflow Execution to issue requests to that Workflow Execution and receive a response. These requests are delivered to and processed by a client-specified Workflow Execution. Updates are differentiated from Queries in that the processing of an Update is allowed to modify the state of a Workflow Execution. Updates are different from Signals in that an Update returns a response.

Updates can be handled by Workflows written using the latest Go and Java SDKs, with other languages to follow.

Any gRPC client can invoke Updates via the WorkflowService.UpdateWorkflowExecution API, which returns the Update outcome inline. Additionally, past Update outcomes can be observed via the WorkflowService.PollWorkflowExecutionUpdate API.

Update functionality is disabled by default but can be enabled per Namespace by setting the frontend.enableUpdateWorkflowExecution flag to true for that Namespace in dynamic config.

In this release, Update outcomes are stored for the same period as the Workflow history itself and can be observed for as long as that history exists. The execution and retention of Updates is configured via two optional dynamic configuration values:

• history.maxTotalUpdates controls the total number of Updates that a single Workflow Execution can support. The default is 2000.
• history.maxInFlightUpdates controls the number of Updates that can be “in-flight” (that is, concurrently executing, not having completed) for a given Workflow Execution. The default is 10.

Warning: when resolving history event conflict in global namespace, history event for update may be lost. This will be addressed in future release.

Batch Reset via BatchOperation API

Description: The server now supports resetting a batch of Workflow Executions. StartBatchOperationRequest now supports the Operation type of ResetOperation.

This operation depends on two additional parameters:

• ResetReapplyType: Determines whether to reapply Signals after resetting the Workflow.
• ResetType: Determines which Event ID to choose when resetting the batch of Workflows.
  • RESET_TYPE_FIRST_WORKFLOW_TASK: Resets to the Event of the first Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.
  • RESET_TYPE_LAST_WORKFLOW_TASK : Resets to the Event of the last Workflow Task completed. If the Task doesn’t exist, resets to the Event after the Task was scheduled.

Dynamic rate limiter

The dynamic rate limiter is an advanced system-protection mechanism. By default, it’s disabled. When enabled, the per-host persistence rate limiter automatically adjust based on the average number of errors returned from persistence and the average latency of persistence requests.

The rate limit is adjusted using a multiplier between 0.1 and 1.0 (default 1.0) on the host-level persistence rate limiter. If either the ratio of persistence calls that result in an error or the average latency of persistence req...

v1.20.3

Release Highlights

This release fixes few schedule and search attributes issues as well as minor bugs in various parts of the system.

All Changes

2023-05-12 - 3ce6308 - Fix cron for schedules that do not match any time (#4206)
2023-05-12 - d8a6ca7 - Run no-op claim mapper even without auth info (#4197)
2023-05-12 - 9448a6f - Fix metric tag for delete execution task (#4275)
2023-05-12 - 5225f13 - Fix task attempt reset (#4276)
2023-05-12 - f163b88 - Make shard ownership more resilient when shard is busy (#4283)
2023-05-12 - d6f7c91 - Event buffer size limit (#4296)
2023-05-12 - f64bb0e - Use search attribute type map in visibility archival (#4304)
2023-05-12 - 4ef165a - Skip over entire time range if paused and batch and cache time queries (#4215)
2023-05-12 - caac9c2 - Fix adding multiple search attributes of same type at once (#4273)
2023-05-12 - c54890d - Change schedules default catchup window to 1 year (#4332)
2023-05-15 - a97cdef - Truncate activity failure in mutable state (#4338)
2023-05-15 - 6eba60d - Fix caller info for migration activities (#4340)
2023-05-15 - fcebf69 - Tag metric tracking custom order by with namespace (#4344)
2023-05-15 - a4c0b81 - Bump server version for 1.20.3 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.20.2

Release Highlights

This release fixes few schedule and search attributes issues as well as minor bugs in various parts of the system.

All Changes

2023-04-17 - 9729d03 - Reuse timer in schedule workflow (#4129)
2023-04-17 - 8f178a3 - Use correct time for initial schedule memo (#4128)
2023-04-17 - 16d53cb - Reconnect on a few more gocql errors (#4132)
2023-04-17 - 50b7daf - Fix error handling add search attributes (#4148)
2023-04-17 - d017c66 - Validation API should use low priority lock (#4140)
2023-04-17 - eec0dbe - Fix schedule catchup window metric calculation while paused (#4152)
2023-04-17 - f638c97 - Update Go SDK to v1.21.2
2023-04-17 - 99c10cc - Bump server version for 1.20.2 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.20.1

Release Highlights

This release fixes few SQL-based advanced visibility bugs as well as minor bugs in various parts of the system.

All Changes

2023-03-24 - f38e041 - Fix passing pointer of slice element in SQL query converter (#3970)
2023-03-24 - 6d28fe2 - Refactor SQL query converter (#3971)
2023-03-24 - f714a91 - Add SQL query converter tests (#3972)
2023-03-24 - 5c06a2b - Flush buffer event with single cluster namespace (#4031)
2023-03-24 - 1855bb4 - Refactor gocql.NewSession to allow passing in cluster config factory method (#4033)
2023-03-24 - d8aaa5f - Extract cluster config logic into ConfigureCassandraCluster() (#4034)
2023-03-24 - f174671 - Convert query converter error to invalid argument (#4036)
2023-03-24 - 07cf951 - Fix UnhandledCommand error for workflow task completion in CancelTimer (#4050)
2023-03-24 - 305b770 - Change disable order by dynamic config to per namespace (#4063)
2023-03-24 - f8c06b5 - Set default values in returned calendar specs (#3969)
2023-03-24 - 8c1a0fb - Add counter to track custom order by usage in Elasticsearch (#4064)
2023-03-24 - 89f2095 - Fixing saving custom search attribute after upgrading SQL DB (#4062)
2023-03-24 - b0ef992 - Fix SQL visibility page token deserializer (#4074)
2023-03-24 - 5a83f71 - Add deadlines to schedule local activities (#4078)
2023-03-24 - cb7d42c - Use preemptable caller type for low priority history tasks (#3993)
2023-03-24 - 633a056 - Lock current execution on start workflow (#4066)
2023-03-24 - 145c99b - Handle max signal count limit for signal transfer task (#4051)
2023-03-24 - a8b591d - Track and emit metrics for entities in a single workflow (#4065)
2023-03-24 - df2d8eb - Fix namespace handover replication queue notification (#4082)
2023-03-24 - de84092 - Use more reliable workflow mutation check (#4076)
2023-03-24 - c0484f9 - Fix test failure from merge conflict (#4087)
2023-03-24 - dcaa3a3 - Account for local activity run time in schedule sleep (#4079)
2023-03-24 - 57347cb - Flush buffer event when failing workflow task (#4010)
2023-03-24 - cbb9db7 - Support empty queries when listing archived workflows (#4027)
2023-03-24 - aabbae0 - Fail QueryWorkflow if last workflow task failed (#4099)
2023-03-24 - 6b747d3 - Refresh per-namespace worker periodically (#4103)
2023-03-24 - 7c0a0e0 - Change update namespace to upsert custom search attributes (#4080)
2023-03-28 - 212332b - Add flag to enable execution scanner event Id validator (#4114)
2023-03-29 - cf61e9e - Enforce concurrent count limit to all long poll calls (#4120)
2023-03-28 - 292f460 - Bump server version for 1.20.1 patch release

Details about the v1.20.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.20.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.20.0

Release Highlights

Upgrade action

• SERVICE_FLAGS environment variable for docker images is removed, use SERVICES instead.
• The dynamic config setting matching.useOldRouting was removed and old routing is not available anymore. If you’re upgrading and did not set it earlier, set it to false across your cluster before the upgrade to minimize task dispatch disruption during deployment.

Deprecation heads-up

The following softwares will not be supported in future versions:

• MySQL 5.7: planned to be removed by v1.21; please upgrade to MySQL 8.0.17+.
• PostgreSQL 9.6 to 11: planned to be removed by v1.21; please upgrade to PostgreSQL 12+.

Advanced visibility

We’re launching advanced visibility features for SQL databases (see more below). However, this required a significant change in the existing functionality when used with Elasticsearch. Up until v1.19, when you create a custom search attribute of a given type, say Keyword, you could actually store a list of Keyword values. This will no longer be supported. If you have a search attribute of type Keyword, you will only be able to store a single Keyword value. If you wish to store a list of values, you can create a search attribute of type KeywordList. This is the only type that supports list of values, ie., there’s no equivalent for any other type (Int, Datetime, etc.).

$ temporal operator search-attribute create --name CustomField --type KeywordList

If you are using Elasticsearch, you can change the type of a Keyword search attribute to KeywordList type by re-creating the search attribute (warning: this will only work as expected if the value stored is already a list; otherwise, it will lead to unexpected errors):

$ temporal operator search-attribute remove --name CustomField
$ temporal operator search-attribute create --name CustomField --type KeywordList

For backwards compatibility, if you’re using Elasticsearch, list of values is still supported for all search attribute types. However, we discourage such usage. Furthermore, future releases of SDK will deprecate API accepting list of values, and change it to be strong typed so it won't accept list of values.

Advanced visibility on SQL DB (beta testing)

Advanced visibility features are now supported using one of the supported SQL databases (MySQL 8.0.17+, PostgreSQL 12+, and SQLite):

• Per namespace, you can create up to 3 custom search attributes for each of the types Bool, Datetime, Double, Int, Text and KeywordList, and up to 10 custom search attributes of type Keyword.
• You’ll be able to call the List API using SQL-like queries (ORDER BY and LIKE operators are not supported, check the documentation for more details).

Here are the steps to enable this feature [warning: for all commands below, please be aware of replacing the database parameters (user, password, host, port) accordingly to your use case]:

1. Upgrade Temporal Server to v1.20. If you don't want to enable advanced visibility features, then stop here. Everything should work as usual.

2. Upgrade your database to MySQL 8.0.17+ or PostgreSQL 12+ if you’re using an older version. Please check with official documentation on how to upgrade MySQL and PostgreSQL. We also recommend making a backup of your data before taking any actions.

   • [Optional step for MySQL users] If you upgraded from MySQL 5.7, convert the database character set to utf8mb4(learn more about character sets in the MySQL documentation). The following commands are an example of how you can convert both the temporal and temporal_visibility databases (check the MySQL documentation for more details on changing the character set for databases and tables):

     $ DB=temporal; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot
     $ DB=temporal_visibility; ( echo 'SET foreign_key_checks=0; ALTER DATABASE `'$DB'` CHARACTER SET utf8mb4;'; mysql $DB -u root -proot -e "SHOW TABLES" --batch --skip-column-names | xargs -I{} echo 'ALTER TABLE `'{}'` CONVERT TO CHARACTER SET utf8mb4;' ) | mysql $DB -u root -proot

3. Update the schema (if you need to set TLS cert, check our documentation for more details on the temporal-sql-tool command):

   • MySQL 8.0.17+:

     $ ./temporal-sql-tool --ep localhost -p 3306 -u root -pw root --pl mysql8 --db temporal_visibility update-schema -d ./schema/mysql/v8/visibility/versioned

   • PostgreSQL 12+:

     $ ./temporal-sql-tool --ep localhost -p 5432 -u temporal -pw temporal --pl postgres12 --db temporal_visibility update-schema -d ./schema/postgresql/v12/visibility/versioned

4. Change the sql.pluginName config to mysql8 or postgres12 accordingly.

5. Restart Temporal Server.

If you have previously created custom search attributes while using standard visibility, you’ll need to create them again. Do not delete the existing ones, just execute the command to create a new custom search attribute. For example, if you had created a custom search attribute named CustomField of type Keyword, then execute the following command (make sure you update to the latest version of Temporal CLI tool):

$ temporal operator search-attribute create --namespace default --name CustomField --type Keyword

For SQLite users, we do not support schema updates. Therefore, if you're using SQLite file-based storage, you'll need to reset your database from scratch.

Archival feature

We reimplemented the Archival feature using a separate task queue, providing higher efficiency, better resource isolation, and strong durability guarantees. Most importantly, Archival will now infinitely retry with an exponential backoff function before deleting any data. Before this change, Archival would only retry for a maximum of five minutes.

Workflow Update (alpha)

Workflow Updates enable gRPC clients to issue requests to workflow executions that mutate internal workflow state and return a response.

• This is an alpha quality release that must not be used for production workloads. Only workers using the Go SDK are supported. Workflow histories that include update events from this release may not be replayable by future releases.
• Workflow Updates are disabled by default. To enable UpdateWorkflowExecution API change frontend.enableUpdateWorkflowExecution dynamic config value to true.
• There is a new example in the samples-go repository demonstrating both how to invoke and how to handle updates.

Replication between clusters with different shard counts

Temporal Server 1.20 added capability to connect and replicate between 2 clusters with different shard count. The requirement is one cluster’s shard count is another one’s multiple (Y=NX, Y and X are shard count, and N is a natural number). Shard count is a scale unit, cluster could run into scale limit if the choose shard count is too small. This new feature make it possible to move live workload to a new cluster with larger shard count. It works the other way as well if you want to reduce your shard count.

Note: Temporal server 1.19 and earlier versions require 2 clusters to have exact same shard count if they want to be connected.

BatchOperation to support DeleteWorkflow

Add delete workflow as another supported action to BatchOperation API.

Otel 0.34

Open telemetry upgraded to 0.34. 0.34 Otel release contains a breaking change. All counters will be append a ‘_total’ suffix.

Default authorizer

If you're using the default authorizer (note that the default authorizer is not used by default, you have to explicitly configure it), you should be aware of a few changes:

• The special case for temporal-system namespace was removed. This namespace is used by various workflows that run on the system worker service.

  Separately, two recent features, Schedules and Batch v2, require the system worker to connect to namespaces other than temporal-system. If your claim mapper was not properly authenticating the system worker service and giving it a System: RoleAdmin claim, then this special case meant that system workflows would work but Schedules and Batch v2 would not.

  Now that this case is removed, your system workers might have trouble connecting to the frontend unless you make some other changes. Either 1) your claim mapper should properly authenticate the system worker and give it a System: RoleAdmin claim, or 2) use Internal Frontend to bypass the claim mapper for internode connections to frontend. See the Internal Frontend section below.

• The special case for requests with no namespace was removed, and replaced with a check for two specific requests: gRPC health checks, and GetSystemInfo. All oth...

v1.19.1

Release Highlights

This release fixes a few minor bugs related to task processing and ES bulk processor. It also enables a security check feature by default to enforce the task token check to prohibit caller from modifiying namespace info in the task token.

All Changes

2022-12-29 - 6b2e448 - Fix NPE in task channelWeightFn (#3766)
2023-01-12 - 23d8cf9 - Fix ES bulk processor commit timeout (#3696)
2023-01-12 - 7b64aa0 - Remove internal bulk processor retries (#3739)
2023-01-12 - c46fccb - Capture task processing panic (#3779)
2023-01-12 - ea8af50 - Disable eager activities for Python 0.1a1, 0.1a2, and0.1b1 too (#3793)
2023-01-12 - 5237e7f - Prioritize worker pool shutdown (#3795)
2023-01-12 - c221288 - Capture panic in replication task processing (#3799)
2023-01-12 - a8da358 - Fix recover call (#3804)
2023-01-12 - 5e784b4 - Drop task on serialization error (#3803)
2023-01-13 - 96d6b00 - Turn on frontend.enableTokenNamespaceEnforcement by default
2023-01-13 - 95a4094 - Upgrade version to 1.19.1

Details about the v1.19.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.19.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.19.0

Release Highlights

Schema changes

Before upgrading to your Temporal Cluster to release v1.19, you must upgrade your storage schema version to the following:

• MySQL schema version 1.9
• PostgreSQL schema version 1.9
• Cassandra schema version 1.7

Use the appropriate schema upgrade tool to upgrade your schema version.
For details, see https://docs.temporal.io/cluster-deployment-guide#upgrade-server.

Note that schema version is not the same as the storage type (database) version.

Golang version

• Upgraded golang to 1.19.

tctl

• tctl release with v1.17.2 (https://github.com/temporalio/tctl/releases/tag/v1.17.2).

Metric

A major refactoring has been done in the metrics code. Both metrics.Client and metrics.Scope are replaced by metrics.metricsHandler.

Customized metricsHandler is also supported by using temporal.WithCustomMetricsHandler .

Deprecated metrics

All metrics that have the suffix _per_tl are deprecated.

Improvements and fixes

Clean up mutable state when Workflow is closed

When the Workflow is closed, you can trigger a cleanup to remove Memos and Search Attributes from the mutable state. This helps reducing the size of the mutable state for closed Workflows. However, because the cleanup deletes data, it’s best to pair Advanced Visibility with Elasticsearch to ensure that Search Attributes can be retrieved. This feature is controlled by the dynamic config history.visibilityProcessorEnableCloseWorkflowCleanup.

Count limits for pending Child Workflows, Activities, Signals, and cancellation requests

This change adds a new system protection limit for the maximum number of pending Child Workflows, pending Activities, pending Signals to external Workflows, and pending requests to cancel external Workflows.
Operators can set different values for those new limits via the dynamic config. The default is currently 50,000 for each value.

limit.numPendingChildExecutions.error
limit.numPendingActivities.error
limit.numPendingSignals.error
limit.numPendingCancelRequests.error

Maximum number of concurrent pollers

The dynamic config frontend.namespaceCount controls how many concurrent pollers are allowed to connect to Temporal Server. Currently, this limit applies to the total number of pollers; that is, Activity plus Workflow pollers. In v1.19.0, this limit now applies to the number of pollers per type; that is, Activity and Workflow pollers won’t compete with each other for a connection to Temporal Server.

Batch deletion

The batch delete operation is now support in the batch operation API. Use BatchOperationDeletion option in the StartBatchOperationRequest for a batch delete operation.

History task processing

Default implementation

• Host-level Task scheduler is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler is true.
• Multi-cursor queue implementation is now enabled by default, meaning the default value of history.[timer|transfer|visibility]ProcessorEnableMultiCursor is true.

Rate limiting

• Added priority rate limiter for Task processing. By default this rate limiter is disabled but can be enabled by setting history.taskSchedulerEnableRateLimiter to true. The rate limiter should be enabled only when multi-cursor queue implementation is enabled.
• The rate is controlled by history.taskSchedulerMaxQPS for per-host limit and history.taskSchedulerNamespaceMaxQPS for per-Namespace limit. All queues (timer, transfer, visibility) share the same limit. Please note that deletion Tasks have lower priority when consuming rate limit tokens and do not count against the per-Namespace limit.
• The default value for the preceding two configurations is 0, which means they will fall back to and use the value specified by history.persistenceMaxQPS and history.persistenceNamespaceMaxQPS.

Metrics

• task_latency_processing_nouserlatency, task_latency_userlatency, task_latency_nouserlatency, and task_latency_queue_nouserlatency are removed.
• New task_latency_user metric, which captures the latency for acquiring Workflow lock in a single Task Execution attempt.

History Scavenger

A retention validation on workflow has been added in the history scavenger. This feature is enabled by fault with worker.historyScannerVerifyRetention. A default grace period on this retention validation is 90 days + namespace retention time. The grace period is configurable by worker.executionDataDurationBuffer.

All changes

2022-09-19 - 9703d33 - Post-release: bump version and upgrade dependencies (#3408)
2022-09-19 - 91c31f1 - Fix merge map of payload (#3412)
2022-09-19 - 1088382 - Fix task reschedule policy (#3413)
2022-09-20 - c672c37 - Adds a retryable error for when we try to delete open executions (#3405)
2022-09-20 - d890a2f - Propagate CloseVisibilityTaskId to DeleteExecutionVisibilityTask (#3404)
2022-09-20 - be727b3 - Retry attempts to delete open visibility records (#3402)
2022-09-21 - 8f24d1f - Revert "Retry attempts to delete open visibility records" (#3420)
2022-09-21 - 9dfdf75 - Rename parameters of MergeMapOfPayload (#3418)
2022-09-22 - ff40b89 - Retry attempts to delete open visibility records (#3402) (#3421)
2022-09-23 - c6fb6b8 - Fixes an issue where last-heartbeat time was set to the first event's timestamp (#3361)
2022-09-26 - f4af2d5 - Fix list batch operation to include division (#3431)
2022-09-27 - b4b61ff - Reorder grpc interceptors (#3423)
2022-09-27 - 85f400e - Add postgres es development script (#3429)
2022-09-27 - e3e1cce - Retry attempts to delete open workflow executions (#3411)
2022-09-27 - 769b865 - Add cluster ID into ringpop app (#3415)
2022-09-27 - 6530847 - Use original branch token instead of deserializing as branch info and then re-serializing (#3384)
2022-09-27 - d27ea89 - Fix action metrics (#3434)
2022-09-28 - 8929def - Namespace replication for failover history (#3439)
2022-09-28 - c150474 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - e0bbabe - Per-namespace workers should only run on active cluster (#3426)
2022-09-29 - b497033 - Ensure urfave/cli accepts flag values with comma (#3440)
2022-09-29 - 363d4c0 - Log warning only when there is an error in SA size validation (#3443)
2022-09-29 - 97d0ec3 - Supply optionally configured workflow settings as hints (#3442)
2022-09-29 - 47bb155 - Update dependencies and pin otel (#3444)
2022-09-29 - 3815915 - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-09-30 - 3099274 - Fix reset workflow in replication reapply (#3449)
2022-10-03 - 175b916 - Use safer default TIMESTAMP for MySQL. (#3424)
2022-10-04 - a86d455 - Index history size when workflow closes (#3451)
2022-10-05 - fc37cd4 - Update ns version history in handover state (#3456)
2022-10-06 - 0ae4cc5 - Add config filter by task type (#3455)
2022-10-06 - 0df719d - Update replication use branch token (#3447)
2022-10-10 - fa51f5d - Move record child workflow completed to api package (#3350)
2022-10-10 - 62cd143 - Move verify child workflow completion recorded to api package (#3351)
2022-10-10 - 71a1de3 - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - bc451db - Compare task type enum (#3464)
2022-10-10 - c6b472e - Fix exclude tags with withTags method (#3466)
2022-10-10 - 158737a - Remove old logic for checking workflow deletion task dependencies from delete_manager (#3427)
2022-10-10 - d33559a - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-11 - 0e70cf7 - Move get / poll mutable state to api package (#3467)
2022-10-11 - 7278168 - Move describe workflow to api package (#3469)
2022-10-11 - 0dddd3c - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-11 - 429c0af - Move replication APIs to api package (#3472)
2022-10-11 - d90f3ab - Move NDC logic to ndc package, move workflow reset to api package (#3465)
2022-10-11 - 740c7a3 - Move reapply events to api package (#3476)
2022-10-11 - d252656 - Move remove signal from mutable state to api package (#3475)
2022-10-11 - b292c22 - Move delete workflow to api package (#3473)
2022-10-11 - 32ea0bf - Move refresh workflow to api package (#3477)
2022-10-12 - 16184da - Fix scheduled queue max read level update for single processor (#3474)
2022-10-12 - ceae241 - Handle sync workflow state task in replication dlq handler (#3482)
2022-10-13 - bf8e1f1 - Bump UI to v2.7.0 (#3480)
2022-10-13 - 46dcb4f - Turns on the history scavenger for SQL backends (#3462)
2022-10-13 - 86fde8e - Move query workflow to api package (#3486)
2022-10-13 - ef21405 - Fix sanitize mutable state after replication (#3479)
2022-10-13 - 457eb05 - Clean up duplicate empty task id (#3490)
2022-10-13 - a0ea177 - Add execution scavenger for retention (#3457)
2022-10-15 - a4fe3a7 - Remove UI v1 from development environment (#3485)
2022-10-17 - dc84e87 - Update replication timestamp with no task (#3487)
2022-10-17 - 8e5ccb5 - schama -> schema (#3501)
2022-10-17 - b8d46a5 - Do not add version 0 to failover history (#3483)
2022-10-17 - cebbde9 - Sync proto API 1.12.0 (#3500)
2022-10-17 - 968c576 - Create visibility GetWorkflowExecution API (#3488)
2022-10-18 - ecde543 - Remove now parameter from task generator interface (#3478)
2022-10-18 - 9cb646b - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 8a7538a - Warning log on new event during set workflow (#3508)
2022-10-19 - 8d645e8 - Increase visibility RPS an...

v1.18.5

Release Highlights

This release fixes minor bugs in history scavenger, gocql client recovery logic, and internal queue processing.

All Changes

2022-11-15 - 00bb513 - Update version to 1.18.5
2022-11-15 - f103b19 - Fix history scavenger bug on delete mutable state (#3588)
2022-11-15 - ec332ef - Wrap gocql.Iter with custom wrapper (#3577)
2022-11-15 - 077c5e0 - Fix resilient shard (#3584)
2022-11-15 - 0f68d26 - Queue processor handle shard ownership lost (#3553)

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.18.4

Release Highlights

This release fixes a few minor bugs and upgrades the gocql dependency. It also upgrades the expat package in the admin-tools Docker image for CVE-2022-40674 (this doesn't affect the Temporal server itself).

All Changes

2022-10-27 - 0616c5c - Port util.CloneMapNonNil to release branch (#3530)
2022-10-31 - 327d46e - Upgrade gocql dependency to v1.2.1
2022-10-31 - 69e2bb9 - Change execution scavenger to call admin delete (#3526)
2022-10-31 - c6cfe6d - Fix replication task nil case (#3531)
2022-10-31 - 26782e9 - Add retention validation in history scavenger (#3541)
2022-10-31 - fe6a873 - Take rpc address from config for local cluster (#3546)
2022-10-31 - 03d6c03 - Update version to 1.18.4

In docker-builds:

2022-10-31 - 85d405a - Use larger GitHub-hosted runners for Docker builds (# 74)
2022-10-31 - e85d0cc - Upgrade expat in admin-tools (# 75)
2022-10-31 - 07e4b80 - Update temporal submodule for branch release/v1.18.x

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools