v1.18.3

Release Highlights

This release fixes a bug that was introduced in 1.18.2 that would prevent the server from starting in a single-node configuration.

All Changes

2022-10-19 - 5709090 - Fix scanner start dep (#3513)
2022-10-20 - c0eb02c - Update version to 1.18.3

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.18.2

This release has a bug that prevents server startup when running in a single-node configuration. Please use v1.18.3 instead.

Release Highlights

This patch releases fixes a few small bugs in namespace migration.
It also has two changes to scavengers: the history scavenger is now enabled on SQL persistence, and the execution scavenger now finds mutable state that's past its retention timeout.

All Changes

2022-10-17 - 1b8e28f - Implement GetAllHistoryTreeBranches for SQL persistence backends (#3438)
2022-10-17 - 2127976 - Properly handle min task ID > max task ID case during shard re-balancing (#3470)
2022-10-17 - d602bff - Fix timer task visibility timestamp for workflow refresh (#3460)
2022-10-17 - 8cd481a - Fix scheduled queue max read level update for single processor (#3474)
2022-10-17 - 45bd55d - Turns on the history scavenger for SQL backends (#3462)
2022-10-17 - bb7b1f4 - Fix sanitize mutable state after replication (#3479)
2022-10-17 - ff47b2c - Add execution scavenger for retention (#3457)
2022-10-17 - 78366f1 - Update replication timestamp with no task (#3487)
2022-10-17 - b57b930 - Do not add version 0 to failover history (#3483)
2022-10-18 - 9a427c2 - Rewrite mysql PaginateBranchesFromHistoryTree query (#3509)
2022-10-18 - 07ab3e1 - Warning log on new event during set workflow (#3508)
2022-10-18 - 0aa5948 - Update version to 1.18.2

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.18.1

Release Highlights

This patch release fixes a few minor issues with search attribute/memo upsert, task rescheduling, schedule api validation, replication, and batch operations. We recommend that everyone upgrade to it.

All Changes

2022-09-28 - 0603535 - Fix merge map of payload (#3412)
2022-09-28 - ac2593c - Fix task reschedule policy (#3413)
2022-09-28 - ca5d0fd - Reorder grpc interceptors (#3423)
2022-09-28 - 88d6fc9 - Validate structured calendar specs and improve error messages (#3425)
2022-09-28 - 02e2edf - Per-namespace workers should only run on active cluster (#3426)
2022-09-28 - 3c0105b - Fix list batch operation to include division (#3431)
2022-09-28 - adc4cc8 - Fix action metrics (#3434)
2022-09-28 - 70ca71b - Namespace replication for failover history (#3439)
2022-09-29 - f84be21 - Log warning only when there is an error in SA size validation (#3443)
2022-10-10 - 082713b - Fix reset workflow in replication reapply (#3449)
2022-10-10 - 1e46237 - Add config filter by task type (#3455)
2022-10-10 - 2f6df41 - Use separate metric for resource exhausted error in task processing (#3463)
2022-10-10 - e2e7474 - Update replication use branch token (#3447)
2022-10-10 - f089fda - Update ns version history in handover state (#3456)
2022-10-10 - cb2f574 - Compare task type enum (#3464)
2022-10-10 - 78a5e36 - Fix exclude tags with withTags method (#3466)
2022-10-10 - 1a6bae2 - Prepare 1.18.1 release

Details about the v1.18.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.18.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.18.0

Release Highlights

Upgrade action item summary

• Before upgrade
  • If you’re using ES v6: do not deploy this version until after moving to ES v7 or later
  • If you’re using ES v7+: upgrade schema to include new builtin search attributes (see Schedules section below)
  • If you used the experimental Schedules feature in 1.17: delete all schedules and recreate them after upgrading
• After upgrade
  • Change dynamic config matching.useOldRouting to false on all nodes as close in time as possible. This will cause temporary matching disruption during the propagation.
  • Consider enabling host-level worker pool and multi-cursor task processing (see Task processing section below)
  • Consider setting persistence rate limits (see Persistence rate limiting section below)

Elasticsearch v6 support is removed

Elasticsearch v7 became the default supported version in 1.12.0 release. In 1.18.0, Elasticsearch v6 support is completely removed. If you are still using Elasticsearch v6, don't upgrade to 1.18.0. Upgrade Elasticsearch first following the migration guide.

Along with v6 support removal, we added Elasticsearch v8 support. Elasticsearch v8 doesn’t have breaking changes that affect Temporal.

Batch API

This release introduces new batch operation APIs in the Frontend Service. The feature is enabled by default.

• StartBatchOperation: Start a batch operation.
• StopBatchOperation: Stop a running batch operation.
• DescribeBatchOperation: Get detail information about a batch operation.
• ListBatchOperations: List all the batch operations.

By default, the APIs support one concurrent operation per Namespace. This value is configurable by dynamic config: frontend.MaxConcurrentBatchOperationPerNamespace.

Use DescribeBatchOperation or metrics batcher_processor_requests and batcher_processor_errors to monitor the progress of batch operations.

Cluster API

The following Cluster APIs moved from the admin service to the operator service. Those APIs will be deprecated in the admin service in a future release.

• AddOrUpdateRemoteCluster: Add or update a connection configuration to a remote Cluster.
• RemoveRemoteCluster: Remove a connection to a remote Cluster.
• ListClusters: List the configuration of all connected Clusters.

Task processing

Host level task scheduler

• Host level task scheduler now uses separate task channels for each namespace for better resource isolation.
• Deprecated the dynamic config history.[timer|transfer|visibility]TaskHighPriorityRPS.

Multi-cursor queue

New multi-cursor queue implementation for better resource isolation and handling cases not covered by host-level task scheduler: too many pending tasks, stuck queue ack level, etc.

• The new implementation can be enabled by setting the value of dynamic config history.[timer|transfer|visibility]ProcessorEnableMultiCursor to true.
• The host-level worker pool for the corresponding queue should also be enabled, otherwise the above dynamic config won’t take effect. This can be done by setting history.[timer|transfer|visibility]ProcessorEnablePriorityTaskScheduler to true.

Task Loading

• The dynamic config history.[timer|transfer|visibility]ProcessorMaxPollHostRPS can be used to limit the throughput of the queue processor. This is very useful for recovering from a persistence outage which leads to a large task queue backlog in persistence. Set the value to a small number and gradually increase to ensure a smooth draining of the backlog. By default, value for this config is 0 and will fallback to 30% of the history.persistenceMaxQPS for transfer and timer queue and 15% for visibility queue.

Metrics

New task processing related metrics are added for better visibility.

• task_latency_load: measures the duration from task generation to task loading (task schedule to start latency for persistence queue).
• task_latency_schedule: measures the duration from task submission (to task scheduler) to processing (task schedule to start latency for in-memory queue).
• queue_latency_schedule: measures the time it takes to schedule 100 tasks in one task channel in the host level task scheduler. If there are less than 100 tasks in the task channel for 30s, the latency will be scaled to 100 tasks upon emission. NOTE: this is still an experimental metric and is subject to change.

Retry behavior

Consolidated retry logic for APIs calls to two places: service handler (by an interceptor) and service client for calling other temporal services or persistence. Limited retry maximum attempts and removed retry logic in other places to avoid potential retry storm.

Persistence rate limiting

Persistence layer rate limiter prioritizes user requests (e.g. start workflow, signal workflow, etc.) over system background requests (e.g. task loading, requests incurred by task processing, replication, etc.)

Persistence max QPS for each namespace can be set by tuning the dynamic config [frontend|history|matching|worker].persistenceNamespaceMaxQPS. By default, this value is 0 and will fallback to the overall host persistence QPS set by [frontend|history|matching|worker].persistenceMaxQPS.

Persistence metrics persistence_requests, persistence_latency and persistence_error* now also contains namespace tag for better observability.

UpsertMemo

This release adds UpsertMemo to modify existing workflow memo. It works similarly to UpsertSearchAttributes. However, on the visibility side, the memo is updated only when used with advanced visibility (i.e. Elasticsearch). When used with standard visibility (SQL databases), the memo is not being updated currently, and the workflow memo can be retrieved correctly only from the mutable state (that is, you can call DescribeWorkflowExecution to retrieve it).

Matching routing

Task queues are now distributed among matching nodes based on namespace and type, for better load balancing. This change is disabled by default to avoid disruption during the upgrade. After upgrading to 1.18, you should make a dynamic config change on all nodes simultaneously to set matching.useOldRouting to false. You can use the following snippet:

# Remove this after upgrading to 1.19:
matching.useOldRouting:
- value: false

This change will cause a short disruption to Task dispatch as nodes reload the dynamic config and then Task Queues get moved between matching nodes. You’ll also see some persistence errors in logs for a few minutes.

We plan to make this routing the default in 1.19, so if you don’t make the dynamic config change, it’ll happen during the next upgrade.

Schedules

A few incompatible changes were made to the Schedules feature, which was introduced as experimental in 1.17. If you have created any Schedules, you should delete them before upgrading to this release, and recreate them after the upgrade.

The Schedule feature is now enabled by default.

If you're using Advanced Visibility (i.e. Elasticsearch), Schedules don’t appear in Workflow lists anymore. If you’re not using Elasticsearch, Schedules continue to appear in Workflow lists for now.

There is one new builtin search attribute to support Schedule visibility. You can add it to Elasticsearch by running the upgrade script from the server repo. See the top of the script for environment variables to use to point it at your Elasticsearch. You can do this before or after the upgrade, but before creating any Schedules. (If you didn’t perform the v2 upgrade when upgrading to 1.17, which also added search attributes for Schedules, also do that now.)

schema/elasticsearch/visibility/versioned/v3/upgrade.sh

Frontend connections

Update: these release notes previously described a new method of making internal frontend connections. This new method works in many, but not all, server configurations (specifically related to mTLS and custom authorizers). To make it work for all configurations, we're going to make some more changes to the feature. In the meantime, there's no need to change any configuration. (If you've already made the config change that was suggested here and it's working, there's also no need to change it back.)

Maximum retention

The maximum Namespace retention limit of 30 days is removed. Namespaces now can use any retention as needed, as long as the persistence has enough capacity for the storage.

Dynamic config interface

The dynamicconfig.Client interface was changed and simplified.

If you’re using Temporal with our pre-build container images or binaries, there’s nothing to do.

If you’re building your own Temporal binary and only refer to dynamicconfig.Client or dynamicconfig.NewFileBasedClient in your ServerOptions, you should be able to rebuild with no code changes.

If you’ve written a custom dynamic config implementation, you’ll need to change it to the new interface. This should be pretty straightforward, but if you have any questions, please contact us; we can help.

All changes

2022-06-21 - 231655c - Rename queryTermination to queryCompletion (#3000)
2022-06-22 - 73881a3 - Prepare for 1.18 release (#3009)
2022-06-22 - f86b8d2 - Per-service fx-ified OTEL tracing (#2896)
2022-06-22 - c7d831b - Explicitly specify timezone for TIMESTAMP values (#3012)
2022-06-22 - 958bde7 - Update mysql image version to support arm64 (#3013)
2022-06-23 - 6cab7e5 - Add developer do...

v1.17.6

Release Highlights

This release includes a fix for an issue where a task queue could fail to process backlogged tasks if it's loaded or reloaded while persistence is unavailable.

All changes

2022-09-13 - 0fcc1b6 - Prepare 1.17.6 patch
2022-09-13 - f4b94d2 - Move build proto dependencies to separate go.mod (#3377)
2022-09-13 - 6e592f6 - Wait to acquire lease in matchingEngine (#3033)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.6)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.17.5

Release Highlights

This release includes a bug fix for Retry Policy logic in Frontend Service handler. If you are not using the custom Search Attributes mapper, you are not affected.

All changes

2022-08-30 - 9bbd336 - Prepare 1.17.5 patch
2022-08-30 - 4e50c8f - Remove frontend handler retries (#3285)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.5)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.17.4

Release Highlights

This release includes bug fixes for Archival and Retry logic, and removes 30 days max Retention Period limit.

IMPORTANT NOTE FOR ARCHIVAL USERS
If you are using the Archival feature and if v1.17.4 of the History Service runs with an older version of the Worker Service during the upgrade, it is possible to have a "dangling Workflow mutable state" where the Event History has been deleted.
To avoid that when upgrading to this release, perform the following steps:

1. Stop the old version of the Worker Service.
2. Deploy this patched release.
3. Start the Worker Service after the upgrade.

All changes

2022-08-16 - 7f72106 - Prepare 1.17.3 patch
2022-08-16 - 06d641f - Fix formating
2022-08-16 - 4426702 - Fix failover queue shutdown (#3232)
2022-08-15 - 0bd7435 - Fix archival activities error handling (#3227)
2022-08-12 - f8a59f7 - Use detached context for shard context operations (#3194)
2022-08-12 - dae8a51 - Fix queue processor throttling logic (#3195)
2022-08-12 - cea0709 - Remove namespace max retention (#3148)
2022-08-12 - cdb9ae0 - Add version check for delete workflow transfer task (#3159)
2022-08-12 - c61450f - Fix inline history archival (#3216)
2022-08-12 - ba452a2 - additional history scavenger heartbeat (#3186)
2022-08-12 - 5b05741 - Simplify system retry logic: Part 1 (#3172)
2022-08-12 - 51817aa - Use a prefix for scheduler workflow ids (#3201)
2022-08-12 - 4b7507d - Improve task processing retry behavior (#3218)
2022-08-12 - 23e40a9 - Fix shard close error type (#3215)
2022-08-12 - bd22047 - Run one per-ns worker per namespace instead of namespace × component (#3116)
2022-08-12 - 6aaddf3 - Run task rescheduler in background (#3038)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.4)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.17.2

Release Highlights

This release includes fixes for metrics, rate limiters and Elasticsearch based visibility performance optimizations.

All changes

2022-08-02 - a39ba31 - Prepare 1.17.2 patch
2022-08-02 - 3b8f5c9 - Fix tally timer metric (#3173)
2022-08-02 - 26af874 - Cherry Pick: Adds ability to specify sanitizeOptions for prometheus metrics (#3170) (#3174)
2022-07-29 - 7d9b262 - Core persistence priority rate limiting (#3139)
2022-07-29 - d186259 - Don't dispatch expired tasks from taskReader buffer (#3161)
2022-07-29 - 28cd79b - Remove RunId from visibility query sort for ES 7 (#3140)
2022-07-29 - 0b28e9a - Consolidate rpc client metrics logic (#3147)
2022-07-29 - 478582c - Generate most of the gRPC wrappers in client (#3001)
2022-07-29 - efc3203 - Add config to disable ORDER BY clause (#3128)
2022-07-29 - 4a5f6ed - Use default consistency level for visibility DeleteWorkflowExecution (#3132)
2022-07-29 - b48cf2a - Fix timer metric bucket (#3136)
2022-07-29 - 84149d3 - Fix task rate limiting (#3137)
2022-07-29 - 4d7772d - Fix queue rate limiting busy loop (#3111)
2022-07-28 - a1d1650 - Config worker use public frontend endpoint (#3162)
2022-07-28 - 8dea880 - Update shard replication timestamp with max generated task (#3158)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.2)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.17.1

Release Highlights

This release includes fixes for replication stack, delete namespace functionality, and potential deadlock.

All changes

2022-07-11 - e88aa24 - Prepare 1.17.1 patch
2022-07-11 - 13cd5a5 - Reorder frontend rate limiter (#3080)
2022-07-08 - 0b9f15c - Resource aware operation retry (#3069)
2022-07-08 - 56c5295 - Do not double execute close execution task if visibility archival is disabled (#3072)
2022-07-08 - 646f12d - Fix namespace failover callback (#3066)
2022-07-08 - 6737693 - Handle workflow already completed when verifying first workflow task scheduled (#3061)
2022-07-08 - 12c173c - Fix schedule jitter calculation bug (#3059)
2022-07-08 - a95c2fc - Fix error shadowing in standby executors (#3053)
2022-07-08 - 0b01a72 - Improve error handling for standby transfer task verification (#3050)
2022-07-08 - 3278111 - Clean up history branch ancestor operation (#3046)
2022-07-08 - fb208d5 - Cancel activity when eager execution and request cancellation are in the same WFT (#3029)
2022-07-08 - 6513af8 - Release shard lock earlier during delete workflow execution (#3028)
2022-07-08 - 4657f0f - Don't check delete workflow execution transfer task version (#3021)
2022-07-06 - 972af2b - Do not hold lock when running namespace change callback (#3067)
2022-06-30 - 0f13b23 - add support for multiple pragma statements and driver syntax (#3031)
2022-06-30 - c6810d5 - Update backfill history in sync workflow state (#3042)

Details about v1.17.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.17.1)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools

v1.16.3

Release Highlights

This release includes fixes for replication stack as well as for shard lock scope.

All changes

2022-07-11 - 83c2b6a - Prepare 1.16.3 patch
2022-07-11 - afd374d - Cherry pick #3080 for 1.16 (#3081)
2022-07-08 - 082d7c9 - Release shard lock earlier during delete workflow execution (#3028)
2022-06-01 - 05a90f8 - Move cluster ack level on ack level update (#2927)
2022-06-01 - 021d501 - Fix dlq retry policy
2022-06-01 - 360062b - Fix failover queue creation after shard reload (#2862)
2022-06-01 - 79102bf - Handle NDC active -> passive transition with transient workflow & no buffered event
2022-06-01 - bd87aa3 - Minor tweak to migration workflow wait replication check (#2839)
2022-06-01 - aad0f85 - Misc changes to migration workflow (#2838)
2022-06-01 - b7562a4 - Check workflow task after reapply events (#2840)
2022-05-20 - 3fff3f0 - Fix record child completion mutable state stale check (#2885)

Details about v1.16.0 release can be found here.

Helpful links to get you started with Temporal

Temporal Docs
Server
Docker Compose
Helm Chart

Docker images for this release (use tag 1.16.3)

Server
Server With Auto Setup (what is Auto-Setup?)
Admin-Tools