Assets 2

This is a bug fix release that addresses the following issues:

  • Status "Succeeded" on Application Insights when orchestrator failed (#362, #411)
  • Orchestrator stuck waiting for activity task completion (#576, Azure/durabletask#241)
  • Race condition with RaiseEventAsync and ContinueAsNew (#572)
  • Fixes an issue in the Durable Task Framework where messages can be delayed for up to 5 minutes in certain failure scenarios (#555, Azure/durabletask#245)

This release can be downloaded from nuget.org:
https://www.nuget.org/packages/Microsoft.Azure.WebJobs.Extensions.DurableTask/1.7.1

For more information on Durable Functions, see our official documentation:
https://docs.microsoft.com/azure/azure-functions/durable/durable-functions-overview

@cgillum cgillum released this Nov 30, 2018 · 19 commits to master since this release

Assets 2

Overview

The v1.7.0 release is a minor update for Durable Functions. While it is "minor" according to the semver definition, it is a significant release in terms of the actual payload. There are many new features, improvements, important bug fixes, and is fully compatible with both Functions 1.0 and Functions 2.0. It also includes support for command-line management of orchestrations via the Azure Functions Core Tools.

Thanks to everyone who contributed, either via code, asking good questions, or by filing issues! Special thanks goes to @gled4er, @TsuyoshiUshio, @cmkerner20 (one of our Summer interns), @k-miyake (first-time contributor), @mlankamp (first-time contributor) and @brandonh-msft for their community contributions, including code and documentation!

New Features / Enhancements

New features are primarily oriented around orchestration instance and data management.

  • Orchestration history purging (#498): A new API has been added which allows purging the history of completed, failed, or terminated orchestration instances. This is useful for saving space, or for compliance with data retention policies (docs). Contributed by @gled4er.
  • Orchestration status query paging (#473): The instance query HTTP API has been enhanced to support paging. This is useful when a task hub contains hundreds or thousands of orchestrations and you want to list them all (docs). Contributed by @k-miyake and @TsuyoshiUshio.
  • HTTP API for starting new orchestration (#61): Previously, starting orchestration instances required writing a trigger function. It is now possible to start a new orchestration instance via a built-in HTTP API (docs coming soon).
  • NewGuid() API for generating orchestrator-safe GUIDs (#514): Orchestrator functions are not allowed to create random GUID values since random data is forbidden. With this new DurableOrchestrationContext.NewGuid() API, it's now possible to create randomly generated GUID/UUID values in a way that is safe for replays (docs). Contributed by @gled4er.
  • Ability to hide inputs from instance query results (#490): By default, orchestration inputs are shown when querying orchestration instances using the HTTP API. A new showInput parameter has been added which allows the caller to remove the inputs from the response (docs).
  • Improved mocking capabilities for activity functions (#494): By popular demand, we've added a new DurableActivityContextBase class that can be used for mocking activity functions (docs). Contributed by @gled4er.
  • Adding support for configuring task hub name from application settings (#503): This is important when automating the deployment of multiple versions of an application. Contributed by @gled4er.
  • New overloads for RaiseEventAsync and WaitForExternalEvent APIs (#517): Non-generic overloads for these methods now exist for when events do not utilize data payloads. Contributed by @brandonh-msft.

Bug Fixes

There are some very significant bug fixes in this release, making it a highly-recommended upgrade.

  • Excessive storage cost running durable functions (#508, #462). Thanks to @shibayan for finding this bug!
  • High latency between scheduling and starting orchestrator (#449)
  • Internal message loss due to partition movement, resulting in hung orchestration (#481, #460, #412)
  • Large exception payloads will hang orchestration instances (#502, #518)
  • NullReferenceException in query string parsing code (#470)
  • Incompatible Microsoft.Extensions.* dependencies for Functions 1.0 projects (#446). Contributed by @mlankamp.
  • "Invalid table name" error if task hub name contains special characters (#505). Contributed by @gled4er.
  • Failed JavaScript orchestrations incorrectly report "Non-Deterministic workflow detected" (#516)

Misc.

  • Durable Functions documentation has been reorganized under a new, dedicated sub-topic (see below)!
  • Source-link support, including embedded PDBs for easier debugging into extension code (#528). Contributed by @gled4er.
  • Status query results now show blob storage URLs for large messages
  • DurableOrchestrationStatus objects are now serializeable (#485). Contributed by @gled4er.

Introducing Durable Functions commands in Azure Functions Core Tools

Worth calling out separately, this release also includes support for a new Durable Functions-specific commands in the next release (>= 2.3.131) of the Azure Functions Core Tools for Azure Functions 2.0. These commands are intended to enable users to manage their orchestrations without the need for calling REST APIs.

The GitHub Pull Request for these updates can be found here: Azure/azure-functions-core-tools#838).

Supported commands include:

  • func durable start-new ... to start a new instance of an orchestrator function
  • func durable get-runtime-status ... to query the status of a running orchestration
  • func durable get-instances ... to query for orchestrations based on their status, creation time, and/or completion time
  • func durable terminate ... to terminate orchestration instances
  • func durable raise-event ... to send event notifications to running orchestrations
  • func durable rewind ... to resurrect failed orchestration instances (rewind functionality is currently in preview)
  • func durable purge-history ... to purge history data from Azure Storage based on a particular time range
  • func durable delete-task-hub ... to permanently delete all Azure Storage data associated with your durable function app (queues, tables, and blobs).

These commands work by making calls directly to Azure Storage, so they do not require the durable function app to be actually running. They also work with both real Azure Storage accounts and the Azure Storage Emulator on Windows devices. More details can be found in the Instance Management documentation topic. Originally contributed by @cmkerner20.

New Documentation Layout

Previously the Durable Functions documentation was mixed in with the regular Azure Functions documentation in a way that made it fairly difficult to navigate. As part of this release, we have also revamped the organization of the docs into their own sub-section. The new home for the Durable docs can be found here: https://docs.microsoft.com/azure/azure-functions/durable. It has the following sections:

  • Overview: Go here to learn about Azure Functions, Durable Functions, and to learn how they compare to other similar products.
  • Quickstarts: Go here to get started coding with Durable Functions. Language specific guides currently exist for C# and JavaScript.
  • Tutorials: Learn how to accomplish basic tasks using Durable Functions.
  • Samples: Find sample end-to-end applications that use Durable Functions.
  • Concepts: Documentation for all of the foundational features of Durable Functions.
  • How-to-guides: Guides for application patterns, monitoring, troubleshooting, performance, etc.
  • Reference: HTTP API reference (eventually we'll move the language-specific API references here as well).

Be sure to check out the new documentation layout. If there's anything you think is missing or needs fixing up, feel free to leave us feedback or open a docs PR using the links on the page!

Note that some of the old links might return 404 currently. We're working on fixing those so that they redirect to the new locations.

Installation Instructions

The latest nuget package can be downloaded here: Microsoft.Azure.WebJobs.Extensions.DurableTask. The package is digitally signed by Microsoft to ensure its integrity and authenticity.

@cgillum cgillum released this Sep 24, 2018 · 52 commits to master since this release

Assets 2

Overview

The v1.6.2 release is a patch update for Durable Functions. Here are the important details:

  1. This is a GA release for the Azure Functions 2.0 runtime, which is also now GA).
  2. The assemblies in this release are strong-name signed.
  3. This release is compatible with the v1.0.0 release of Durable JS for JavaScript function apps on Functions 2.0.

All samples (with the exception of WebJobs) have also been updated to target Azure Functions 2.0 using this release.

For more information about Durable Functions and how to get started, please see our official documentation on docs.microsoft.com: https://docs.microsoft.com/azure/azure-functions/durable-functions-overview

@cgillum cgillum released this Aug 30, 2018 · 65 commits to master since this release

Assets 2

Overview

The v1.6.0 release is a minor update for Durable Functions. It contains new features, various improvements, and bug fixes. The two biggest areas of focus are:

  1. Performance and reliability improvements for all users
  2. Azure Functions v2.0 (preview) breaking change compatibility

Regarding #2, all Azure Functions v2 users will need to upgrade to this release. Previous versions of Durable Functions are no longer expected to work with Azure Functions v2 as of the most recently announced v2 update. Please see the Functions v2 Breaking Changes section below for additional notes and instructions.

Thanks to everyone who contributed, either via code, asking good questions, or by filing issues! Special thanks goes to @TsuyoshiUshio, @markheath, @yuka1984 for their community contributions (including code and documentation)! And extra special thanks goes to @cmkerner20, one of our summer interns, who contributed a new "orchestration rewind" feature!

New Features

  • Rewind failed orchestrations (preview, #301): A new API has been added which allows you to "rewind" failed orchestration instances back to their previously good state. This can be a life-saver when you have long-running orchestrations that fail in unexpected ways. Thanks to @cmkerner20 for this contribution! Official documentation and a blog post is coming soon.
  • Instance query filters (#358): In the v1.5.0 release, we enabled users to query for all orchestration instances in their task hub. In this release, we've added support for filtering those results based on instance status, their creation date, or their completion date. Thanks to @TsuyoshiUshio for this community contribution!

Improvements

  • Orchestration throughput (#368): We made some big improvements to the underlying DurableTask.AzureStorage library which resulted in a ~25% throughput improvement in workloads involving large numbers of concurrent orchestration instances.
  • WaitForExternalEvent with built-in timeouts (#365): Waiting for external events with a timeout has always been possible but is clumsy to implement. In this release, a timeout parameter has been added to this API, greatly simplifying this common pattern. Thanks to @markheath for this community contribution!
  • Improved cross-function app event raising (#406): A new taskHub parameter has been added to the RaiseEventAsync API which allows a function in one function app to easily raise events to an orchestration running in another function app. Thanks to @yuka1984 for this community contribution!
  • Authenticode signing (#203): All Durable Functions and Durable Task Framework nuget packages and assemblies are now authenticode-signed by Microsoft, guaranteeing their integrity and authenticity.
  • Improved internal telemetry: When running orchestrations in Azure, detailed ETW metrics and telemetry are collected automatically and made available to customer support engineers. There are many telemetry improvements in this release which will make it easier for our team to investigate issues when users open support requests.

Bug Fixes

  • Messages are not always processed in the order they are received (#400)
  • Fan-out/Fan-in results in Host Threshold Exceeded (Connections) (#389)
  • Orchestration instances hang until host restart after history fetch failure (#376)
  • Concurrency throttle implementation slows down processing unnecessarily (#370)

Durable Task Framework changes

The names of the dependent Durable Task Framework nuget packages have changed.

Functions v2 Breaking Changes

It was recently announced that major Azure Functions v2 breaking changes were coming (please subscribe to the announcements repo if you were not aware of this!). This release is intended to be compatible with the announced breaking changes. The sections below will go into further detail.

Azure Functions compatibility

These compatibility constraints apply to both the Azure Functions runtime running in Azure as well as the Azure Functions CLI tools.

  • This release is compatible ONLY with Functions v2.0.12050-alpha and above.
  • This release is NOT compatible with earlier versions of Functions v2. For earlier versions, you must use Durable Functions v1.5.0.
  • This release is compatible with and recommended for ALL recent versions of Functions v1.x.

Host.json breaking changes

All host.json settings for Durable Functions must now be placed inside an "extensions" section (other non-durable trigger-specific settings have also moved here). For example:

{
  "version":"2.0",
  "extensions": {
    "durableTask": {
      "hubName": "MyTaskHubV2",
      "azureStorageConnectionStringName": "storagewestus12345"
    }
  }
}

If you do not make these changes, then the default settings will be used and there are no warnings or errors to notify you about this.

REST API breaking changes

The Azure Functions runtime controls the URL prefixes for all built-in extension endpoints. With the v2 release, the /DurableTaskExtension/ segment of each URL has been changed to /durabletask/. For example, when querying for the status of an orchestration instance:

GET /runtime/webhooks/DurableTaskExtension/instances/{instanceId}?code=XYZ

...now becomes...

GET /runtime/webhooks/durabletask/instances/{instanceId}?code=XYZ

Additional notes related to this breaking change on how this may impact you:

  • Hardcoded clients will need to be updated to work with the new URL formats.
  • External clients that are polling existing long-running instances (e.g. from a Logic App or from some custom code) will start seeing 404 responses.
  • External clients should not be impacted for new instances they create if they follow the recommended HTTP discovery patterns.
  • Users have the option of implementing all of our built-in webhooks with very simple, custom HTTP functions if they want to avoid being broken by the URL changes.

These updates will be reflected in the HTTP API documentation in the near future.

Installation Instructions

The latest nuget package can be downloaded here: Microsoft.Azure.WebJobs.Extensions.DurableTask

Please see our installation documentation for instructions for all supported platforms.

@cgillum cgillum released this Jun 22, 2018 · 79 commits to master since this release

Assets 2

Overview

The v1.5.0 release is a minor update for Durable Functions. It contains new features, various improvements, and bug fixes. The biggest focus was on improving on some key pain points since the GA release, as well as enabling support for the latest Azure Functions v2 runtime.

Thanks to everyone who contributed, either via code, asking good questions, or by filing issues! Special thanks goes to @gled4er, @TsuyoshiUshio, and @yuka1984 for their community contributions (including code and documentation)! The specific contributions are called out below.

New Features

  • Support for complex object parameters in JavaScript orchestrations: Previously only primitive types (strings, numbers, etc.) were supported.
  • New CreateHttpManagementPayload API to get status/terminate/raise-event URLs (#155): Previously only HTTP-triggered functions could discover management endpoint URLs. With this API, any function trigger can now discover this information via DurableOrchestrationClient. See the Instance management documentation for more details. Thanks @gled4er for this community contribution!
  • New APIs to enumerate all orchestrations (#323): Previously we required users to track orchestration instance IDs themselves externally. We now expose an API which will enumerate all orchestration instances in the task hub via a .NET API as well as a REST API. Thanks @TsuyoshiUshio for this community contribution!

Improvements

  • Adds support for the latest Azure Functions v2 runtime (#342).
  • Application Insights tracking logs now omit replay executions by default. Note that this is a behavior change. See the Diagnostics topic for more details. Thanks @gled4er for this community contribution!
  • Azure Event Grid integration improvements, including publishing retry configuration and app settings-based endpoint configuration. Thanks @yuka1984 for these community contributions!
  • Various telemetry improvements. This will make it a lot easier for us to help you diagnose orchestration runtime issues in Azure.
  • The official nuget package is now digitally signed.

Also note that due to an internal policy change on nuget.org, the dependent DurableTask.AzureStorage and DurableTask.Core nuget packages have been temporarily renamed to DurableTask.AzureStorage.Private and DurableTask.Core.Private.

Installation Instructions

The latest nuget package can be downloaded here: Microsoft.Azure.WebJobs.Extensions.DurableTask

Please see our installation documentation for instructions for all supported platforms (be sure to always select the latest version of the extension).

Note that the Azure Functions portal, the Azure Functions CLI, and the Visual Studio tools for Azure Functions and Azure WebJobs may run into issues when using the latest version of the Durable Functions extension at the time of writing. Updates for these tools are in the works.

May 10, 2018

@cgillum cgillum released this May 7, 2018 · 95 commits to master since this release

Assets 2

Overview

This is the very first GA release for Durable Functions. It contains new features, various fixes, and a few small compile-time breaking changes (last-minute public surface area cleanup). The biggest focus was on improving performance and scale.

This is an officially supported release. Subsequent 1.x releases will be non-breaking upgrades.

Thanks to everyone who contributed, either via code, asking good questions, or by filing issues! Special thanks goes to @gled4er, @TsuyoshiUshio, @mikhailshilkov, @CarlosSardo, and @MS-Santi for their community contributions (including code, samples, and documentation)!

New Features

  • JavaScript orchestrator functions (preview): You can now write orchestrator functions in JavaScript. More details in our new durable-js repo: https://github.com/Azure/azure-functions-durable-js. JavaScript samples can be found here.
  • Extended sessions: Allows orchestrator functions to stay in memory longer, removing the need to frequently replay them. This is a feature which can boost performance in various scenarios. More details in our Orchestrator function replay documentation documentation.
  • Concurrency control: Configure the maximum number of orchestrator or activity functions that can run concurrently on a single host instance. More details in the Concurrency throttles documentation.
  • Configurable partition counts: Configure the number of partitions in your task hub. Previously this was fixed at 4. Now this can be between 1 and 16. More details in the Orchestrator scale-out documentation.

Fixed

  • Custom status values were not displayed in the HTTP status query API (#271)
  • Waiting for multiple event types inside a loop only responded to first event type received (#275)
  • Task hub listeners could fail to start repeatedly and endlessly (#295)
  • Scale controller didn't auto-scale out based on queue latency (#200)
  • Scaling in consumption plan didn't work when using different storage account (#179)
  • Added protection against "split brain" in Azure Storage provider (#38)

Misc

  • Change custom Event Grid eventType property from an int to a string (#294) [Breaking change!]
  • Made GetInputAsJson and GetRawInput internal [Breaking change!]
  • Replaced nullable arguments and optional parameters with proper overloads [Breaking change!]
  • Removed "version" information from traces (#282)
  • Updated all samples (C# precompiled, C# scripting, F#, and WebJobs SDK)
  • Updated .NET API documentation

Installation

Please see our installation documentation for instructions for all supported platforms.

May 4, 2018
Pre-release

@cgillum cgillum released this Apr 16, 2018 · 128 commits to master since this release

Assets 2

Overview

This is the RC (Release Candidate) release of the Durable Functions extension to Azure Functions, and contains a significant number of updates. As with most pre-releases, it contains several important fixes, new features, and breaking changes. Thanks to everyone who contributed, either via code, asking good questions, or by filing issues!

Special thanks goes to @gled4er, @TsuyoshiUshio, @yuka1984, @kingkino, @rukasakurai, and @hamamotsu for their community contributions!

New Features

  • Custom orchestration status values (#193): A new SetCustomStatus API has been introduced which allows an orchestrator function to expose "progress" state to external callers. The state can be a simple string or a complex JSON object.
  • Automatic Azure Event Grid publishing (#220): This feature allows publishing orchestration lifecycle events (created, completed, failed, etc.) to a custom Azure Event Grid topic.
  • Large message support (#26): Function parameters, function return values, and external events can now be of any size (previously there was a 60 KB size limitation).
  • Inner exceptions (#250): Orchestrator functions can now see the inner exception details when an exception is thrown from an activity or sub-orchestrator function.

Fixed

  • Fix for HTTP 500 errors from HTTP APIs on cold start (#205)
  • Improved error message for invalid async orchestrator function logic (#208)
  • Support for waiting on multiple concurrent external events in parallel with the same name (#207)
  • Precompiled samples won't compile for netstandard2.0 (#233)
  • Orchestration loops forever when terminated (#146)
  • Unintuitive error message when UseDevelopmentStorage=true and Storage Emulator isn't running (#238)
  • Orchestration Replay calls activities twice (#231)
  • ContinueAsNew doesn't continue (#182)
  • Function App doesn't scale when custom HubName is specified (#111)
  • StartNewAsync fails with timeout exceptions (#177)
  • Tracking events in Application Insights cannot be ordered reliably (#71)
  • Status query HTTP API should return a failure status code when the orchestrator fails (#154) [Breaking Change]

Misc

  • Replace "Counter" sample with "Monitor" sample (#118)
  • Expose Task Hub name in DurableOrchestrationClient (#224)
  • Create separate "Instances" table for storing orchestration status (#183) [Breaking Change]
  • Expose parent orchestration instance id in a sub-orchestrator (DurableOrchestrationContext.ParentInstanceId) (#257)

Installation

Anyone can start using the v1.3.1-rc release by referencing the latest Microsoft.Azure.WebJobs.Extensions.DurableTask NuGet package in their projects.

Visual Studio 2017 (Functions 1.0 or Functions 2.0 preview)

Update your .csproj file with the following package reference:

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="1.3.1-rc" />

Azure Functions CLI (Functions 2.0 preview)

Use the following command to install the updated extension into your project:

func extensions install -p Microsoft.Azure.WebJobs.Extensions.DurableTask -v 1.3.1-rc

Azure Portal

The Azure Portal Durable Functions templates for projects targeting Functions 2.0 (preview) will automatically install this version of the extension at a later date.

Breaking Changes

There are runtime breaking changes in this release, so it is highly recommended to use this new package version with new function apps.

If you have an existing function app that you would like to migrate to this version, please see the versioning and migration strategies documentations for best practices.

Quick Documentation

Documentation has not been published yet for these new features, but expect the docs to trickle in over the next few weeks. In the meantime, here are some quick notes for how to get started with some of the new features.

Custom orchestration status values

This is a feature which allows users to set a custom status value from their orchestrator function, and have it exposed via the HTTP GetStatus API or the DurableOrchestrationClient.GetStatusAsync API.

Here is a simple example of how it works, starting with the orchestrator function code:

public static async Task SetStatusTest([OrchestrationTrigger] DurableOrchestrationContext ctx)
{
    // ...do work...

    // update the status of the orchestration with some arbitrary data
    var customStatus = new { nextActions = new [] {"A", "B", "C"}, foo = 2, };
    ctx.SetCustomStatus(customStatus);

    // ...do more work...
}

While the orchestration is running, external clients can fetch this custom status. For example:

GET /admin/extensions/DurableTaskExtension/instances/instance123

...yields the following JSON status payload...

{
  "runtimeStatus": "Running",
  "input": null,
  "customStatus": { "nextActions": ["A", "B", "C"], "foo": 2 },
  "output": null,
  "createdTime": "2017-10-06T18:30:24Z",
  "lastUpdatedTime": "2017-10-06T19:40:30Z"
}

The custom status payload is limited to 16 KB of UTF-16 JSON text. If larger custom status payloads are required, the recommendation is to store them outside the orchestration state in a store like Azure Storage or Cosmos DB.

Azure Event Grid Publishing

Azure Event Grid orchestration lifecycle publishing can be enabled by specifying the eventGridTopEndpoint and eventGridKeyName fields in the durableTask section of host.json:

{
  "durableTask": {
    "eventGridTopicEndpoint": "YOUR_EVENT_GRID_TOPIC_ENDPOINT_SAS_URL",
    "eventGridKeySettingName": "YOUR_EVENT_GRID_TOPIC_KEY_APP_SETTING"
  }
}

Note that the eventGridKeySettingName field should be the name of an Application Setting or environment variable which contains the actual secret key value. You should never put your Event Grid key directly in host.json.

The eventType for these events is orchestratorEvent and the subject is durable/orchestrator/{orchestrationRuntimeStatus}, where {orchestrationRuntimeStatus} can be Running, Completed, Failed, or Terminated.