Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

csheets · 2018-10-14T22:33:37Z

I was successfully running Durable Functions for a few weeks but now all of a sudden I am not able to run them and am getting the following output. Not sure what has changed -- any pointers on what to look at would be most appreciated. I have also included a related stacktrace below.

Azure Functions Core Tools (2.1.725 Commit hash: 68f448fe6a60e1cade88c2004bf6491
af7e5f1df)
Function Runtime Version: 2.0.12134.0
[10/14/2018 10:17:59 PM] Building host: startup suppressed:False, configuration
suppressed: False
[10/14/2018 10:17:59 PM] Reading host configuration file 'C:\Users\csheets\sourc
e\repos\FunctionApp4\FunctionApp1\bin\Debug\netstandard2.0\host.json'
[10/14/2018 10:17:59 PM] Host configuration file read:
[10/14/2018 10:17:59 PM] {
[10/14/2018 10:17:59 PM]   "version": "2.0"
[10/14/2018 10:17:59 PM] }
[10/14/2018 10:18:38 PM] A host error has occurred
[10/14/2018 10:18:38 PM] Microsoft.WindowsAzure.Storage: Server encountered an i
nternal error. Please try again after some time.
Listening on http://0.0.0.0:7071/
Hit CTRL-C to exit...

The following is a stacktrace that is being thrown:

System.AggregateException
  HResult=0x80131500
  Message=One or more errors occurred.
  Source=System.Private.CoreLib
  StackTrace:
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.<Dispose>b__8_0()
   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext(Action actionToRun)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteToBlobAsync>d__22.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 167
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteAsync>d__18.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 128
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<PersistSecretsAsync>d__26`1.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 412
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<GetHostSecretsAsync>d__11.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.<GetOrCreateExtensionKey>d__6.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 72
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetExtensionWebHookRoute(String extensionName) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 64
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetUrl(IExtensionConfigProvider extension) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 49
   at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Config\ExtensionConfigContext.cs:line 55
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.Microsoft.Azure.WebJobs.Host.Config.IExtensionConfigProvider.Initialize(ExtensionConfigContext context)
   at Microsoft.Azure.WebJobs.Host.DefaultExtensionRegistryFactory.Create() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\DefaultExtensionRegistryFactory.cs:line 38
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_0(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 50
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_4(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 83
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c.<AddScriptHostCore>b__2_5(IServiceProvider p) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\ScriptHostBuilderExtensions.cs:line 115
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Azure.Functions.Cli.Actions.HostActions.StartHostAction.<RunAsync>d__44.MoveNext()

Inner Exception 1:
StorageException: Server encountered an internal error. Please try again after some time.

The text was updated successfully, but these errors were encountered:

csheets · 2018-10-15T00:12:46Z

I switched from using local storage emulator to using Azure StorageAccount and the function app run ok -- so it is something to do with the Azure Storage emulator but I have no idea what. I have attempted to uninstall and re-install the storage emulator but the same error occurs after the uninstall/re-install process.

cgillum · 2018-10-16T01:12:03Z

Which version of the storage emulator are you using? This actually doesn't appear to be a Durable Functions issue, but rather an issue with the Azure Functions host itself, which uses storage for things like key management (key management frames appear in your callstack).

csheets · 2018-10-16T13:46:21Z

I am using version: Windows Azure Storage Emulator 5.8.0.0 command line tool

I am able to run "regular" Azure Functions locally without any issues. Seems to be just Durable Functions that are having the problem.

cgillum · 2018-10-16T18:00:29Z

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

csheets · 2018-10-16T20:06:01Z

ok - thanks

prior to your reply i ended up uninstalling and re-installing VS 2017 and that resolved the issue -- looks like it re-installed Storage Emulator 5.6 in the process so just going to stay there for now

cgillum · 2018-10-16T21:23:58Z

Good to know there is a way to go back (I might consider doing that as well).

I've also opened a bug in the Azure-Functions-Host repo to track this since it will likely affect more than just Durable Functions: #3621

fabiocav · 2018-10-16T21:43:59Z

Indeed, this is not specific to Durable Functions, but on a host code path it relies on.

This looks like an issue with the more recent version of the Emulator, we'll investigate.

tsubakimoto · 2018-11-18T06:09:18Z

I also encountered the same error.
Also, the problem is temporarily resolved by the method @cgillum presented. Thanks.
https://github.com/Azure/azure-functions-durable-extension/issues/474#issuecomment-430337085

I expect that the error will be resolved.

fabiocav · 2018-11-19T17:12:35Z

Moving this to host repo for triage and assignment

apsteger · 2019-01-07T15:42:01Z

I experienced this issue after installing the Visual Studio 2019 Preview. Installing Storage Emulator 5.9 fixed it for me.

iwaldman · 2019-01-23T01:45:32Z

I experienced the same issue. Thank you @cgillum for the workaround.

opyke · 2019-01-23T13:18:23Z

I am also experiencing this issue. Using the workaround.

MaximRouiller · 2019-03-18T18:04:14Z

UPDATE: said blog post

Alright. I'll be writing a blog post about this issue to make it more visible but basically, updating Storage Emulator to 5.9 solves your issue.

I was running Azure Durable Functions on .NET and NodeJS with the workaround for my samples:

I was running the v5.8 of Azure Storage Emulator. No idea what changed since the docs isn't updated yet. I've opened an issue to ensure that we get some notes.

So... updating to v5.9 fixed the issue in .NET as well as with NodeJS using the Windows Storage Emulator.

My Recommendation

Leave the workaround in place. People may have the 5.8 version which could lead to issues. There's no clear message that the parameter is missing which makes it hard to debug. Leaving the parameter in place doesn't seem to be causing any issues with 5.9 either.

Follow-Up

@fabiocav @cgillum Any point in making the message clearer for the users? Basically, 5.7 works, 5.9 works, 5.8 🔥.

Obviously not an issue from Azure Functions point of view but it does look more like a breaking change in the Emulator's API.

Thoughts?

chrislangston · 2019-06-29T22:05:43Z

Same thing is happening even if you have VS 2019 Installed and running Azure Storage Emulator 5.9. I had to add this entry to the local.settings.json to get the Azure Function to work.

MaximRouiller · 2019-07-08T18:04:46Z

@chrislangston Which version of Azure Functions are you using?

If you run func on your CLI, can you copy the lines below the magical lightning? Mine looks like this:

Azure Functions Core Tools (2.7.1149 Commit hash: 476884e4a09c54686e460fd0c050d15139bf073e)
Function Runtime Version: 2.0.12438.0

yojagad · 2019-07-15T23:45:14Z

Adding this to triaged to verify whether there are issues when running with Storage Emulator 5.9.

r0bturner · 2020-01-03T10:27:39Z

I'm still unable to run durable functions with Storage Emulator 5.10 and AzureWebJobsSecretStorageType set to "files" in the local.settings.json file.

If I remove the AzureWebJobsSecretStorageType setting (so it defaults to blobs) I get this exception in the Storage Account logs:

03/01/2020 09:53:22 [Error_debug] [ActivityId=919f1fa3-9cd5-441d-a07a-411fec2f6828] Exception raised during processing of request.
 Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.FatalServerCrashingException: 
The fatal unexpected exception 'Exception has been thrown by the target of an invocation.' encountered during processing of request. 
---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. 
---> System.InvalidOperationException: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.
   at System.Security.Cryptography.MD5CryptoServiceProvider..ctor()

This means I can't upload blobs to the storage emulator. I'm unable to disable FIPS compliant algorithms for encryption because i'm using a work machine. Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

HQJaTu · 2020-02-07T12:17:40Z

Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

Actually, I failed to successfully deploy my Durable Functions into Azure because of this same thing.
Manual removal of the blob's contents (the keys) solved the issue.

vinayakmoh · 2020-03-16T11:11:32Z

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".
{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}
/cc @fabiocav

This solved issue for me. I had upgraded to VS 2019 and Emulator 5.1

HQJaTu · 2020-03-16T12:03:26Z

Yes, that will solve the problem for local dev. Deleting the Blobs in question will solve it when the same thing happens in Azure.

csheets changed the title ~~Unable to Run Durable Functions~~ Unable to Run Durable Functions due to storage error Oct 14, 2018

cgillum changed the title ~~Unable to Run Durable Functions due to storage error~~ Unable to run Durable Functions with Azure Storage Emulator v5.8 Oct 16, 2018

fabiocav transferred this issue from Azure/azure-functions-durable-extension Nov 19, 2018

olivoarroba mentioned this issue Apr 4, 2019

Tutorial does not work with Azure Storage Emulator 5.8.0.0 MicrosoftDocs/azure-docs#28798

Closed

yojagad added this to the Triaged milestone Jul 15, 2019

MrTortoise mentioned this issue Aug 29, 2019

Microsoft.WindowsAzure.Storage: Server encountered an internal error. Is Back Azure/azure-functions-core-tools#1529

Open

Azure / azure-functions-host

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

csheets commented Oct 14, 2018

csheets commented Oct 15, 2018

cgillum commented Oct 16, 2018

csheets commented Oct 16, 2018

cgillum commented Oct 16, 2018

csheets commented Oct 16, 2018

cgillum commented Oct 16, 2018

fabiocav commented Oct 16, 2018

tsubakimoto commented Nov 18, 2018

fabiocav commented Nov 19, 2018

apsteger commented Jan 7, 2019

iwaldman commented Jan 23, 2019

opyke commented Jan 23, 2019

MaximRouiller commented Mar 18, 2019 •

edited

chrislangston commented Jun 29, 2019

MaximRouiller commented Jul 8, 2019

yojagad commented Jul 15, 2019

r0bturner commented Jan 3, 2020

HQJaTu commented Feb 7, 2020

vinayakmoh commented Mar 16, 2020

Workaround

HQJaTu commented Mar 16, 2020

Azure / azure-functions-host

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

Comments

csheets commented Oct 14, 2018

csheets commented Oct 15, 2018

cgillum commented Oct 16, 2018

csheets commented Oct 16, 2018

cgillum commented Oct 16, 2018

Workaround

csheets commented Oct 16, 2018

cgillum commented Oct 16, 2018

fabiocav commented Oct 16, 2018

tsubakimoto commented Nov 18, 2018

fabiocav commented Nov 19, 2018

apsteger commented Jan 7, 2019

iwaldman commented Jan 23, 2019

opyke commented Jan 23, 2019

MaximRouiller commented Mar 18, 2019 • edited

My Recommendation

Follow-Up

chrislangston commented Jun 29, 2019

MaximRouiller commented Jul 8, 2019

yojagad commented Jul 15, 2019

r0bturner commented Jan 3, 2020

HQJaTu commented Feb 7, 2020

vinayakmoh commented Mar 16, 2020

Workaround

HQJaTu commented Mar 16, 2020

MaximRouiller commented Mar 18, 2019 •

edited