Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

Open
csheets opened this issue Oct 14, 2018 · 20 comments
Open

Unable to run Durable Functions with Azure Storage Emulator v5.8 #3795

csheets opened this issue Oct 14, 2018 · 20 comments
Milestone

Comments

@csheets
Copy link

@csheets csheets commented Oct 14, 2018

I was successfully running Durable Functions for a few weeks but now all of a sudden I am not able to run them and am getting the following output. Not sure what has changed -- any pointers on what to look at would be most appreciated. I have also included a related stacktrace below.

Azure Functions Core Tools (2.1.725 Commit hash: 68f448fe6a60e1cade88c2004bf6491
af7e5f1df)
Function Runtime Version: 2.0.12134.0
[10/14/2018 10:17:59 PM] Building host: startup suppressed:False, configuration
suppressed: False
[10/14/2018 10:17:59 PM] Reading host configuration file 'C:\Users\csheets\sourc
e\repos\FunctionApp4\FunctionApp1\bin\Debug\netstandard2.0\host.json'
[10/14/2018 10:17:59 PM] Host configuration file read:
[10/14/2018 10:17:59 PM] {
[10/14/2018 10:17:59 PM]   "version": "2.0"
[10/14/2018 10:17:59 PM] }
[10/14/2018 10:18:38 PM] A host error has occurred
[10/14/2018 10:18:38 PM] Microsoft.WindowsAzure.Storage: Server encountered an i
nternal error. Please try again after some time.
Listening on http://0.0.0.0:7071/
Hit CTRL-C to exit...

The following is a stacktrace that is being thrown:

System.AggregateException
  HResult=0x80131500
  Message=One or more errors occurred.
  Source=System.Private.CoreLib
  StackTrace:
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.<Dispose>b__8_0()
   at Microsoft.WindowsAzure.Storage.Core.Util.CommonUtility.RunWithoutSynchronizationContext(Action actionToRun)
   at Microsoft.WindowsAzure.Storage.Blob.BlobWriteStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteToBlobAsync>d__22.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 167
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.BlobStorageSecretsRepository.<WriteAsync>d__18.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\BlobStorageSecretsRepository.cs:line 128
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<PersistSecretsAsync>d__26`1.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 412
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.SecretManager.<GetHostSecretsAsync>d__11.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\Security\KeyManagement\SecretManager.cs:line 82
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.<GetOrCreateExtensionKey>d__6.MoveNext() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 72
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetExtensionWebHookRoute(String extensionName) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 64
   at Microsoft.Azure.WebJobs.Script.WebHost.DefaultScriptWebHookProvider.GetUrl(IExtensionConfigProvider extension) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\WebHooks\DefaultScriptWebHookProvider.cs:line 49
   at Microsoft.Azure.WebJobs.Host.Config.ExtensionConfigContext.GetWebhookHandler() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Config\ExtensionConfigContext.cs:line 55
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension.Microsoft.Azure.WebJobs.Host.Config.IExtensionConfigProvider.Initialize(ExtensionConfigContext context)
   at Microsoft.Azure.WebJobs.Host.DefaultExtensionRegistryFactory.Create() in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\DefaultExtensionRegistryFactory.cs:line 38
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_0(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 50
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.WebJobsServiceCollectionExtensions.<>c.<AddWebJobs>b__1_4(IServiceProvider p) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsServiceCollectionExtensions.cs:line 83
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, Object serviceKey, IfUnresolved ifUnresolved, Type requiredServiceType, Request preResolveParent, Object[] args) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 290
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.ReflectionFactory.CreateExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7059
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6534
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c.<AddScriptHostCore>b__2_5(IServiceProvider p) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\ScriptHostBuilderExtensions.cs:line 115
   at DryIoc.Microsoft.DependencyInjection.DryIocAdapter.<>c__DisplayClass3_0.<RegisterDescriptor>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\DryIocAdapter.cs:line 156
   at DryIoc.Registrator.<>c__DisplayClass27_0.<RegisterDelegate>b__0(IResolverContext r) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 4530
   at DryIoc.Factory.<>c__DisplayClass26_1.<ApplyReuse>b__2() in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6584
   at DryIoc.Scope.TryGetOrAdd(ImMap`1 items, Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7825
   at DryIoc.Scope.GetOrAdd(Int32 id, CreateScopedValue createValue, Int32 disposalOrder) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7810
   at DryIoc.Factory.ApplyReuse(Expression serviceExpr, Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6583
   at DryIoc.Factory.GetExpressionOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6544
   at DryIoc.Factory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 6611
   at DryIoc.DelegateFactory.GetDelegateOrDefault(Request request) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 7719
   at DryIoc.Container.ResolveAndCacheDefaultFactoryDelegate(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 210
   at DryIoc.Container.DryIoc.IResolver.Resolve(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\DryIoc\Container.cs:line 195
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetService(Type serviceType, IfUnresolved ifUnresolved) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 96
   at Microsoft.Azure.WebJobs.Script.WebHost.DependencyInjection.JobHostServiceProvider.GetRequiredService(Type serviceType) in C:\azure-webjobs-sdk-script\src\WebJobs.Script.WebHost\DependencyInjection\JobHostServiceProvider.cs:line 79
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at Azure.Functions.Cli.Actions.HostActions.StartHostAction.<RunAsync>d__44.MoveNext()

Inner Exception 1:
StorageException: Server encountered an internal error. Please try again after some time.
@csheets csheets changed the title Unable to Run Durable Functions Unable to Run Durable Functions due to storage error Oct 14, 2018
@csheets
Copy link
Author

@csheets csheets commented Oct 15, 2018

I switched from using local storage emulator to using Azure StorageAccount and the function app run ok -- so it is something to do with the Azure Storage emulator but I have no idea what. I have attempted to uninstall and re-install the storage emulator but the same error occurs after the uninstall/re-install process.

@cgillum
Copy link
Contributor

@cgillum cgillum commented Oct 16, 2018

Which version of the storage emulator are you using? This actually doesn't appear to be a Durable Functions issue, but rather an issue with the Azure Functions host itself, which uses storage for things like key management (key management frames appear in your callstack).

@csheets
Copy link
Author

@csheets csheets commented Oct 16, 2018

I am using version: Windows Azure Storage Emulator 5.8.0.0 command line tool

I am able to run "regular" Azure Functions locally without any issues. Seems to be just Durable Functions that are having the problem.

@cgillum cgillum changed the title Unable to Run Durable Functions due to storage error Unable to run Durable Functions with Azure Storage Emulator v5.8 Oct 16, 2018
@cgillum
Copy link
Contributor

@cgillum cgillum commented Oct 16, 2018

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

@csheets
Copy link
Author

@csheets csheets commented Oct 16, 2018

ok - thanks

prior to your reply i ended up uninstalling and re-installing VS 2017 and that resolved the issue -- looks like it re-installed Storage Emulator 5.6 in the process so just going to stay there for now

@cgillum
Copy link
Contributor

@cgillum cgillum commented Oct 16, 2018

Good to know there is a way to go back (I might consider doing that as well).

I've also opened a bug in the Azure-Functions-Host repo to track this since it will likely affect more than just Durable Functions: #3621

@fabiocav
Copy link
Member

@fabiocav fabiocav commented Oct 16, 2018

Indeed, this is not specific to Durable Functions, but on a host code path it relies on.

This looks like an issue with the more recent version of the Emulator, we'll investigate.

@tsubakimoto
Copy link

@tsubakimoto tsubakimoto commented Nov 18, 2018

I also encountered the same error.
Also, the problem is temporarily resolved by the method @cgillum presented. Thanks.
https://github.com/Azure/azure-functions-durable-extension/issues/474#issuecomment-430337085

I expect that the error will be resolved.

@fabiocav fabiocav transferred this issue from Azure/azure-functions-durable-extension Nov 19, 2018
@fabiocav
Copy link
Member

@fabiocav fabiocav commented Nov 19, 2018

Moving this to host repo for triage and assignment

@apsteger
Copy link

@apsteger apsteger commented Jan 7, 2019

I experienced this issue after installing the Visual Studio 2019 Preview. Installing Storage Emulator 5.9 fixed it for me.

@iwaldman
Copy link

@iwaldman iwaldman commented Jan 23, 2019

I experienced the same issue. Thank you @cgillum for the workaround.

@opyke
Copy link

@opyke opyke commented Jan 23, 2019

I am also experiencing this issue. Using the workaround.

@MaximRouiller
Copy link
Contributor

@MaximRouiller MaximRouiller commented Mar 18, 2019

UPDATE: said blog post


Alright. I'll be writing a blog post about this issue to make it more visible but basically, updating Storage Emulator to 5.9 solves your issue.

I was running Azure Durable Functions on .NET and NodeJS with the workaround for my samples:

I was running the v5.8 of Azure Storage Emulator. No idea what changed since the docs isn't updated yet. I've opened an issue to ensure that we get some notes.

So... updating to v5.9 fixed the issue in .NET as well as with NodeJS using the Windows Storage Emulator.

My Recommendation

Leave the workaround in place. People may have the 5.8 version which could lead to issues. There's no clear message that the parameter is missing which makes it hard to debug. Leaving the parameter in place doesn't seem to be causing any issues with 5.9 either.

Follow-Up

@fabiocav @cgillum Any point in making the message clearer for the users? Basically, 5.7 works, 5.9 works, 5.8 🔥.

Obviously not an issue from Azure Functions point of view but it does look more like a breaking change in the Emulator's API.

Thoughts?

@chrislangston
Copy link

@chrislangston chrislangston commented Jun 29, 2019

Same thing is happening even if you have VS 2019 Installed and running Azure Storage Emulator 5.9. I had to add this entry to the local.settings.json to get the Azure Function to work.

@MaximRouiller
Copy link
Contributor

@MaximRouiller MaximRouiller commented Jul 8, 2019

@chrislangston Which version of Azure Functions are you using?

If you run func on your CLI, can you copy the lines below the magical lightning? Mine looks like this:

Azure Functions Core Tools (2.7.1149 Commit hash: 476884e4a09c54686e460fd0c050d15139bf073e)
Function Runtime Version: 2.0.12438.0
@yojagad yojagad added this to the Triaged milestone Jul 15, 2019
@yojagad
Copy link
Contributor

@yojagad yojagad commented Jul 15, 2019

Adding this to triaged to verify whether there are issues when running with Storage Emulator 5.9.

@r0bturner
Copy link

@r0bturner r0bturner commented Jan 3, 2020

I'm still unable to run durable functions with Storage Emulator 5.10 and AzureWebJobsSecretStorageType set to "files" in the local.settings.json file.

If I remove the AzureWebJobsSecretStorageType setting (so it defaults to blobs) I get this exception in the Storage Account logs:

03/01/2020 09:53:22 [Error_debug] [ActivityId=919f1fa3-9cd5-441d-a07a-411fec2f6828] Exception raised during processing of request.
 Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.FatalServerCrashingException: 
The fatal unexpected exception 'Exception has been thrown by the target of an invocation.' encountered during processing of request. 
---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. 
---> System.InvalidOperationException: This implementation is not part of the Windows Platform FIPS validated cryptographic algorithms.
   at System.Security.Cryptography.MD5CryptoServiceProvider..ctor()

This means I can't upload blobs to the storage emulator. I'm unable to disable FIPS compliant algorithms for encryption because i'm using a work machine. Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

@HQJaTu
Copy link

@HQJaTu HQJaTu commented Feb 7, 2020

Looks like Durable Functions rely on blobs in the azure-webjobs-hosts and azure-webjobs-secrets containers. It's just a shame the AzureWebJobsSecretStorageType setting set to "files" doesn't overcome this issue. Wonder if this would be possible to fix in the future?

Of course I can workaround this by using a real storage account instead of the emulator..

Actually, I failed to successfully deploy my Durable Functions into Azure because of this same thing.
Manual removal of the blob's contents (the keys) solved the issue.

@vinayakmoh
Copy link

@vinayakmoh vinayakmoh commented Mar 16, 2020

Thanks - I downloaded 5.8 and observed the same problem. I have no idea why the Durable extension seems to be triggering this issue because the code path seems completely unrelated. I'll follow up with the Azure Functions Host team. I think this might possibly be a problem with extensions that use web hooks.

Workaround

Here is a workaround I identified from the Functions Key Management documentation. In local.settings.json, add a new setting called AzureWebJobsSecretStorageType and set it to "files".

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "AzureWebJobsSecretStorageType": "files"
  }
}

/cc @fabiocav

This solved issue for me. I had upgraded to VS 2019 and Emulator 5.1

@HQJaTu
Copy link

@HQJaTu HQJaTu commented Mar 16, 2020

Yes, that will solve the problem for local dev. Deleting the Blobs in question will solve it when the same thing happens in Azure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.