Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft Azure WebJobs SDK 'Storage' connection string is missing or empty #89

Closed
marcelvandendungen opened this issue Mar 13, 2017 · 52 comments

Comments

@marcelvandendungen
Copy link

commented Mar 13, 2017

I'm trying out the latest Azure-functions-cli, but cannot seem to get past this issue. When I run 'func function run , the web service window is launched and an error appears:

A ScriptHost error has occurred
Microsoft.Azure.WebJobs.Host: Microsoft Azure WebJobs SDK 'Storage' connection string is missing or empty. The Microsoft Azure Storage account connection string
can be set in the following ways:

  1. Set the connection string named 'AzureWebJobsStorage' in the connectionStrings section of the .config file in the following format
    , or
  2. Set the environment variable named 'AzureWebJobsStorage', or
  3. Set corresponding property of JobHostConfiguration.

I tried both 1 and 2, but neither seems to have any effect.

Azure Functions Cli (1.0.0-beta.93)
Function Runtime Version: 1.0.10774.0

Thanks,
-- Marcel

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

You need to set that value in the appsettings.json file. or through the command func settings add AzureWebJobsStorage <connectionString>

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

@lindydonna
we currently favor appsettings.json over set environment variables. Maybe it should be the other way around?

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

@marcelvandendungen Unfortunately, the error message string comes from the WebJobs SDK (a dependency of Azure Functions), so the message is not helpful. We have a tracking bug to improve this: #38

@ahmelsayed That's a good question--I don't know which direction is the right one. My gut feeling is that our current approach is the right one (appsettings.json is more specific than environment variables), but we should get more feedback.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Mar 14, 2017

@mabdelmonemali Thanks for reporting the error, we're going to also make it so that environment variables take precedence over appsettings.json. See #91

@lindydonna lindydonna closed this Mar 14, 2017

@marcelvandendungen

This comment has been minimized.

Copy link
Author

commented Mar 15, 2017

Thanks, I've got it working. Yes, environment variables before appsettings.json makes sense.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2017

@marcelvandendungen Thanks for confirming; there was some discussion in the team about the right order of convention. :)

@davidsk

This comment has been minimized.

Copy link

commented Apr 1, 2017

There is a requirement for an AzureWebJobsStorage property in the appsettings.json purely for compatibility with the WebJobs SDK, it's not required by the azure function cli, correct?

@mkamonster

This comment has been minimized.

Copy link

commented Apr 1, 2017

@davidsk actually the AzureWebJobsStorage is required when you want to run a Timer triggered Azure Function. You don't need it for the http-triggered Azure Functions indeed.

@davidsk

This comment has been minimized.

Copy link

commented Apr 1, 2017

And this should be set to the actual value even when developing locally?

@mkamonster

This comment has been minimized.

Copy link

commented Apr 2, 2017

I'm using the Azure Storage Emulator during development.

"AzureWebJobsStorage": "UseDevelopmentStorage=true"

Works like a charm.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2017

@davidsk You can use the Azure storage emulator, but it may not work exactly like Azure Storage. As @mkamonster explained, the setting is required for timer triggers, and in fact all triggers aside from HTTP.

@davidsk

This comment has been minimized.

Copy link

commented Apr 3, 2017

@lindydonna I've got the storage emulator installed now. It's not particularly clear in the docs or blog posts that storage is necessary for timer triggers and easy to confuse the requirement with storage based triggers

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2017

@davidsk Thanks for the feedback, we're going to improve the documentation and CLI error messages around this. See Azure/Azure-Functions#108 and #38

@mkamonster

This comment has been minimized.

Copy link

commented Apr 4, 2017

but it may not work exactly like Azure Storage

@lindydonna Can you eloborate a bit on this statement? Should we prevent testing against the Azure Storage Emulator because of anomalies?

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2017

@mkamonster It's hard to enumerate all the differences, but in general, you should not assume that if something worked correctly with the Storage Emulator, that it will also work correctly in Azure. If you want to use the emulator, then you should do another round of testing with a real storage account before deploying to Azure.

@securityvoid

This comment has been minimized.

Copy link

commented May 3, 2017

FYI, this is still pretty confusing. I set the following in my function.json:

      "connection":"AZURE_STORAGE_CONNECTION_STRING"

At least for local development this made it so I had to specify both variables in appsettings.json: AZURE_STORAGE_CONNECTION_STRING
AzureWebJobsStorage

I'll be able to test later if when it goes to the actual app if its smart enough to only use one or the other, or if I really have to specify both. If so, then mine as well stick with the default variable.

It is also worth noting that the default appsettings.json has "IsEncrypted": true set, which made it completely ignore my unencypted settings when I added them manually, which kept this error from going away.

Once I changed it over to "IsEncrypted" : false, it took my plaintext values and this error went away.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented May 3, 2017

@securityvoid You can use the value "connection": "AzureWebJobsStorage" if you want the default. AzureWebJobsStorage is required as the runtime creates queues for internal use.

@jfoshee

This comment has been minimized.

Copy link

commented May 9, 2017

I've tried everything I can think of w/ web.config, appsettings.json and local.settings.json and I still cannot resolve the error connection string is missing or empty...
I download the func tool yesterday: 1.0.0-beta.96.

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented May 9, 2017

@jfoshee can you check whether the IsEncrypted flag represent the actual state of the file?

@jfoshee

This comment has been minimized.

Copy link

commented May 10, 2017

Yes, thanks @ahmelsayed and @securityvoid, the IsEncrypted flag was not present and defaulting to true. But there were multiple layers of confusion for me. In the hopes that my feedback can be useful, here are all the ways I was stymied:

  1. The error message specifically first recommends updating web.config XML in the traditional way:
    1. Set the connection string named 'AzureWebJobsAzureWebJobsDashboard' in the connectionStrings section of the .config file in the following format <add name="AzureWebJobsAzureWebJobsDashboard" connectionString="DefaultEndpointsProtocol=http|https;AccountName=NAME;AccountKey=KEY" />
    As am working in a web project, as directed by this guide, I already have a web.config file, so I added my connection string there.... I was never able to get this to work.

  2. The second thing that's confusing is obviously the weird AzureWebJobsAzureWebJobsStorage connection string name... My function.json includes "connection": "AzureWebJobsDashboard", which is what I grabbed from an existing .csx AzureFunction. So it's pretty confusing whether I need to prepend my connection string name, or remove AzureWebJobs from my connection: value. /shrug

  3. When I tried to use appsettings.json I overlooked the error at the top of the output that says it should be named local.settings.json. I saw the discussion about renaming that file in another issue. Clearly it's frustrating if these patterns keep changing. Especially given that I have live functioning Core web apps that are still using appsettings.json. So is this change peculiar to func? Or will it be changing for standard Core web apps too?

  4. When I tried to add my connection string to local.settings.json I overlooked the tell-tale error at the top of the output:
    The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
    Clearly it would be better if the error specifically pointed out I was providing an unencrypted connection string while IsEncrypted was set to true. And that I could encrypt the string from the command line or explicitly set the flag to false. I lost the most time to this issue. I recommend scanning the provided connection string for the tokens DefaultEndpointsProtocol, AccountName, AccountKey, ... If one of those tokens is there, the string is not encrypted.
    I also recommend IsEncrypted default to false. I presume the goal is to help people not commit secrets to source control. But if this encryption is using the machine key, will the encrypted string work on other developers' machines? More importantly though, new users want to first get this working, then think about key control. Moreover web.config and appsettings.json default to unencrypted connection strings.

A minor confusion which persists with the .json files is whether connection strings should go under values: or connectionStrings:... /shrug

The feeling I'm left with is that the func team is departing from established patterns used in web projects. But I'm not on the bleeding edge of that stuff, so I don't know for sure. My ultimate recommendation would be to stay as consistent as possible with typical web projects so that developers can bring over that experience. I can add a connection string to a web.config in my sleep, and that experience did not help me here.

I'm a fan of Azure Functions, which is the reason I persisted this far. So I am very glad you are continuing to improve the tooling. Thanks for listening. Now, back to watching MS Build. ;-)

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented May 17, 2017

@jfoshee Thanks for the detailed feedback. I agree that the error messages are quite confusing!

To answer your questions:

  1. We're going to ship a fix soon to provide a detailed warning when connection strings are missing. And, we won't even launch the functions host if that connection string is missing (we can't control that error message, because it comes from the underlying functions runtime, which uses WebJobs). See When connection strings are missing, provide a tip on how to fetch app settings #38

  2. This error should go away once we fix issue 1.

  3. This file is specific to func.exe, and we found that the previous name caused a lot of confusion. The old filename will still work, it's just deprecated. Core appsettings.json are a different format, which is another reason we changed our name.

  4. We have two fixes for this: Provide a better error message when failing to decrypt a setting #95 and Change default of func init to IsDecrypted=false #140

You're right that we have a different pattern than web projects, but we're working on making the error messages and help more clear so that customers can learn how we do things.

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@lindydonna I'm encountering the AzureWebJobsAzureWebJobsStorage missing error message, running with host CLI 1.0.10726.0. I'm trying to run two function projects locally, so I'm using func host start -p 7072 for the second one and that's where it's failing. Is that the wrong way to do this?

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

@eryon can you verify the IsEncrypted flag in local.settings.json is correctly reflecting the state of the file?

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@ahmelsayed sure, it is false, and my values are not encrypted

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

What about the path you're launching func host start -p 7072 from. Is that the root of your project where local.settings.json is? There was a bug that when launching the command from a sub directory in your function app it won't load the settings. it should be fixed in 1223f59 which will go out with the next update

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

I was launching it from the bin\Debug\net461\ directory, as I'm using the VS2017 pre-compiled stuff and that's where all the generated stuff ends up (from the project root, it couldn't detect any functions as function.json hasn't been generated until build).

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

can you set your local.settings.json to be copied to the output? it just needs to have that file in the CurrentDirectory of the process when launching func host start

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

It's there, and is where I checked first to ensure the normal storage key was in place :)

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

hmmmm, then I'm puzzled. I don't really know what could be going wrong. If you do func settings list (optional --showValue) does it correctly list the settings you have?

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@ahmelsayed it does not! See below:

D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> func settings list
App Settings:
Connection Strings:
D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> ls


    Directory: D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         6/2/2017     10:44                Spawner
-a----         6/2/2017     11:06              4 host.json
-a----         6/2/2017     08:51            778 local.settings.json

And just to prove the settings file really has content:

D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> cat .\local.settings.json
{
  "IsEncrypted": false,
  "Values": {
    "ApplicationId": "<privacy>",
    "ApplicationSecret": "<privacy>",
    "AuthenticationAuthority": "https://login.windows.net/<privacy>",
    "AuthenticationTokenEndpoint": "https://management.azure.com/",
    "AzureSubscriptionId": "<privacy>",
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=<privacy>"
  }
}
@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

Maybe it's a file encoding issue? Can you open it in VS code and ensure it has the right encoding and line endings?

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@lindydonna just the local.settings.json? Sure, it's UTF-8, CRLF all around.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

@eryon Try this: rename your current file to local.settings.json.bak and add a couple of settings manually through the CLI (with encryption off). Then diff the two files.

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@lindydonna when I do that, it generates an appsettings.json instead. Encoding and line endings are also UTF-8 and CRLF in the new file.

Interestingly, when I then run func host start I get a different error:

A ScriptHost error has occurred
Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.Spawner'. Microsoft.Azure.WebJobs.Host: Cannot bind parameter 'log' to type
TraceWriter. Make sure the parameter Type is supported by the binding. If you're using binding extensions (e.g. ServiceBus, Timers, etc.) m
ake sure you've called the registration method for the extension(s) in your startup code (e.g. config.UseServiceBus(), config.UseTimers(),
etc.).

My function has the following signature:

[FunctionName("Spawner")]
public static async Task Run([QueueTrigger("to-render", Connection = "AzureWebJobsStorage")]string item, TraceWriter log)

And for what it's worth, this does run from VS2017 Preview 3 without a hitch. It's only having these problems when I try to use the CLI directly.

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

huh, what version are you using? the current version is 1.0.0-beta.97, you can get that just by running func

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@ahmelsayed VS2017 installed beta97, so I assumed that was it, but when I run it in powershell, it's reporting beta91! Is there an easy way to upgrade the CLI used by powershell?

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@ahmelsayed @lindydonna thanks for the support. I uninstalled azure-functions-cli and installed azure-functions-core-tools using npm -g and now I'm running the beta97 CLI, and it's working fine!

I also found this article indicating that I can specify the host port in the settings file. I tried that, in order to run it from VS2017, but it's not having any effect... still launches on 7071 when my local.settings.json has

  "Host": {
    "LocalHttpPort": 7072
  }

Maybe a separate issue?

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

yeah, unfortunately VS passes --port 7071 which takes precedence over the file.

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@ahmelsayed Is there a github repo I can open a feature request for to stop that? :)

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

@eryon yes, this repo :)

I think VS should not pass that port number and let the cli or the user decide. But feel free to open an issue on this repo and will discuss it.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

@ahmelsayed Good point about VS, I'll open a bug to track that feature.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

@eryon You can change the port that VS uses by modifying the launch options. Change it to "host start -p 7072"

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Jun 2, 2017

I opened this bug for the change to VS: Azure/Azure-Functions#325

@eryon

This comment has been minimized.

Copy link

commented Jun 2, 2017

@lindydonna thanks so much! By launch options, I took you to mean Application arguments in the Project properties > Debug section, and that works just fine :)

@ssfcultra

This comment has been minimized.

Copy link

commented Aug 9, 2017

I am running into this same issue when trying to write to TableStorage with the following binding:

[Table("Chapter01", Connection = "AzureWebJobsStorage")] IAsyncCollector outputTable

Any thoughts? I definitely have the value AzureWebJobsStorage in my appsettings.json file.

@lindydonna

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2017

@ssfcultra Please open a new issue as we generally don't monitor closed ones.

Can you paste the contents of appsettings.json, eliding any connection strings? Also, note that you should rename your settings file to local.settings.json, as the older name is deprecated.

@ssfcultra

This comment has been minimized.

Copy link

commented Aug 10, 2017

Thanks @lindydonna - I have added a new issue here: 204. I have also added an issue for binding to Blob Storage using the Web Jobs bindings syntax. That issue is here: 201.

I am including both here since I imagine that once I solve one they both will make sense.

Thanks!

@ConnorMcMahon

This comment has been minimized.

Copy link

commented Jul 5, 2018

@ahmelsayed, it looks like this issue isn't fixed with dockerized Azure Functions.

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2018

@ConnorMcMahon are you referring to the error message?

@ConnorMcMahon

This comment has been minimized.

Copy link

commented Jul 5, 2018

Yes. I think that this issue would have been a better place for me to link that post.

@ahmelsayed

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2018

This has nothing to do with the cli (core tools) the cli doesn't run in the docker container, just the runtime. The runtime outputs this bogus error message. The issue should be on the runtime

@pilarodriguez

This comment has been minimized.

Copy link

commented Jul 5, 2019

I have a WebJob with some scheduled jobs (using TimerTrigger). I've specified the AzureWebJobsStorage connection string in the connectionStrings.config file
<add name="AzureWebJobsStorage" connectionString="UseDevelopmentStorage=true" />
But I get this error

The listener for function 'ScheduleTask.CronJob' was unable to start.
Value cannot be null.
Parameter name: connectionString

If I add it as an appSetting, it works fine
<add key="AzureWebJobsStorage" value="UseDevelopmentStorage=true" />

In some other places in my code I use the AzureWebJobsStorage as a connection string and I don't want to add an extra app setting. Is there a way to make this work specifying the value as a connection string and not as app setting?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.