New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Azure Storage Emulator #53

Closed
rustd opened this Issue Aug 8, 2014 · 87 comments

Comments

Projects
None yet
@rustd

rustd commented Aug 8, 2014

We should be able to run locally on the emulator. Today the emulator has some limitations as compared to Azure Storage service. Maybe we throw exceptions for the cases where we don't work.
The getting started experience is hard without this support

@rustd rustd added this to the Backlog milestone Aug 20, 2014

@marckassay

This comment has been minimized.

marckassay commented May 29, 2015

I seem to be experiencing this with Emulator 4.0.0.0. Has this issue been resolved yet?

@rustd

This comment has been minimized.

rustd commented Jun 10, 2015

Not yet. The SDK only works with Azure Storage

@niclaslindberg

This comment has been minimized.

niclaslindberg commented Jul 1, 2015

Are there any known workarounds for this?

@mathewc

This comment has been minimized.

Contributor

mathewc commented Jul 1, 2015

Unfortunately no. Local emulator is missing features that the SDK requires to function (e.g. blob logs).

@agrigg

This comment has been minimized.

agrigg commented Aug 21, 2015

+1

@MarkStokes

This comment has been minimized.

MarkStokes commented Aug 26, 2015

+1 We need this ASAP.

Forcing developers to pay for azure storage during their development is not cool when we used to be able to do it locally on our dev machines.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Sep 3, 2015

We hear you guys, and we'll try to take a look at this soon. While there are some features of storage the SDK leverages (e.g. Blob logs) that don't work in the emulator, we may be able to provide a downgraded experience when running locally (you can probably live w/o having BlobTrigger trigger immediately for new blobs, which requires Blob logs).

@jculverwell

This comment has been minimized.

jculverwell commented Sep 10, 2015

Even a downgraded experience would be extremely useful.

@gweinhold

This comment has been minimized.

gweinhold commented Sep 10, 2015

I've been able to work around this by including an #if DEBUG statement to manually pull from a queue instead of the lack of support for the JobHost class (and corresponding QueueTrigger) in development.

https://gist.github.com/gweinhold/58f4e928378ab6329a3f

@jculverwell

This comment has been minimized.

jculverwell commented Sep 12, 2015

Thanks gweinhold, this is a great workaround

@spewu

This comment has been minimized.

spewu commented Dec 4, 2015

@mathewc Any news on this?

@mathewc mathewc modified the milestones: 1.2.0, Backlog Dec 4, 2015

@mathewc

This comment has been minimized.

Contributor

mathewc commented Dec 4, 2015

I've just moved this from backlog to the v1.2.0 milestone for "soonish" investigation.

@dkallen

This comment has been minimized.

dkallen commented Dec 22, 2015

I would use this. But it's not essential as I can use Azure accounts in the meantime.

@bmrpatel

This comment has been minimized.

bmrpatel commented Jan 5, 2016

👍 Cool, it will be true dev experience if supported locally. Thanks

@tyeth

This comment has been minimized.

tyeth commented Jan 21, 2016

I can't use Azure to test due to multiple MS accounts that are f-d and still linked to my card, so the local emulator is my only hope. Shame it's lacking... Maybe time to consider Azure/C# alternatives, everyone knows and loves JS

@fwd079

This comment has been minimized.

fwd079 commented Feb 19, 2016

I tried this link:
https://github.com/Azure/azure-webjobs-sdk/wiki/Running-Locally

And my code is:

Environment.SetEnvironmentVariable("AzureWebJobsEnv", "Development");
            if (!VerifyConfiguration())
            {
                Console.ReadLine();
                return;
            }
            CreateDemoData();
            var config = new JobHostConfiguration();

            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }

            JobHost host = new JobHost(config);
            host.RunAndBlock();

But it failed with this :

Exception thrown: 'System.InvalidOperationException' in mscorlib.dll

Additional information: Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure.

If I stop at: if (config.IsDevelopment) the value is true but I still get the exception.

Please can you help? Thanks.
.

@NBelham

This comment has been minimized.

NBelham commented Feb 23, 2016

snag-20160222t180136
When developing my WebJob with a local development queue, mscorlib.dll is throwing an InvalidOperationException with message "Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure".

So forcing me to use an Azure hosted Queue when developing my Web Job seems a rather awkward step I must take and goes against the tenant of the Azure SDK's capabilities of emulating Azure Storage capabilities locally.

Please can you provide a time frame when I can develop my WebJob without a dependency on a live Azure Storage account?

@dkallen

This comment has been minimized.

dkallen commented Feb 23, 2016

For those who are developing locally and don't want to pay for an Azure account to develop, I just learned of the free Developer program that offers $25/month in free Azure services. https://www.visualstudio.com/en-us/products/visual-studio-dev-essentials-vs.aspx. That doesn't sound like much, but a Queue or Blob does not cost that much to operate with amounts of data and transaction volumes typical of development activities. So it may be a good option for many people.

@fwd079

This comment has been minimized.

fwd079 commented Feb 26, 2016

I have a paid account on azure :) but I'd rather test locally before
putting it up. Thanks for the info though.

On Tuesday, 23 February 2016, David Kreth Allen notifications@github.com
wrote:

For those who are developing locally and don't want to pay for an Azure
account to develop, I just learned of the free Developer program that
offers $25/month in free Azure services.
https://www.visualstudio.com/en-us/products/visual-studio-dev-essentials-vs.aspx.
That doesn't sound like much, but a Queue or Blob does not cost that much
to operate with amounts of data and transaction volumes typical of
development activities. So it may be a good option for many people.


Reply to this email directly or view it on GitHub
#53 (comment)
.

https://www.profiles.google.com/fwd079

@LeeCampbell

This comment has been minimized.

LeeCampbell commented Apr 6, 2016

This isn't just a cost issue. Not all development (or testing) should happen with network connectivity required. Other platforms allow you to test or dev locally or even in in-memory mode. It is an insight to the culture of the team releasing this software that this wasn't in by default and still hasn't been actioned in 20months.

@plaurin

This comment has been minimized.

plaurin commented Apr 14, 2016

My problem is that I am working in a team currently working with the Storage Emulator was easy for us now I need to setup all members of my team with proper Azure Subscriptions or find a way to all work on the same Subscription but different Storage Account or Queues. Storage Emulator would be easier than configuring all our web apps and tooling to work in Azure in separated environments.

@ghost

This comment has been minimized.

ghost commented Apr 19, 2016

Any update on when this issue will be resolved? We are starting a new project, and will be relying heavily on Azure storage with WebJobs - not being able to run them locally is a HUGE issue for us, as we have times where we need to develop/demo our product without internet connections being available.

How has this been able to happen for so long? Many people rely on Azure for large scale apps, and issues like this really slow down development velocity!

@ScottyMac52

This comment has been minimized.

ScottyMac52 commented Apr 21, 2016

I personally think it is ridiculous and unprofessional for this pretty major issue to go 2 years without being resolved or at least seriously looked at, I have plenty of evidence to support this claim:

  1. This issue is not even assigned to anyone
  2. The milestone hasn't moved
  3. No labels have been applied

I mean, what gives??? If you are NOT going to fix the issue then say so and then allow someone from the community to come up with a work around solution.

@christopheranderson

This comment has been minimized.

Member

christopheranderson commented Apr 21, 2016

Sorry for the lack of a clear answer here. It's a tricky issue because, for one reason or another, the SDK was never designed with the emulator in mind and there are a lot of gaps between the emulator and the actual Storage products which would impact the SDK.

I don't think that this is a good time for the Azure WebJobs team to investigate this any further, thus I'm moving this into backlog for the time being.

With that said, we're always open for community requests on issues we haven't closed. If you're interested in investigating the gaps and finding work arounds for them, we're happy to work with you to do that in the best way possible.

You can always reach out to me at chrande (at) microsoft (dot) com if you want to discuss this or any other feature and its relative priority.

@christopheranderson christopheranderson modified the milestones: Backlog, 1.2.0 Apr 21, 2016

@ctolkien

This comment has been minimized.

ctolkien commented Apr 21, 2016

Should this discussion be flipped around? Instead of getting WebJobs SDK working on the emulator, should the emulator be improved to close the gap between itself and "real" Azure storage?

Doesn't look like the emulator bits are on github however.

@mathewc mathewc removed this from the Backlog milestone Aug 10, 2016

@PureKrome

This comment has been minimized.

PureKrome commented Aug 10, 2016

Thanks kindly @mathewc :) Looking forward to the 2.0 RTM/RTW.

Keep up the good work, MS / webjobs-team. 🍰

@xt0rted

This comment has been minimized.

Contributor

xt0rted commented Aug 20, 2016

Watch out if you're using the myget feed to get this update. My builds just started failing because older packages were removed from the feed.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Aug 22, 2016

@xt0rted What myget package are you saying doesn't work? Both the nuget and myget packages should work just fine. I'm not aware of any issues :)

@xt0rted

This comment has been minimized.

Contributor

xt0rted commented Aug 22, 2016

@mathewc I was using version 2.0.0-alpha-10313 of the Microsoft.Azure.WebJobs package. My build server started failing on the 19th so this package was probably removed earlier that day or later on the 18th when 2.0.0-beta1-10381 was published.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Aug 22, 2016

I think myget is doing auto-purge on pre-release versions - we haven't been deleting those ourselves. We need to look into this more - might be something we can configure.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Aug 22, 2016

Yes, I confirmed that a default package retention policy that we never created was in effect, and it was configured to retain only 20 versions. I've removed that restriction so now all versions will be kept going forward. Thanks for reporting this :)

@danpetitt

This comment has been minimized.

danpetitt commented Nov 7, 2016

Any news on a v2 RTM?

@martinfletcher

This comment has been minimized.

martinfletcher commented Nov 16, 2016

Could this feature be added to a 1.x release rather than waiting for 2.0?

This has been a known issue for a long time and has put a burden on developers wanting to use queues locally! Pre-Release packages are fine when checking out new features or getting to know a new SDK, but developing against them for production apps is far from ideal!

If 2.0 is not right around the corner from being release (judging from the milestone, it seems far from it!), please release a patch to fix this issue for the current 1.x.

@zeeshanejaz

This comment has been minimized.

zeeshanejaz commented Nov 21, 2016

How come this issue is closed? I don't see the emulator support in the SDK :-/ Am I missing something?

@xt0rted

This comment has been minimized.

Contributor

xt0rted commented Nov 21, 2016

@zeeshanejaz sadly it never made it into v1. It's in v2 but there hasn't been a stable release yet. You can get it on NuGet in one of the the 2.0.0-beta2 builds. I've been using the preview builds ever since this was merged and haven't had any issues with them.

It would be nice if this could be pulled into v1 since we've been waiting 6 months for a release and there hasn't been one.

@dapug

This comment has been minimized.

dapug commented Jan 14, 2017

I'm not getting this to work at all using 2.0.0-beta2, Emulator v4.5. Should I open a new issue?

error: System.InvalidOperationException: 'Invalid storage account 'devstoreaccount1'. Please make sure your credentials are correct.'

string: "UseDevelopmentStorage=true;"

Total loss here, reading this thread it seems this should totally work.

@mathewc

This comment has been minimized.

Contributor

mathewc commented Jan 20, 2017

Yes, if you're having issues with the emulator in the beta builds, please create a new issue.

@zeeshanejaz

This comment has been minimized.

zeeshanejaz commented Jan 30, 2017

In my experience WebJobs SDK 2.0.0-beta2 works with Emulator if you are using WindowsAzure.Storage 7.2.1 but doesn't work if you are using WindowsAzure.Storage 8.0.1. In the latter case, it throws up an exception "Invalid storage account: 'devstoreaccount1'".

I hope this helps someone

@martinfletcher

This comment has been minimized.

martinfletcher commented Feb 7, 2017

Any update on release date for V2 - This fix is much needed by many people!

@NathanNZ

This comment has been minimized.

NathanNZ commented Feb 17, 2017

@zeeshanejaz : This is resolved with 8.x of the WindowsAzure.Storage library if you grab the latest emulator (v4.6 at time of writing) https://go.microsoft.com/fwlink/?linkid=717179&clcid=0x409

@ScottyMac52

This comment has been minimized.

ScottyMac52 commented Feb 17, 2017

@mchudinov

This comment has been minimized.

mchudinov commented Feb 28, 2017

I'm running emulator v 4.6. Still does not work at least from VS 2015 ASP.NET MVC 4.6.2 project.
WindowsAzure.Storage v 8.1.1

Same error "Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure." from my WebJob

connection strings
<connectionStrings> <add name="AzureWebJobsDashboard" connectionString="UseDevelopmentStorage=true" /> <add name="AzureWebJobsStorage" connectionString="UseDevelopmentStorage=true" /> </connectionStrings>

@isaacabraham

This comment has been minimized.

isaacabraham commented Feb 28, 2017

This seems to work reasonably well with 4.6. However, if you give it a reasonable amount of blobs to process in parallel - say, 10, it often just pops with a 500 server error. Not been able to trace it to anything more than that.

@HerrKater

This comment has been minimized.

HerrKater commented Mar 2, 2017

Still not working for me on .net 4.6 with the latest storage SDK & emulator. I can't believe that this has been an issue for 3 years.

@isaacabraham

This comment has been minimized.

isaacabraham commented Mar 2, 2017

The latest emulator seems to have some breaking changes in it as well - at last one has broken the F# Storage Type Provider - that lead to null reference exceptions being thrown.

@StefH

This comment has been minimized.

StefH commented Mar 6, 2017

I'm using version 4.6 in VS 2015, and I can connect (no exceptions), however I don't know how I can define to which storage account I want to connect to.

Note that my local storage account is define like: http://127.0.0.1:10000/devstoreaccount1/webjob-test

And my handler method looks like:

public static void RenameFile(
    [BlobTrigger("input/{blobName}.exe")] Stream input,
    [Blob("output/processed_{blobName}_final.exe", FileAccess.Write)] Stream output)
{
    input.CopyTo(output);
}

Can you please help me?

@mathewc

This comment has been minimized.

Contributor

mathewc commented Mar 8, 2017

I vote that we leave this issue and open new issues as needed for new emulator problems. This issue was all about removing some explicit restrictions we had in place, which we've done. This will make it easier to track things for us, thanks.

@isaacabraham

This comment has been minimized.

isaacabraham commented Mar 8, 2017

@MatthewC do you want me to raise another one then related to many blobs created at once?

@mathewc

This comment has been minimized.

Contributor

mathewc commented Mar 9, 2017

Yes, I want this issue to die so we can start fresh, and track any new issues people have in new issues :) However, note that if you're running into bugs in the emulator or limitations with the emulator, we can't fix those - you should instead log bugs against the storage SDK / emulator itself. Our statement on emulator support as resolved for this issue is that we no longer block it as we were before, however we make no strong guarantees on how well it will work for you. E.g. we know there are features of the WebJobs SDK that use storage features that the emulator doesn't support (e.g. blob leases, at least in the past that was an issue) so YMMV. I view the emulator as a way to try out simple scenarios, however I always use a full test account for my work.

@alexjustus

This comment has been minimized.

alexjustus commented Sep 26, 2017

Regarding the invalid operation exception:

Exception thrown: 'System.InvalidOperationException' in mscorlib.dll

Additional information: Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure.

One way to get around this limitation is to create a Console app rather than an Azure WebJob project. You can then add the Microsoft.Azure.WebJobs nuget package manually, add the standard WebJob boilerplate code to Program.cs and everything should just work.

Visual Studio even lets you publish it as an Azure WebJob like normal via the right-click menu (at least VS 2017 does). It also runs just fine in Azure.

@JuanDVanegas

This comment has been minimized.

JuanDVanegas commented May 7, 2018

I found the correct way to connect, I have not seen any similar comments so I share it.

<add name="AzureWebJobsDashboard" connectionString="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;" /> <add name="AzureWebJobsStorage" connectionString="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;TableEndpoint=http://127.0.0.1:10002/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;" />

Note: Not work with UseDevelopmentStorage=true

https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string?toc=%2fazure%2fstorage%2fqueues%2ftoc.json

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment