Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In-Memory Integration Testing #1390

Open
jaredcnance opened this issue Oct 17, 2017 · 14 comments
Open

In-Memory Integration Testing #1390

jaredcnance opened this issue Oct 17, 2017 · 14 comments

Comments

@jaredcnance
Copy link

@jaredcnance jaredcnance commented Oct 17, 2017

The problem

I would like to be able to run integration tests using the Azure Function runtime.

Workaround

I can use a FunctionRunner like this to fire up the runtime and then validate the results after some specified timeout.

There are several problems:

  • If I attempt to debug the test, I won't be able to set breakpoints within the actual function code since they are not in the same process. I have to run the test from the command line and add some while(Debugger.IsAttached == false) {/*...*/} hack.
  • I am required to set some arbitrary timeout and hope the function completes execution during that time
  • I cannot run a single function, all functions within the compiled function project will be started when the test host starts
  • requires func.exe to be installed on the testing server in a hardcoded location

Desired Solution

It would be great if we had a solution similar to AspNetCore that might look something like:

// arrange
var testHost = new TestHost(relativePathToCsproj);

// act
await testHost.RunFunctionAsync("MyFunction", maxInvocations: 1);

// assert...

A few things I would like to see included:

  • specify which function should be executed (and probably the binding as well)
  • specify the max number of times the function should be invoked before closing the host
  • a reasonable debugging story
  • assembly/type load errors should reproduce in tests. This probably means the test project should be executed by the function runtime

If the tools to do this already exist (within this project or any other), please let me know.

@paulbatum paulbatum added this to the Active Questions milestone Oct 18, 2017
@paulbatum
Copy link
Member

@paulbatum paulbatum commented Oct 18, 2017

I would start by looking at our own e2e tests. For example we have this test fixture that should get you started.

If you end up building a test harness along the lines of what you outlined above that could be pretty useful in general, you should let us know.

Reopen this if you're still blocked.

Loading

@paulbatum paulbatum closed this Oct 18, 2017
@jaredcnance
Copy link
Author

@jaredcnance jaredcnance commented Oct 18, 2017

It looks like that fixture is dependent upon System.Web.Http, will that be changing with the movement to dotnet core?

Loading

@paulbatum
Copy link
Member

@paulbatum paulbatum commented Oct 18, 2017

Good question. I'm not sure what test fixture updates are planned, @fabiocav should be able to answer.

Loading

@jaredcnance
Copy link
Author

@jaredcnance jaredcnance commented Oct 25, 2017

I don't want to try and craft a solution based on volatile infrastructure, but this is something that is sorely needed, especially when the risk of runtime errors is so high because of Azure/azure-functions-host#992

It looks like JobHost may be the thing I'm looking for in the future?

But then there is also this FunctionsLocalServer which seems to just start a new AspNetCore web host by spawning a new process to run the host via StartHostAction.

Can anyone provide me with the current design/direction so I don't do work that boxes me into a corner in future versions? Between this repository and the azure-functions-cli repo, I'm thoroughly confused as to how the runtime actually works.

Loading

@jaredcnance
Copy link
Author

@jaredcnance jaredcnance commented Dec 1, 2017

Can we re-open this issue? I'm not sure why it is closed. I don't think a proper solution has been provided nor do I have enough information to produce one.

Loading

@paulbatum paulbatum reopened this Dec 7, 2017
@charisk
Copy link

@charisk charisk commented Jan 30, 2018

I'm also facing the same challenge and agree with @jaredcnance - it would be great if there was some guidance here.

Loading

@jeffhollan
Copy link
Member

@jeffhollan jeffhollan commented Jun 11, 2018

Spent some time this weekend playing with this during some travel. I shared my project here. As @paulbatum suggested it follows the same patterns used by the azure functions host project. It's a proof of concept but does allow me to execute the Azure Functions Host inside the test suite and not as an external executable. Has some kinks (wasn't able to ever get the breakpoints inside the function to fire when debugging) but should enable you to verify assembly loading as part of a test

https://github.com/jeffhollan/functions-test-helper

Loading

@s-krawczyk
Copy link

@s-krawczyk s-krawczyk commented Apr 30, 2019

+1

Loading

@danhamlin
Copy link

@danhamlin danhamlin commented Jun 4, 2019

Curious if @fabiocav has anything he is able to share about a proper test fixture for function apps.

Loading

@riezebosch
Copy link

@riezebosch riezebosch commented Aug 2, 2019

Checkout my solution: https://github.com/riezebosch/AzureFunctions.TestHelpers. It indeed uses the JobsHost from the SDK with some helper functions & classes aside to invoke http-triggered functions and wait for orchestration completion.

Loading

@jeffhollan
Copy link
Member

@jeffhollan jeffhollan commented Aug 2, 2019

@riezebosch this is awesome! So glad someone took some time to look into this as my sample is very stale at this point. Will check this out

Loading

@matt-lethargic
Copy link

@matt-lethargic matt-lethargic commented Mar 31, 2021

3.5 years on, is there any official advice on how to achieve this?

Loading

@fabiocav
Copy link
Member

@fabiocav fabiocav commented Mar 31, 2021

@matt-lethargic there have been recommendations from @jeffhollan and @paulbatum on this issue. Aside from that, that has been no other work done.

Please do understand that, though the original question/request was made a while ago, there have been no commitments to bring anything beyond what we've been able to share. This is not something that have seen enough demand to be prioritized over other items with higher impact.

We decided to keep this open, as the idea is good and valid, but I want to make sure that wasn't sending the wrong message. In the meantime, I'll add a help wanted label to this as it would be a good opportunity for community engagement.

Loading

@milad-ghafoori
Copy link

@milad-ghafoori milad-ghafoori commented Nov 25, 2021

I am looking for guidelines and tips to write in-memory integration tests for a set of V4 Azure Functions (.Net Core 6 & C#) in isolation mode (out-of-process). Is there any relevant content/sample/article available?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants