New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Async testing with F# #34

Open
haf opened this Issue Oct 20, 2013 · 30 comments

Comments

10 participants
@haf

haf commented Oct 20, 2013

I have been trying to find on how to test asynchronous code from F# without having to manually unwrap the async value, but rather having the test framework do that for you.

Since it seems 3.0 will bring with it improvements to async as well as the ability to parallelise tests, it would be great if there was some guidance on how to use the project with F#.

My question is simpler than this however; how do I create tests with F# that are async without wrapping them in TPL?

@simoneb

This comment has been minimized.

Contributor

simoneb commented Oct 20, 2013

Hi Henrik,

I believe, and your post seems to confirm, that F# might be implementing
async in a completely different way compared to C#'s async/await. If that's
the case, then there is no async support for F# in the current or upcoming
(for the time being) versions of NUnit. In the other case then support for
async methods (which reflect their requirement of being async in to the
calling methods) comes out of the box so you can declare your test methods
to be async and NUnit will handle them properly.

Simone

On Sun, Oct 20, 2013 at 7:36 PM, Henrik Feldt notifications@github.comwrote:

I have been trying to find on how to test asynchronous code from F#
without having to manually unwrap the async value, but rather having the
test framework do that for you.

Since it seems 3.0 will bring with it improvements to async as well as the
ability to parallelise tests, it would be great if there was some guidance
on how to use the project with F#.

My question is simpler than this however; how do I create tests with F#
that are async without wrapping them in TPL?


Reply to this email directly or view it on GitHubhttps://github.com//issues/34
.

@haf

This comment has been minimized.

haf commented Oct 20, 2013

What would be required to implement support for F# async? What's the hook that decides the 'unwrapper' to use for the return value?

@haf

This comment has been minimized.

haf commented Oct 20, 2013

Also; how do I run the nunit tests? Is there a pre-compiled test runner?

@haf

This comment has been minimized.

haf commented Oct 20, 2013

Never mind my questions, I figured it out. I'm done soon.

@haf

This comment has been minimized.

haf commented Oct 20, 2013

Running into a roadblock where a core-NUnit test case is failing:
yield return new object[] { Method("AsyncTaskResultCheckSuccess"), ResultState.Success, 1 };

Is failing due to this line:

if (returnsValWrapper && (parms == null || !parms.HasExpectedResult && !parms.ExceptionExpected))
    return MarkAsNotRunnable(testMethod, "Async test method must have Task or void (C#) or async<unit> (F#) return type when no result is expected");

Because pars is null, and the ExpectedResult is off.

  Wrong result state
  Expected: <Passed>
  But was:  <Skipped:Invalid>

   at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message, Object[] args) in Assert.cs: line 395
   at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message) in Assert.cs: line 372
   at NUnit.Framework.Internal.NUnitAsyncTestMethodTests.RunTests(MethodInfo method, ResultState resultState, Int32 assertionCount) in AsyncTestMethodTests.cs: line 85
@simoneb

This comment has been minimized.

Contributor

simoneb commented Oct 21, 2013

One thing to watch out for, we don't want to require end users to have F#
installed in order to run NUnit, so we should probably not keep strong
references to F#-specific types in NUnit's codebase. This is also what we
used to do in NUnit v2 for async/await (in that case those were references
to TPL types), but seeing that in 3.0 we have strong references I assume we
dropped support for .NET < 4 altogether.

Simone

On Sun, Oct 20, 2013 at 9:53 PM, Henrik Feldt notifications@github.comwrote:

Running into a roadblock where a core-NUnit test case is failing:
yield return new object[] { Method("AsyncTaskResultCheckSuccess"),
ResultState.Success, 1 };

Is failing due to this line:

                if (returnsValWrapper && (parms == null || !parms.HasExpectedResult && !parms.ExceptionExpected))
                    return MarkAsNotRunnable(testMethod, "Async test method must have Task or void (C#) or async<unit> (F#) return type when no result is expected");

Because pars is null, and the ExpectedResult is off.

Wrong result state
Expected:
But was: Skipped:Invalid

at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message, Object[] args) in Assert.cs: line 395
at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message) in Assert.cs: line 372
at NUnit.Framework.Internal.NUnitAsyncTestMethodTests.RunTests(MethodInfo method, ResultState resultState, Int32 assertionCount) in AsyncTestMethodTests.cs: line 85


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26682378
.

@haf

This comment has been minimized.

haf commented Oct 21, 2013

Yes, I see. However, it's probably easier to get it to work initially by just taking the reference and then either typing it "stringly" or doing some other stuff that we'll figure out.

I would appreciate some guidance on how to configure the new code with all of the projects, where the CI server is, what tests are supposed to fail on master right now and what runtimes it should be possible to compile on (CI server?).

@simoneb

This comment has been minimized.

Contributor

simoneb commented Oct 21, 2013

Sure, it was just a pointer to how we should probably do it to avoid
introducing unwanted dependencies. There's not much setup needed, the build
runs with NAnt, which you need to download before doing anything else, then
you can look at how the batch scripts in the root folder invoke it. You can
see what targets are included in the build file, they include net from
version 2 to 4.5 and mono from version 2 to 4.

There is an unofficial build server set up on teamcity
herehttp://teamcity.jetbrains.com/project.html?projectId=NUnit_NUnit3_Framework&tab=projectOverview.
You can have a look around and feel free to ask questions if you need to
know more. Log in as guest if you don't have a user there, or just create
one.

Simone

On Mon, Oct 21, 2013 at 9:25 AM, Henrik Feldt notifications@github.comwrote:

Yes, I see. However, it's probably easier to get it to work initially by
just taking the reference and then either typing it "stringly" or doing
some other stuff that we'll figure out.

I would appreciate some guidance on how to configure the new code with all
of the projects, where the CI server is, what tests are supposed to fail on
master right now and what runtimes it should be possible to compile on (CI
server?).


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26699742
.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Oct 21, 2013

Hi Henrik,

Looking at the code in your post as well as the modified code in your pull
request, I'm struck that you may be conflating two entirely different
things - with the reasonable excuse that MS has given them the same name.

It seems as if C# async and F# async may be two entirely different
implementations. In that case, we shouldn't try to make the same code
handle them. OTOH, if F# async is just like C# async at a lower level (that
is, uses an AsyncRegion, etc.) then we should figure out how to handle both
at that lower level.

Does anyone know which it is?

Charlie

On Sun, Oct 20, 2013 at 1:53 PM, Henrik Feldt notifications@github.comwrote:

Running into a roadblock where a core-NUnit test case is failing:
yield return new object[] { Method("AsyncTaskResultCheckSuccess"),
ResultState.Success, 1 };

Is failing due to this line:

                if (returnsValWrapper && (parms == null || !parms.HasExpectedResult && !parms.ExceptionExpected))
                    return MarkAsNotRunnable(testMethod, "Async test method must have Task or void (C#) or async<unit> (F#) return type when no result is expected");

Because pars is null, and the ExpectedResult is off.

Wrong result state
Expected:
But was: Skipped:Invalid

at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message, Object[] args) in Assert.cs: line 395
at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint expression, String message) in Assert.cs: line 372
at NUnit.Framework.Internal.NUnitAsyncTestMethodTests.RunTests(MethodInfo method, ResultState resultState, Int32 assertionCount) in AsyncTestMethodTests.cs: line 85


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26682378
.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Oct 21, 2013

In addition, we need to check that F# is available on all platforms where
we want to build. I'm OK with requiring anyone who wants to work on NUnit
to install F#, but it has to be possible on their platform.

Can we use F# on Linux and OS/X?

Charlie

On Mon, Oct 21, 2013 at 1:21 AM, simoneb notifications@github.com wrote:

One thing to watch out for, we don't want to require end users to have F#
installed in order to run NUnit, so we should probably not keep strong
references to F#-specific types in NUnit's codebase. This is also what we
used to do in NUnit v2 for async/await (in that case those were references
to TPL types), but seeing that in 3.0 we have strong references I assume
we
dropped support for .NET < 4 altogether.

Simone

On Sun, Oct 20, 2013 at 9:53 PM, Henrik Feldt notifications@github.comwrote:

Running into a roadblock where a core-NUnit test case is failing:
yield return new object[] { Method("AsyncTaskResultCheckSuccess"),
ResultState.Success, 1 };

Is failing due to this line:

if (returnsValWrapper && (parms == null || !parms.HasExpectedResult &&
!parms.ExceptionExpected))
return MarkAsNotRunnable(testMethod, "Async test method must have Task
or void (C#) or async (F#) return type when no result is expected");

Because pars is null, and the ExpectedResult is off.

Wrong result state
Expected:
But was: Skipped:Invalid

at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint
expression, String message, Object[] args) in Assert.cs: line 395
at NUnit.Framework.Assert.That(TActual actual, IResolveConstraint
expression, String message) in Assert.cs: line 372
at
NUnit.Framework.Internal.NUnitAsyncTestMethodTests.RunTests(MethodInfo
method, ResultState resultState, Int32 assertionCount) in
AsyncTestMethodTests.cs: line 85


Reply to this email directly or view it on GitHub<
https://github.com/nunit/nunit-framework/issues/34#issuecomment-26682378>
.


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26699563
.

@simoneb

This comment has been minimized.

Contributor

simoneb commented Oct 21, 2013

haf, pull requests are configured to be built on the build system we're using temporarily. right now there seems to be a problem with files missing from the nant scripts, you probably forgot to reference them. See all the results here. Some of the failures are not genuine, currently we can't build on Linux and for .NET 4.5.

@haf

This comment has been minimized.

haf commented Oct 21, 2013

F# async has been around since 2007, and it's a library-thing, not a compiler thing. The C# compiler isn't as strong, so it needs to rely on a library, as opposed to the F# compiler that doesn't know about async.

As such, there's no AsyncRegion being generated in F# nor is it something language-specific. Except that it's a frequently used feature, given that Task of T implements IAsyncResult and therefore is possible to map into async<'T>. So it's (almost) identical in usage and philosphy as C# tasks, but it's not built into the compiler itself.

So given that you're supporting tasks, I felt it would be nice if you were supporting async; the same sort of code gets written for both.

Yes, F# is most definitely usable on OS X and Linux, I'm using those operating systems myself.

@CharliePoole

This comment has been minimized.

Member

CharliePoole commented Oct 22, 2013

What .NET version is required to use the F# async support? I'm suspecting it's available earlier than 4.5.

@simoneb

This comment has been minimized.

Contributor

simoneb commented Oct 22, 2013

(meanwhile I've asked the guys who maintain the build server to install F#
tooling there)

On Tue, Oct 22, 2013 at 7:13 PM, CharliePoole notifications@github.comwrote:

What .NET version is required to use the F# async support? I'm suspecting
it's available earlier than 4.5.


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26827727
.

@elksson

This comment has been minimized.

Member

elksson commented Oct 22, 2013

It looks like Async began support with a pre-release in f# 1.9.2.9 however you might want to read the link because It says here while async can be used for multi threaded applications it is not the best option. They recommend using PLINQ and Futures instead.

http://blogs.msdn.com/b/dsyme/archive/2007/10/11/introducing-f-asynchronous-workflows.aspx

On Oct 22, 2013, at 11:47 PM, simoneb notifications@github.com wrote:

(meanwhile I've asked the guys who maintain the build server to install F#
tooling there)

On Tue, Oct 22, 2013 at 7:13 PM, CharliePoole notifications@github.comwrote:

What .NET version is required to use the F# async support? I'm suspecting
it's available earlier than 4.5.


Reply to this email directly or view it on GitHubhttps://github.com//issues/34#issuecomment-26827727
.


Reply to this email directly or view it on GitHub.

@haf

This comment has been minimized.

haf commented Oct 22, 2013

@elksson I prefer actors and message passing myself, but Don is correct. Using async in a parallel setting makes it harder to reason about the code than if you were using messages, because the call contexts are intertwined and there are no semoitic labels such as message names to reason with, but only an async call context.

@elksson

This comment has been minimized.

Member

elksson commented Oct 22, 2013

I could pretend I knew what Actors and message passing are but have no clue as I have no F# experiance
On Oct 23, 2013, at 1:01 AM, Henrik Feldt notifications@github.com wrote:

@elksson I prefer actors and message passing myself, Don is correct. Using async in a parallel setting makes it harder to reason about the code than if you were using messages, because the call contexts are intertwined and there are no semoitic labels such as message names to reason with, but only an async call context.


Reply to this email directly or view it on GitHub.

@CharliePoole CharliePoole added this to the 3.0 milestone Feb 21, 2015

@CharliePoole CharliePoole modified the milestones: 3.2, 3.0 Aug 23, 2015

@CharliePoole CharliePoole modified the milestones: 3.2, Backlog Dec 5, 2015

@CharliePoole CharliePoole removed this from the Backlog milestone Jul 25, 2016

@nosami

This comment has been minimized.

nosami commented Aug 25, 2016

I just discovered a pattern that works for F#

open System.Threading.Tasks
open System.Runtime.CompilerServices

let toTask computation : Task = Async.StartAsTask computation :> _

[<Test>]
[<AsyncStateMachine(typeof<Task>)>]
member x.``Test``() = toTask <| async {
    do! asyncStuff()
}
@nosami

This comment has been minimized.

nosami commented Aug 25, 2016

Would be much nicer if NUnit supported F# async natively though

@rprouse

This comment has been minimized.

Member

rprouse commented Aug 25, 2016

@nosami, yes it would. Contributions from the F# community are welcome, or even just some help for us functional neophytes 😄

@yreynhout

This comment has been minimized.

yreynhout commented Sep 20, 2016

I just discovered another pattern that works for F#

open System.Threading.Tasks
open NUnit.Framework

let toAsyncTestDelegate computation = 
    new AsyncTestDelegate(fun () -> Async.StartAsTask computation :> Task)

[<Test>]
member x.``TestWithNUnit``() = 
    Assert.ThrowsAsync<InvalidOperationException>(asyncStuff 123 |> toAsyncTestDelegate)
    |> ignore

[<Test>]
member x.``TestWithFsUnit``() = 
    asyncStuff 123 
    |> toAsyncTestDelegate
    |> should throw typeof<InvalidOperationException>
@nosami

This comment has been minimized.

nosami commented Jul 6, 2017

This is how @bradwilson implemented it for xunit

@jnm2 jnm2 added this to To do in Async improvements May 27, 2018

@jnm2 jnm2 moved this from To do to In progress in Async improvements May 27, 2018

@jnm2 jnm2 moved this from In progress to Investigate in Async improvements May 27, 2018

@jnm2

This comment has been minimized.

Contributor

jnm2 commented May 28, 2018

@rprouse Should I add an .fsproj to the solution (nunit.testdata.fsharp) for the IDE experience, or should I compile F# test fixtures on the fly using by referencing the FSharp.Compiler.Service nupkg from nunit.framework.tests?

The .fsproj would require all NUnit contributors to have the desktop F# SDK and the .NET Core F# SDK installed which may or may not be default. I could check.

@jnm2 jnm2 moved this from Investigate to Waiting for team in Async improvements May 28, 2018

@mikkelbu

This comment has been minimized.

Member

mikkelbu commented May 28, 2018

I prefer the second option, but I'm also a little biased as I already have F# installed 😄.

@rprouse

This comment has been minimized.

Member

rprouse commented May 30, 2018

I would prefer an F# project because it is easiest and I have it installed too, but it isn't a default workload in VS. How hard is it to use the FSharp.Compiler.Service?

@bradwilson

This comment has been minimized.

bradwilson commented May 30, 2018

In @xunit, we don't take any direct F# references (in the production code), but rather rely on reflection to discover the things that might be around and usable. Notably: https://github.com/xunit/xunit/blob/215dcb1a37f0eef0277e5ece002cc7dc030fe2a2/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs#L152-L180

@jnm2

This comment has been minimized.

Contributor

jnm2 commented May 30, 2018

@bradwilson Awesome, nice to know I'm on the right track!

@jnm2

This comment has been minimized.

Contributor

jnm2 commented May 30, 2018

@rprouse I haven't tried to use it yet, but it can't be fundamentally different than Roslyn and I've used that in tests before. Would you like to see what it would look like?

@rprouse

This comment has been minimized.

Member

rprouse commented May 30, 2018

@jnm2 I'd be fine with Roslyn as long as it isn't too much work for you.

@jnm2

This comment has been minimized.

Contributor

jnm2 commented May 31, 2018

@rprouse Okay, I'll try out FSharp.Compiler.Service (the Roslyn-like option) then! I'm honestly on the fence, so I'm equally happy either way.

@jnm2 jnm2 moved this from Waiting for team to In progress in Async improvements May 31, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment