Some tests always fail with build -testDesktop, and some always fail with build -testCoreClr

[abelbraaksma]

testDesktop

Found while working on #10031. Basically, running build -testDesktop returns these errors, regardless of settings like -ci or -c Debug/Release

• FSharp.Compiler.Scripting.UnitTests.InteractiveTests.System.Device.Gpio - Ensure we reference the runtime version of the assembly
• FSharp.Compiler.Scripting.UnitTests.InteractiveTests.Eval script with invalid PackageName should fail immediately
• FSharp.Compiler.Scripting.DependencyManager.UnitTests.DependencyManagerInteractiveTests.Multiple Instances of DependencyProvider should be isolated
• FSharp.Compiler.Scripting.DependencyManager.UnitTests.DependencyManagerInteractiveTests.Use Dependency Manager to resolve dependency

Test Run Failed.
Total tests: 55
     Passed: 50
     Failed: 4
    Skipped: 1
 Total time: 39,5384 Seconds

testCoreClr

I've logged these separately, as these may warrant some discussion: #10030.

[KevinRansom]

@abelbraaksma , again not on my machine, can you run them -binarylog and post your build logs, I think I need to figure out what bits you are using.

Thanks

Kevin

[abelbraaksma]

Interesting.

The other one was on the wrong branch, as I mentioned there (I closed that issue), but here I double checked. I'll run them again with binary logs enabled.

[KevinRansom]

@abelbraaksma , by and large these options are expected to work. They exist, because they are a core part of my workflow. I think most other people work a bit differently. But everyone here laughs at me as the notepad developer. Because my editor of choice is ultraedit, and it has no f# integration at all.

Kevin

[abelbraaksma]

@KevinRansom Do you mean that for you something like build -testFSharpQA just works? Because everyone I talked to only got it working with -c Release -ci -nobl. It led me to updating the TESTGUIDE, because it took me so long to figure out which combinations were valid and which weren't (some tests always fail in Debug build, for instance).

TestDesktop run Release.zip

(the build.binlog is 35MB in size, even after compression, and Github doesn't allow uploading, hopefully these logs are enough to get the details you need)

Because my editor of choice is ultraedit, and it has no f# integration at all.

Intriguing! Means you're probably very good at memorizing library functions and stuff, given that you don't have context sensitive help and completion ;).

[abelbraaksma]

Btw, what do you use to read the binlog files? I seem to be able to unzip them, but that still leaves a lot of null chars and other weird stuff littered through the files.

[KevinRansom]

@abelbraaksma , I use them a lot, it's why they're there. The set up to run them used to be quite fragile, but I don't think it is any more. For me they mostly work, ... and take a longish time. The thing that I do do, a lot more than anyone else I think is run git clean -fxdq, and I have a kill script that kills the process most likely to be holding on to msbuild and coreclr instances.

I pretty much only run them -c release. because debug messes with codegen to much

I think running them vs also running fsharp or visualfsharp.sln may be an issue.

It looks like:

c:\Temp\preview8packageissues>type D:\Utilities\kill.bat
@echo off
taskkill /f /im dotnet.exe
taskkill /f /im msbuild.exe
taskkill /f /im hostedcompilerserver.exe
taskkill /f /im VBCSCompiler.exe

As an experiment, on my machine now I will run fsharpqa, and report back on the results:

Kevin

[KevinRansom]

[baronfel]

@abelbraaksma you should use https://msbuildlog.com/ to view the binary log files. There's even an avalonia-based version that you can build from source for linux/mac analysis. This is what I use to great effect.

[abelbraaksma]

I have a kill script that kills the process most likely to be holding on to msbuild and coreclr instances.

@KevinRansom I recently added that info to the TESTGUIDE.md as well 😉. A script is handy, but may kill other dotnet.exe processes. I noticed that if they're hanging, their parent process will be gone (which is powershell), and to only kill the right ones, you just need to find the processes with those names that have no parent.

a lot more than anyone else I think is run git clean -fxdq

I do that too (which is how I found out a while back that processes are hanging, actually), or I use -rebuild, but I've the feeling that rebuild doesn't solve all issues. Even after all those things, the mentioned tests are still failing. I'll do a new clean etc and try once more. Or could it be because I run them from a Developer Command Prompt?

I think running them vs also running fsharp or visualfsharp.sln may be an issue.

Good point (was also added to the TESTGUIDE.md in last edits), but I just realized I didn't follow my own advice on the last few testruns 😆. Let's try once more.

As an experiment, on my machine now I will run fsharpqa, and report back on the results:

Wow! After months of on and off struggling with those tests, I've never seen a "proof" that it can actually "just work". I'm going to try again if I can make it work without -ci -nobl (that option is rather annoying as it always rebuilds everything from scratch and sometimes you don't want that).

@baronfel, thanks!

[abelbraaksma]

For me they mostly work, ... and take a longish time

That's why I figured out a way to run them with a test-filter, which makes using these build commands a lot more useful: run one, two, or many tests makes a difference ;). They don't work with Cambridge/QA tests as they are not NUnit/xUnit tests, but those filters do work for anything else. I believe I needed to hack something for it to make it work, though (the /p param gets ignored or something), I'll make a PR for that as I think it is useful.

[KevinRansom]

@abelbraaksma When you create a command shell, how do you do it?

I have an Administrator shell, and I run the Developer script from the rtm Visual Studio, I do run Windows Terminal, and setting that up to run administrator requires is a bit of faffing about.

[KevinRansom]

Administrator is not required, the benefit is that we can run ngen on the protocompiler and built compiler.

[abelbraaksma]

For rewriting the test guide, I tried all commands (well, perhaps not all) on an admin and normal and dev prompt, and I found the results were equal,so I figured the old rule that admin was needed didn't apply anymore.

[abelbraaksma]

Yes, if I do a lot command work I use ngen too, but for these runs I didn't. It does make everything run a lot faster.

(current test is in a non admin shell dev 2019 prompt)

[KevinRansom]

which release of dev2019?

[KevinRansom]

Just running a non-admin build and test of fsharpqa.

[abelbraaksma]

VS 16.7.1. But that only changed to 16.7.2 a few days ago. I can update, of course.

What's weird is that these failing tests seem to be part of the "scripting" tests, and when I run the -testScripting, they all succeed. Didn't check if they indeed are the exact same sets.

[KevinRansom]

Nah that should work fine.

[abelbraaksma]

Last run was after a verified git clean -xdf, still same 4 errors as before.

[KevinRansom]

I had noticed they were all scripting tests too ...

[KevinRansom]

Skype?

[abelbraaksma]

Hmm, it seems to just test the error:

Expected to see output contains 'error NU1101:' and 'FSharp.Really.Not.A.Package'

Other tests failing (reported separately) were because my system is culture nl-NL. I'll dive into the tests, maybe it throws a different error before this that may be caused by the culture.

[abelbraaksma]

Yeah, hold on, I'll find my headset ;)

[KevinRansom]

Closing