Update to Android 7.1

[ChrisMaddock]

Updating to Android 7.1 would allow the runner to use the wider netstandard-1.6 framework, and allow updating Xamarin Forms. It's currently non-trivial due to not being included in the default image for the appveyor CI - but that's in the pipeline to be updated, see: appveyor/ci#1280

@rprouse - how much are we worrying about backwards compatibility here? I've never done much any Android development, but it doesn't see too hot on backwards compatibility. I'm not sure on the effects of trying to load a netstandard dll in an older Android level - I imagine they could be unpleasant!

(This all came out of #85)

[gtbuchanan]

From #85:

Unfortunately - this had meant the 2.x version of Xamarin Forms I'd updated to previously would potentially fail at runtime as some of it's dependencies were targeted to Android 7.0. (See: https://forums.xamarin.com/discussion/44678/psa-you-must-set-your-target-compile-version-of-your-android-projects-to-latest) I've reverted to 1.5.x for now.

Will fixing this issue will also update support for Xamarin 2.x? I'm currently on Xamarin 2.3.4.247. When I upgrade nunit.xamarin in my current project from 3.0.1 to 3.6.1, I encounter this major issue starting the app which prevents running of any tests:

06-07 09:30:37.092 D/Mono    (22004): Assembly Loader probing location: 'System.Runtime.Loader'.
06-07 09:30:37.092 F/monodroid-assembly(22004): Could not load assembly 'System.Runtime.Loader' during startup registration.
06-07 09:30:37.092 F/monodroid-assembly(22004): This might be due to an invalid debug installation.
06-07 09:30:37.092 F/monodroid-assembly(22004): A common cause is to 'adb install' the app directly instead of doing from the IDE.

nunit.xamarin 3.0.1 actually worked with Xamarin 2.x somewhat, but I have tests from other assemblies that aren't being included due to #59. This is why I tried upgrading. However, none of my tests will run now on my Samsung S8 running Android 7.0.

Trying to make sure I understand all of this correctly.

[ChrisMaddock]

nunit.xamarin 3.0.1 and nunit.xamarin 3.6.1 shipped with the same version of Xamarin forms - so I'm not sure if that would be the cause of the difference. From a quick check, looks like they both also targeted Android 6.

Accordingly, I'm not really sure, sorry. 😞 @rprouse - any ideas? The best option may just be to try it out - if it works, we'd appreciate a pull request!

[gtbuchanan]

Interesting. Thanks for the info. After further testing, I found this issue only occurs with v3.6.1 while using a project.json file instead of packages.config. I created a separate project using the NUnit Android template and updated the Xamarin Forms package. It runs fine with packages.config, but I have the same issue when I convert it to use project.json to match my app.

I'm not sure what would have changed between 3.0.1 and 3.6.1 to cause this, but looks like you're correct in that it's not related to Xamarin Forms or Android version. I can't tell if the issue is due to a change in nunit.xamarin or a change with nunit itself...

[ChrisMaddock]

Ah - this is interesting! Can you check which build of the nunit.framework dll you project references, in each case? Check the paths to dll's the references are pointing to inside the NuGet package.

My thought process: Android 7 would by default reference the .NET Standard 1.6 build of the framework, whilst Android 6 would reference PCL. I thought there was differences between what project.json vs packages.config actually pull in - but I can't keep up with the latest NuGet changes at the moment.

With your not-working, project.json version, try explicitly adding the three dependencies the .NET Standard 1.6 version of the NUnit framework requires (link). I'd expect these dependencies to be inherited automatically, but with the conflicting versions, perhaps something isn't happening as it should.

[ChrisMaddock]

Regarding the difference between 3.0.1 and 3.6.1 - v3.0.1 predates the .NET Standard build of the NUnit framework, so your project would always end up with the PCL build - which has no additional dependencies.

[gtbuchanan]

Adding the NUnit dependencies manually results in the same error, in addition to the following "build errors" (that do not actually prevent the app from deploying):

System.Runtime.Loader 4.3.0 provides a compile-time reference assembly for System.Runtime.Loader on MonoAndroid,Version=v7.1, but there is no run-time assembly compatible with win.

One or more packages are incompatible with MonoAndroid,Version=v7.1 (win).

I'm assuming that's what's silently going wrong with project.json. I've attached the sample project I've been working with. (EDIT: Fixed the csproj to remove obsolete Xamarin 1.5 target)

[bgmeiner]

Hi,

I'm having the excact same problem and I've been pulling my hair out over this ^^ @gtbuchanan did you find any solution for this?

[gtbuchanan]

@bgmeiner Not really. I've resorted to switching my Android project back to use packages.config. This is not desirable due to the number of indirect references in my project, but it's the only workaround I could get to work. I'll probably switch back to xUnit in the long run since it supposedly supports PCL test projects, which is something else I'm having issues with.

[ChrisMaddock]

Ok, this is screwy. Here's what I think is happening.

NUnit 3.6.1 packaged two relevant builds of the framework - the .NET Standard 1.6 build - in the NuGet package under netstandard16 and a PCL - packaged for various targets, but specifically MonoAndroid.

Long story short, the .NET Standard 1.6 build won't work for Android, due to what I believe is a Microsoft package only 'pretending' to fully support .NET Standard. :-| More on that at nunit/nunit#2237

Another strange bit, is that I believe packages.config and project.json are adding the dependency differently. My guess is that packages.config is using the assembly in the MonoAndroid dir (the PCL, which doesn't rely on the broken System.Runtime.Loader package), whereas project.json seems to be ignoring the more specific MonoAndroid directory, and going straight to use the netstandard16 build. My guess is that this is related to nunit/nunit#2165 - because there isn't a specific monoandroid dependency group, it jumps straight for .NET Standard 1.6, and ignores the MonoAndroid dir. Totally logical.

The only workaround I can think of for now, is using packages.config rather than project.json - unless anyone can figure out how to force project.json to look in a particular directory. We'll need to work out how to do that for our future packaging, if our .NET Standard 1.6 isn't truly .NET Standard 1.6. Either that, or we drop the dependency somehow.

[ChrisMaddock]

Ok, so I'm out of time for today, and I don't know when or if I'll be able to get back to this any time soon. But if anyone else is able to pick this up, here's what I would have tried next...

• Clone this nunit.xamarin project, and try updating to Android 7, and NUnit 3.7.1.
• I suspect this might still fail, due to System.Runtime.Loader not available for iOS/Android nunit#2237. In that case, I'd attempt to build a test version of the NUnit NuGet package that puts the .NET Standard 1.3 build in the MonoAndroid directory, and includes an empty dependency group specifically for MonoAndroid (like in Trying to install NUnit 3.6.1 on .NET Framework asks for download of 20 more packages nunit#2165 (comment)), to see if that forces project.json to load the .NET Standard 1.3 build, as opposed to the .NET Standard 1.6.

If that still doesn't work - then I may well be barking up entirely the wrong tree. As I said, I don't claim to have a lot of knowledge on what NuGet have been doing lately. 😄 A PR would be most welcome, if someone has the time to play about and see if we can get all dependencies in sync.

[ChrisMaddock]

Before all of that, possibly worth checking my suspicions about the difference between packages.config and project.json are correct. If they are, DirectoryAssert should be available in the project.json, but not packages.config. Hopefully it should be possible to check that via Intellisense, before it fails at runtime.

[ChrisMaddock]

I've just addressed nunit/nunit#2237 - testing that required most of the work that would resolve this issue.

I'll do a PR for this some point over the next week or so.

[ChrisMaddock]

Last weekends attempt at this didn't go so well, and I'll shortly be away for a while. If anyone else wants to take it on in the interim, please feel free.

[mrbiggred]

I got the same error message @gtbuchanan "Could not load assembly 'System.Runtime.Loader' during startup registration.". This was for a Xamarin Android project using a project.json file.

On a whim I tried forcing the project to use NUnit 3.8.1. When upgrading I didn't get any errors about NUnit.Xamarin requiring NUnit 3.6.1, which does seem a bit weird to me.

My project.json file looks like:

{
  "supports": {},
  "dependencies": {
    "NUnit": "3.8.1",
    "nunit.xamarin": "3.6.1",
    "Xamarin.Forms": "2.4.0.269-pre2"
  },
  "frameworks": {
    "MonoAndroid,Version=v7.1": {}
  },
  "runtimes": {
    "win": {}
  }
}

I reference Xamarin.Forms 2.4.0.269-pre2 because my non-test Xamarin project is .NET Standard. Not sure if this matters for NUnit or not.

I'm curious if this workaround works for anyone else? Also, why is NUnit.Xamarin forced to NUnit 3.6.1 and does not allow newer versions of NUnit? Thanks.

[ChrisMaddock]

v3.8.1 removes your previous error because it contains the fix for nunit/nunit#2237. We're holding off on updating this project right now, as there's some critical issues with test parallelisation and collection constraints in v3.8.1 - which will hopefully be resolved in v3.8.2. If you're not using any of these features - you may be fine to use v3.8.1.

nunit.xamarin is locked to a specific version of the NUnit framework, as it currently interacts with some internal APIs and objects to run it's tests. In this instance, it appears none of these methods have changed, so your test works - but that's not a guaranteed/supported interface. I actually thought the nunit.xamarin runner was compiled against a specific version of the framework - that appears to have been lost in the converison from packages.config -> project.json. We should fix that! In the long run, we want to convert nunit.xamarin to use the NUnit Engine, which will allow users to use any version of the framework they chose.

A more reliable solution (if you can avoid the bugs currently in NUnit v 3.8.1) would be to clone this repo, update this repo to use NUnit 3.8.1, and the build your own nunit.xamarin package to reference.

[mrbiggred]

@ChrisMaddock Thank you for your help.

I have updated the NUnit Xamarin project to reference NUnit 3.8.1 and also use Android 7.1. Everything appears to work, at least for my simple case. Would you like me to create a pull request? If so should I use the master branch or 3.8.1 branch? If you don't want a pull request I post my version of this nuget package to MyGet.

As far as I could tell the NUnit Xamarin project references the NUnit nuget package. It does not appear to be compiled against a set version of NUnit but I could be mistaken.

[rprouse]

@mrbiggred if you can make a pull request on the master branch we will look at it. It may require other minor tweaks if you are willing to work with us...

[ChrisMaddock]

As far as I could tell the NUnit Xamarin project references the NUnit nuget package. It does not appear to be compiled against a set version of NUnit but I could be mistaken.

No - I don't think it does, but I think it should be!

A pull request would be great! I don't think we should do another full release of nunit.xamarin until NUnit 3.8.2 is released - but if we can get something merged into master, then at least an improvement will be available. 🙂

@rprouse - nunit.xaramin doesn't have a MyGet dev feed, like our other projects yet, does it? Would be good if we could set it up, I think!

[rprouse]

A MyGet feed would be good, I think we currently only have the AppVeyor feed, if that...

[mrbiggred]

@rprouse I will do a pull request shortly and I have not problem making other minor tweaks or fixes. I haven't done many pull requests before so please feel to correct any mistakes I make. I can work on this till about 4pm mountain time today.

[ChrisMaddock]

Fixed by #102