Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert MonoVM Samples into Functional Tests #43865

Closed
33 of 44 tasks
steveisok opened this issue Oct 27, 2020 · 6 comments
Closed
33 of 44 tasks

Convert MonoVM Samples into Functional Tests #43865

steveisok opened this issue Oct 27, 2020 · 6 comments

Comments

@steveisok
Copy link
Member

steveisok commented Oct 27, 2020

The mono samples do a good job of showing off the platforms we support and often serve as a mini test bed. Unfortunately, they have a tendency to break over time and sometimes we'll be slow to realize it. To combat that, we should convert the samples into functional tests.

We will want to:

  • Ensure we build them as part of the build
  • Make sure the functional tests are run on CI
  • Each test should have a return code for pass/fail
  • Set default features flags to match to SDK defaults
  • Extract tests into individual test cases for
    • iOS device (arm64)
      • Interpreter only
      • AOT only
      • PInvoke tests
      • Invariant culture only mode
    • iOS simulator (x64 & arm64)
      • Interpreter only
      • AOT only
      • PInvoke tests
      • Invariant culture only mode
    • tvOS device (arm64)
      • Interpreter only
      • AOT only
      • PInvoke tests
    • tvOS simulator (x64 & arm64)
      • Interpreter only
      • AOT only
      • PInvoke tests
    • Android device+simulator
      • Interpreter
      • JIT
      • AOT
      • PInvoke
      • Invariant culture only mode
    • WebAssembly Console
      • Normal interp
      • AOT
      • PInvoke
    • WebAssembly Browser
      • Normal interp
      • AOT
      • PInvoke
      • Invariant culture only mode

Integrate w/ XHarness

  • iOS
  • Android
  • Wasm
@steveisok steveisok added area-Build-mono tracking This issue is tracking the completion of other related issues. labels Oct 27, 2020
@Dotnet-GitSync-Bot Dotnet-GitSync-Bot added the untriaged New issue has not been triaged by the area owner label Oct 27, 2020
@marek-safar marek-safar removed the untriaged New issue has not been triaged by the area owner label Oct 27, 2020
@marek-safar marek-safar added this to the 6.0.0 milestone Nov 24, 2020
@marek-safar marek-safar changed the title [Mono] Turn Samples into Functional Tests Convert MonoVM Samples into Functional Tests Nov 24, 2020
@steveisok steveisok moved this from To do to In progress in Infrastructure Backlog Mono Nov 24, 2020
MaximLipnin added a commit that referenced this issue Dec 11, 2020
- Included the ios sample project to the library test build. The sample still can be build/run locally using make file.
- Added copying resulting HelloiOS.app to helix directory from where it can be consumed by CI.
- Changed the sample to return 42 when building for CI, which is used as an expected exit code for xharness.
- Xharness for iOS supports running an application on a simulator w/o test runner inside, the sample is run by this approach.
- Checked locally that xharness can run HelloiOS.app successfully.

Related issue: #43865
MaximLipnin added a commit that referenced this issue Dec 11, 2020
- Included the android sample project to the library test build.
- Added copying resulting HelloAndroid.apk to helix test directory from where it can be consumed by CI.
- Set up the return value (42) as an expected exit code for xharness.
- Xharness for android doesn't support running an application w/o test runner yet  

Related issue: #43865
akoeplinger pushed a commit that referenced this issue Dec 16, 2020
- Included the WASM console sample project to the library test build. The sample can still be built/run locally using make file.
- Added zipping/copying the sample application to helix directory from where it can be consumed by CI.
- To run on Helix, tried to use a simple helix work item which wouldn't rely on generated RunTests.sh and just call the xharness command directly.  

Relates to #43865
MaximLipnin added a commit that referenced this issue Dec 18, 2020
This is a continuation of #45768 for the browser sample.  
Which sample to run depends on the Helix scenario:  
- the console sample runs under normal scenario;
- the browser sample runs under WasmTestOnBrowser scenario.

Relates to #43865
steveisok pushed a commit that referenced this issue Jan 6, 2021
Part of #43865

The functional tests:

- live under src/tests/FunctionalTests directory.
- are built as a part of the library tests build.
- are isolated from the build/test setup used for the tests in src/tests
- run on CI
@MaximLipnin
Copy link
Contributor

Add integration tests for pinvokes for all configurations

@marek-safar Could you please clarify or give some example of what in particular should be done here?

@marek-safar
Copy link
Contributor

@MaximLipnin we need tests for each configuration where the test calls unmanaged code via pinvoke. Something like

class C
{
	[DllImport ("somelib.dll")]
	static void ExternalNativeAPI ();

	public static void Main ()
	{
		int result = ExternalNativeAPI ();
		// check result
	}
}

@steveisok
Copy link
Member Author

@MaximLipnin Please arrange the functional tests similar to how the tasks are listed in this issue.

Also, with respect to Set default features flags to match to SDK defaults, you're going to need to look at how XA/XI initially set up their configuration. For example, we want to mirror as much as possible the AOT arguments in XA xamarin/xamarin-android#5539 . It will require some digging.

@radical
Copy link
Member

radical commented Mar 3, 2021

Heads up: Once #47683 is merged, you might want to add these test projects as xunit tests.

MaximLipnin added a commit that referenced this issue Mar 11, 2021
Contributes to #43865

./build.sh mono+libs -os Android -arch x64 -c Release
./dotnet.sh build /t:Test src/tests/FunctionalTests/Android/Device_Emulator/InvariantCultureOnlyMode /p:TargetOS=Android /p:TargetArchitecture=x64 /p:Configuration=Release
MaximLipnin added a commit that referenced this issue Mar 11, 2021
Contributes to #43865  

1. ./build.sh mono+libs -os iOS -arch x64 -c Release  
2. ./dotnet.sh build /t:Test src/tests/FunctionalTests/iOS/Simulator/InvariantCultureOnlyMode /p:TargetOS=iOS /p:TargetArchitecture=x64 /p:Configuration=Release
MaximLipnin added a commit that referenced this issue Mar 24, 2021
Contributes to #43865.  

Based on https://github.com/xamarin/xamarin-macios/blob/834b0888853210ca01d5e8f5ce2c65fe9c55fc10/dotnet/targets/Xamarin.Shared.Sdk.targets#L138-L146

The changes:  
- added a property group with iOS/iOSSimulator default feature flags;  
- aligned the conditions to check the properties activating the defaults for true in order to follow better msbuild practice.
@steveisok steveisok removed the tracking This issue is tracking the completion of other related issues. label Aug 10, 2021
@steveisok steveisok modified the milestones: 6.0.0, 7.0.0 Aug 10, 2021
@radical
Copy link
Member

radical commented Aug 11, 2021

FYI, for wasm, we have tests for using native libraries, and for invariant culture mode, in Wasm.Build.Tests.

@mdh1418
Copy link
Member

mdh1418 commented Jul 20, 2022

@steveisok it sounds like the mono samples in this repo are still useful, do we still want to extract tests for iOS device? Either way it seems like this can either be closed or pushed to 8

@lateralusX lateralusX modified the milestones: 7.0.0, 8.0.0 Jul 25, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

7 participants