Bring back test suites

[jBorkowska]

[@bartekpacia here - I'm editing this message sometimes to keep it updated]

• Add support for Dart-side group()s and multiple test()s #1619
  • patrol develop starts all group()s and test()s in a file, doesn't allow for more granular selection #1713
  • Names of tests in .xcresult don't allow for hierarchy recreation #1714
• Replicate Dart-side test suite structure in the native test report #1502
• Add support for Dart-side test lifecycle callbacks #1503
  • basic lifecycle callbacks:
    • setUp, tearDown (done in Add support for basic test lifecycle callbacks #1721)
    • addTearDown
  • advanced lifecycle callbacks
    • setUpAll (done in Add support for advanced test lifecycle callbacks - setUpAll #1751)
    • tearDownAll
  • interoperability with Hot Restart
  • documentation
  • Patrol tests are failing when run on Firebase Test Lab via Bitrise workflow #1887

[bartekpacia]

Scope: most of the stuff from #1341

In particular:

• Dart groups() must be converted to native JUnit/xcresult test suites. In other words: closely replicate Dart test suite hierarchy on native
• setUp() / tearDown() should work again
  • in particular: complex cases with groups having tests and other groups, each having many setUp()s and tearDown()s should work
• setUpAll() / tearDownAll() should work again
  • they must be called from the instrumentation process, before all tests
  • They won't work with sharding, but we're okay with that

[jBorkowska]

size: L

[bartekpacia]

Starting scope

I'd do iOS first, because I'm more uncertain what we want to do is possible there.

iOS:

• Investigate how to dynamically generate XCTestCase subclasses, so that they're picked by Xcode's test runner.
• Invesigate if any form of nesting of XCTestCase is possible

Android:

• Investigate how to dynamically generate and nest JUnit test suites. If not possible with JUnit 4, check with JUnit 5.

Possible problems - iOS

• XCTestCases cannot be nested. Best I've found is Grouping Tests into Substeps with Activities. We could try hacking it somehow Update: I tried and it isn't what we want.

  Hacking notes

  I wrote a simple UI test with substeps:

  @import XCTest;
  
  @interface HierarchicalTests : XCTestCase
  @end
  
  @implementation HierarchicalTests
  
  - (void)testExample {
    XCUIApplication *app = [[XCUIApplication alloc] init];
    [app activate];
  
    [XCTContext runActivityNamed:@"First group" block:^(id<XCTActivity> activity) {
      [XCTContext runActivityNamed:@"Alpha test" block:^(id<XCTActivity> activity) {}];
      [XCTContext runActivityNamed:@"Bravo test" block:^(id<XCTActivity> activity) {}];
    }];
  
    [XCTContext runActivityNamed:@"Second group" block:^(id<XCTActivity> activity) {
      [XCTContext runActivityNamed:@"Charlie test" block:^(id<XCTActivity> activity) {}];
      [XCTContext runActivityNamed:@"Delta test" block:^(id<XCTActivity> activity) {}];
    }];
  
    XCTAssertTrue(true, "works");
  }
  
  @end
  

  This is still a single test from Xcode's perspective:

  

  The groups substeps are visible in the report:

  Overall, this functionality is for groping chunks of work inside a single test, not grouping multiple tests together.

  ---

  The whole Dart test suite could map into a single XC test, and then we could emulate group()s and test()s using substeps (i.e. XCTContext.runActivityNamed). But this feels very ugly to me.

• We already dynamically create test case methods by XCTestCase.testInvocations. Now we need the equivalent of this method but for dynamic creation of XCTestCase classes, and I didn't find any first-party solutions for it.

  However, it might be possible to solve it with NSPrincipalClass, but it very likely doesn't have access to XCTest APIs such as opening app (which we need to retrieve test cases from the Dart side). Update: I checked this and we indeed can use XCTest APIs from NSPrincipalClass. Fr example, we can make conform to XCTestObservation and call XCUIApplication.launch from XCTestObservation.testBundleWillStart. Example below:

  Principal.h

  First, we have to create a class that will serve as a NSPrincipalClass. Principal class is instantiated automatically when the app is loaded. To make it aware of testing lifecycle, we can make it conform to XCTestObservation.

  #ifndef Principal_h
  #define Principal_h
  
  @import Foundation;
  @import XCTest;
  
  @interface Principal : NSObject<XCTestObservation>
  @end
  
  #endif /* Principal_h */
  

  Principal.m

  @import Foundation;
  @import XCTest;
  #import "Principal.h"
  
  
  @implementation Principal
  
  - (instancetype)init {
    self = [super init];
    if (self) {
      NSLog(@"Principal instance has been initialized.");
    }
  
    [[XCTestObservationCenter sharedTestObservationCenter] addTestObserver:self];
    return self;
  }
  
  - (void)testBundleWillStart:(NSBundle *)testBundle {
    NSLog(@"Test bundle %@ will start", testBundle.bundleIdentifier);
    
    XCUIApplication *app = [[XCUIApplication alloc] init];
    [app launch];
    NSLog(@"Initialized XCUIApplication!");
  }
  
  - (void)testSuiteWillStart:(XCTestSuite *)testSuite {
    NSLog(@"Test suite \"%@\" will start", testSuite.name);
  }
  
  - (void)testSuiteDidFinish:(XCTestSuite *)testSuite {
    NSLog(@"Test suite \"%@\" did finish", testSuite.name);
  }
  
  - (void)testBundleDidFinish:(NSBundle *)testBundle {
    NSLog(@"Test bundle \"%@\" did finish", testBundle.bundleIdentifier);
  }
  
  @end

  The test - nothing fancy:

  PlaygroundTests.m

  @import XCTest;
  @import ObjectiveC.runtime;
  #import "Principal.h"
  
  @interface PlagroundTests : XCTestCase
  @end
  
  @implementation PlagroundTests {
    XCUIApplication *app;
  }
  
  + (void)setUp {
    NSLog(@"%@: class setUp()", NSStringFromClass([self class]));
  }
  
  - (void)setUp {
    NSLog(@"%@: instance setUp()", NSStringFromClass([self class]));
    app = [[XCUIApplication alloc] init];
    [app launch];
  }
  
  - (void)tearDown {
    NSLog(@"%@: instance tearDown()", NSStringFromClass([self class]));
    [app terminate];
  }
  
  + (void)tearDown {
    NSLog(@"%@: class tearDown()", NSStringFromClass([self class]));
  }
  
  - (void)testAlpha {
    NSLog(@"testAlpha");
  }
  
  - (void)testBravo {
    NSLog(@"testBravo");
  }
  
  - (void)testCharlie {
    NSLog(@"testCharlie");
  }
  
  @end

---

Resources I've found:

• try out NSPrincipal:
  • Property List Key - NSPrincipalClass
  • how to add fields to Info.plist when it's missing
  • How to run one-time setup code before executing any XCTest
  • How to set Principal Class for a loadable bundle?
• https://www.mikeash.com/pyblog/friday-qa-2010-11-6-creating-classes-at-runtime-in-objective-c.html
• The classes in the XCTest library:
  • XCTestSuite – Typically, Xcode automatically manages test suites for you. Only use XCTestSuite if you need to define your own custom test suites programmatically. – looks interesting
  • XCTest

[bartekpacia]

iOS - going further

I managed to start the app under test (using XCUIApplication.launch) from within PrincipalClass in its testBundleWillStart callback (from XCTestObservation, which PrincipalClass conforms to).

Next steps:

1. Construct the XCTestCase programatically (i.e. dynamically) and run a simple tests
2. Further investigate if XCTestSuites can contain other XCTestSuites. We need nesting. UPDATE It's likely not possible.
3. Move the long-running gRPC server to PrincipalClass

[bartekpacia]

A different approach

I just had a great talk with @zltnDC and he suggested a fresh approach to the test suites problem.

1. Improve test execution model to support nested group()s and lifecycle callbacks. This is already tracked in Add support for Dart-side test lifecycle callbacks #1503, but currently as part of Bring back test suites #1501.
2. Provide a convenient way for the user to create an "enhanced report" from the "raw report" (the one we currently have). This would replace Replicate Dart-side test suite structure in the native test report #1502.

Why?

Recreating Dart test hierarchy natively is hard

And maybe even impossible (the research above isn't done). See SO question I asked.

Let's just not do it. Patrol's user doesn't care about how the tests are executed – if they're "flat", or if they're natively grouped. What they do care about is the test report. So let's just give them the test report.

Exampe test hierarchy:

Details

integration_test
├── feature_bravo
│   ├── 1_test.dart
│   └── 2_test.dart
└── feature_delta
    ├── 1_test.dart
    └── 2_test.dart

Below is how the report looks like currently (copied from #1341). Let's call it "raw report":

test feature_bravo.1_test.dart ✅
test feature_bravo.2_test.dart ✅
test feature_delta.1_test.dart ✅
test feature_delta.2_test.dart ✅

but we'd like in to look like this (let's call it "enhanced report"):

group feature_bravo
├── test 1_test.dart ✅
└── test 2_test.dart ✅
group feature_delta
├── test 1_test.dart ✅
└── test 2_test.dart ✅

Good news is that the "raw report" contains all information needed to create an "enhanced report" out of it. We could provide a command like patrol reconstruct to convert the .xmls and .xcresults back to the original hierarchy from Dart.

raw_report.xml --> patrol reconstruct --> enhanced_report.xml

We still have to allow for group()s and lifecycle callbacks

The patrol reconstruct described above doesn't solve this problem. But discussion about it belongs into #1503 and #1619 (#1619 (comment)).

Pros and cons

• (+) No native migration required, as it was with v1 -> v2. We'll likely not have to release 3.x with breaking changes in native setup.
• (-) Cloud test runners will still show the "raw report" in their UI. The user will have to download the "raw report" and convert it to an enhanced report using patrol_cli. We might be able to work around it by converting the report at the end of test suite, but that's a long shot.