[XCUITests] Launch tests for all controls

[joannaquu]

Platforms Impacted

• iOS
• macOS

Description of changes

Binary change:

File	Before	After	Delta
libFluentUI.a	54,943,520 bytes	54,943,520 bytes	0 bytes

Added a launch test for every control that simply opens each control and ensures the demo app does not crash. setUpWithError will launch the app, and then navigate to the correct page (UIKit and SwiftUI). In the case that the demo app starts on the SideTarBar page, it will use the dismiss button instead of the back button.

Iterations this PR has undergone

• Originally, these tests were written in Swift and each control contained the same code/logic to navigate to their corresponding page.
• An attempt was made to create a base class in Swift to simplify the code and remove the duplicated code. However, customizing each test to each control required that we pass controlName to the base class. This would call init(invocation:), which was an issue because NSInvocation is unimplemented on Swift. See link. Thus, it was rewritten to use objective c.
• Since most of the controls in the FUA repo, as well as other XCUITests in the monorepo use Swift, the tests were changed back to Swift, now using an extension on XCTestCase to address the code duplication.
• Another attempt was made to use a base class in Swift. This time, to avoid running into the init(invocation:) error, the controlName was stored as a property that each test page would override.

This PR does not contain any tests that are specific to each control; specific tests will be added in subsequent PRs.

Resources others may find useful:

• WWDC video
• Cheat sheet

Verification

UIKit	SwiftUI

Pull request checklist

This PR has considered:

• Light and Dark appearances
• iOS supported versions (all major versions greater than or equal current target deployment version)
• VoiceOver and Keyboard Accessibility
• Internationalization and Right to Left layouts
• Different resolutions (1x, 2x, 3x)
• Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)
• iPad Pointer interaction
• SwiftUI consumption (validation or new demo scenarios needed)
• Objective-C exposure (provide it only if needed)

Microsoft Reviewers: Open in CodeFlow

[lyzhan7]

Some thoughts -

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.
• Also thinking that setting up one or two tests to start with and getting those nailed down first would save some time in the future, instead of changing all the tests at once

[edjamesmsft]

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.

One benefit of writing the tests in objc is that we have low objc coverage currently, and that's where our big breaking changes end up happening. So I think writing these in objc isn't necessarily a total downside!

[lyzhan7]

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.

One benefit of writing the tests in objc is that we have low objc coverage currently, and that's where our big breaking changes end up happening. So I think writing these in objc isn't necessarily a total downside!

Hmm interesting, how does writing the tests in objc instead of swift increase our code coverage of objc? If we're still trying to write the same kind of tests (ex. does clicking this button fire the right event), that seems like it would be testing the same code. Do you mean that if we're setting up the tests in obj c we'll be able to test some code that we wouldn't be able to if the tests were set up in swift?

[edjamesmsft]

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.

One benefit of writing the tests in objc is that we have low objc coverage currently, and that's where our big breaking changes end up happening. So I think writing these in objc isn't necessarily a total downside!

Hmm interesting, how does writing the tests in objc instead of swift increase our code coverage of objc? If we're still trying to write the same kind of tests (ex. does clicking this button fire the right event), that seems like it would be testing the same code. Do you mean that if we're setting up the tests in obj c we'll be able to test some code that we wouldn't be able to if the tests were set up in swift?

If we actually do any serious exercising of calling APIs for any given component in objc, we'll have coverage of those calling patterns in objc. Swift and objc do not have the same rules for things like default values for inits, for example, and we have absolutely introduced breaking changes in objc because "it worked in swift fine!", but not so for objc. It's not absolutely critical, but I do believe we should be using our library the way our customers will, and that means we need SOME coverage in objc. If this is one way to get that, it's not a bad thing.

[joannaquu]

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.

One benefit of writing the tests in objc is that we have low objc coverage currently, and that's where our big breaking changes end up happening. So I think writing these in objc isn't necessarily a total downside!

Hmm interesting, how does writing the tests in objc instead of swift increase our code coverage of objc? If we're still trying to write the same kind of tests (ex. does clicking this button fire the right event), that seems like it would be testing the same code. Do you mean that if we're setting up the tests in obj c we'll be able to test some code that we wouldn't be able to if the tests were set up in swift?

If we actually do any serious exercising of calling APIs for any given component in objc, we'll have coverage of those calling patterns in objc. Swift and objc do not have the same rules for things like default values for inits, for example, and we have absolutely introduced breaking changes in objc because "it worked in swift fine!", but not so for objc. It's not absolutely critical, but I do believe we should be using our library the way our customers will, and that means we need SOME coverage in objc. If this is one way to get that, it's not a bad thing.

In that case, would writing these tests in objc be preferred? I've been working on changing this back to Swift so any preferences?

[lyzhan7]

• So it seems like if we want to use a base class, we'll have to write all our tests in objective c instead of swift? I wonder if that trade off is worth it, maybe we can talk offline.

One benefit of writing the tests in objc is that we have low objc coverage currently, and that's where our big breaking changes end up happening. So I think writing these in objc isn't necessarily a total downside!

Hmm interesting, how does writing the tests in objc instead of swift increase our code coverage of objc? If we're still trying to write the same kind of tests (ex. does clicking this button fire the right event), that seems like it would be testing the same code. Do you mean that if we're setting up the tests in obj c we'll be able to test some code that we wouldn't be able to if the tests were set up in swift?

If we actually do any serious exercising of calling APIs for any given component in objc, we'll have coverage of those calling patterns in objc. Swift and objc do not have the same rules for things like default values for inits, for example, and we have absolutely introduced breaking changes in objc because "it worked in swift fine!", but not so for objc. It's not absolutely critical, but I do believe we should be using our library the way our customers will, and that means we need SOME coverage in objc. If this is one way to get that, it's not a bad thing.

Ah I see, so FUA component APIs can be called by either swift or objective c code, and writing the tests in objective c will not only test the component itself but also will test whether calling the APIs with objective c will work. I wonder if just writing objective c tests will cover everything, or if we would actually want tests in both objective c as well as swift? I.e. maybe there is also some behavior that can be only tested in swift

[lyzhan7]

Left a couple comments you could consider (although take a look at the one where one of the asserts seems different from the other ones), overall looks good! Thanks also for filling out the PR description - might also want to update it again now that we got the base test class working

[lyzhan7]

@huwilkes @edjamesmsft do either of you know whether the changes to the xcscheme files should be there? A modification to the development one makes sense to me but I wonder if it makes sense that the release/dogfood ones were changed

[huwilkes]

@huwilkes @edjamesmsft do either of you know whether the changes to the xcscheme files should be there? A modification to the development one makes sense to me but I wonder if it makes sense that the release/dogfood ones were changed

I think if we want to be able run the tests on any scheme, they would all need the change. If we only wanted to be able to run tests on debug we could revert the others, but since it's just in the demo app it's probably okay to leave the change in all three.