UI testing PoC with the Lombiq UI Testing Toolbox (Lombiq Technologies: OCORE-75)

[Piedone]

A proof of concept to add UI testing to Orchard with the Lombiq UI Testing Toolbox. For now, only a rundown of basic smoke tests are run, see this demo video.

Fixes #12834.

Remaining tasks, after the demo at the Orchard meeting:

• What's behind TestBasicOrchardFeaturesExceptRegistrationAsync would potentially need to be copied to be serious here, since if something changes in Orchard deliberately, it can fail.
• The previous should probably be its own project and NuGet package. The UI Testing Toolbox would then depend on it. However, we still to be able to use a different version of it in the OC source... Perhaps it's enough that just the project reference overrides the NuGet dependency. It's the same issue with the theme tests, see below, since those could be in the package too, reusable by third-parties.
• Recreate the Cypress tests so we can see how it compares.
  • Run TestBasicOrchardFeaturesExceptRegistrationAsync with all recipes and similar to the Cypress tests do some basic checks on the frontend for all themes (can use DotNest tests).
  • Recreate 000 SaaS test.
  • Recreate Agency test.
  • Recreate Blog test.
  • Recreate Coming Soon test.
  • Recreate Headless test.
  • Recreate SaaS test.
  • Is setPageSize needed there (it seems like a smoke test, changes the page size under site settings)? Doesn't seem so.
  • There's also a test that launches OrchardCore.Mvc.Web. This can be something simple that runs the app without any Lombiq.Tests.UI.AppExtensions integration.
  • Recreate SQL Server-using tests (see functional_all_db.yml).
  • Recreate PostgreSQL-using test (see functional_all_db.yml). This can work without having specific Postgres support in the UI Testing Toolbox, just we won't be able to use snapshots and setup operations. The engine to use (i.e. SQL Server, Postgres...) can come from an environment variable, so we can run the whole test suite for each engine. ExecuteTestAfterSetupAsync then should throw NotSupportedException and SetupHelpers needs to be deleted. Document this in the UI Testing Toolbox too. Use service containers as currently (only Linux) or an action to setup Postgres? Postgresmes preinstalled both under Linux and Windows, and this simple action can start it.
  • Recreate MySQL-using test (see functional_all_db.yml). Same as Postgres. Use service containers as currently (only Linux) or actions-setup-mysql or setup-mysql?
  • Re-enable Windows builds with os: [ubuntu-latest, windows-latest].
  • Remove temporary test skips: Replace [Theory(Skip = "Minimal test suite for multi-DB testing."), with [Theory,.
  • Remove functional test execution from all GHA workflows, and remove the tests from the code.
• Use WebApplicationFactory directly instead of running the tested app with the dotnet CLI (OSOE-46) Lombiq/UI-Testing-Toolbox#101 would be useful since then then we don't need to change the web app.
• Thinking about it, we'd also need this so this is not constrained to Windows: OSOE-49: Replace Windows-only code in UI Testing Toolbox, upgrade to latest Atata/Selenium Lombiq/UI-Testing-Toolbox#119 Once this is done, we should try it in the Ubuntu build since it's faster already, perhaps the UI testing would also be faster.
• Azure Blob Storage-using tests with Azurite
• Admin monkey tests
• Docs
• Remove IntentionallyFailingTest.
• Use release version of the Lombiq UI Testing Toolbox once OC 1.5 is out and the UITT is upgraded to that. Also update docs\resources\libraries\README.md. Then perhaps ConnectionStringTemplate needn't be custom in UITestBase and that can be the default?
• Evaluate in which build certain tests should be executed. Not all of them are needed in the PR builds necessarily (like DBs apart from SQLite and Aure Blob Storage). Perhaps: only SQLite tests on Ubuntu for PRs as it is now, and everything in preview_ci (less ideal than having feedback under each PR, but better than randomly running it only manually: issues will be visible relatively soon, without anybody waiting), release_ci, and manually. Refactor workflow DRY-ing with the ui-tests action. Pin Lombiq/GitHub-Actions* actions to an SHA.
• Remove any @issue/ references from workflows.
• Re-enable unit tests.
• Configure build timeouts.

[Jetski5822]

Could we not build a separate csproj for tests, i.e. OracheCore.Cms.Web.UITests.csproj or something.... rather than adding in stuff to the main csproj for testing?

[Piedone]

We could, but it would be a complete copy of this one, and then we'd have to maintain two.

[Piedone]

This should be reverted once we have a better approach, see Lombiq/Open-Source-Orchard-Core-Extensions#264 (comment).

[Piedone]

Please share your feedback on whether to continue this here: #12834

[hishamco]

Thanks for your efforts @Piedone, I might review this while i'm interested into Software Testing along side with Localization ;)

FYI I spent last few weeks working and investigating Playwright it quite interested, I'm not sure if there's a plan to use it over Selenium or I might create a package to make use of it in OCC

[Piedone]

Thank you!

Using Playwright instead of Selenium would be a fundamental change, so we don't have plans for it.

[hishamco]

I know it's funddamental, but the good things about playwright that we can write tests for both client and server-side with the same tetchnology same APIs, which is quite cool instead of using Cypress and selenium or any other technology

For that I might start my library as a part of OCC, probably we can have some interest or collaborations to improve the testing in Orchard Core

Thanks again

[Piedone]

Last call: Please let me know under the #12834 issue if you think this is worth continuing working on. If not, I'll close the issue and PR.

[Piedone]

We can revisit this later if there's any interest. Related: #14840