End to End Testing framework for Tanager

[andonyns]

Previous Experiences

Throughout different projects in Ed-Fi, there have been different approaches to find an end to end testing framework for the tools and applications, this has varied between languages and requirements for the tools, to name a few:

ODS / API

The ODS/API has a set of Postman tests to run different scenarios, this is built to run in CI, this includes a C# test harness setup with a set of postman scenarios to run the tests.
For local development, you need to run the Test Harness first before running the Postman tests.
This is really useful and solid, although very complex and driven by the PowerShell scripts

Admin API

Admin API uses Postman exclusively to run the tests, guided by a group of multiple collections to allow for team collaboration and avoid difficult conflicts when doing changes to the tests. This does not do any kind of setup but runs against a running Admin API instance (the setup is not part of the execution).

The management of multiple collection is a good way to avoid conflicts, but it can be hard to keep track of all collections. Additionally, if a request in one collection requires another request from another collection, you need to manually run the tests first.

Admin App and Data Import

These tools have the difference of having e2e UI tests. This is done by using Playwright with BDD scenarios provided by Gherkin (developed in JavaScript).
This was a good way to accomplish e2e UI testing, and playwright is a growing tool for automation.

Meadowlark

For Meadowlark, we did a research about options of tools to use, where we considered Postman and other tools, see Meadowlark Test Automation for more information.

After the research, We implemented the e2e testing with Jest, using SuperTest as the HTTP library with assertions (this could have been done with Axios and Jest assertions), using TestContainers to setup the test environment.

This allows us to run the tests in an isolated environment both in local and GitHub Actions for CI, this has other advantages such as functions for repeated tasks, and a simple structure to understand and add new tests.

API Demo Endpoints vs API E2E Tests

This is a separation worth doing. In API Tests, we tend to have a group of tests that aim to cover both the list of endpoints used for demo purposes and for exploratory testing and the actual e2e tests, but those should be handled separately, even with the same tool. Some of the reasons for this are:

Demo Endpoints	E2E Tests
Should be simple and easy to follow	Can have more complex setup and teardown steps
Should be simple endpoints to verify functionality	Should be guided by scenarios
Can setup the environment or other tests	Should be isolated and not depend on previous tests or leave data set

This unification of two different set of tests is not a bad practice, but can be hard to maintain and complicated for onboarding of new team members and for community contributions.

Because of this, it can be better to have those two things separated, the Demo Endpoints can be done with Postman, Bruno or REST client (as in meadowlark), for the e2e Tests, the next section will show a list of options to select.

Tool Options for Tanager

Here's a list of tools evaluated and some notes regarding each one:

Postman

Pros:

• The "easy choice". The tool that most people are familiar with.
• Endpoint organization.
• Pre-request scripts at the endpoint level or folder level.
• Tests.
• Additionally, the scenarios already created for the ODS / API can be reused.

Cons:

• Collection maintenance when multiple people are working at the same time.
• Not driven by code.
• Endpoint reutilization.
• No built in setup and tear down, can be accomplished with something like the Test Harness of ODS / API but this increases the complexity.

Bruno

The pros are similar to Postman's, except the lack of pre-request scripts on folder level.

Pros:

• Driven by code.
• Test Assertions.

Cons:

• Feels like an interesting project still in early stages, which can cause issues that are not supported (or will not get supported).
• Could be discontinued since there's only one maintainer.
• The foundations of the tools could be changed since there's not clarity on future versions of the tool. See this issue as an example. This suggests changing core functionality that could include breaking changes.

Jest (Code Driven)

This is a good option because of the way it helped create the test suite in Meadowlark.

Pros:

• Code Driven.
• Step (or endpoint) reutilization.
• Built in Setup and Teardown hooks (which will integrate well with TestContainers).

Cons:

• Requires strong technical background.
• Not suitable for demos.
• The Meadowlark structure is JavaScript specific.

The Code Driven option is a good option to create a robust e2e test framework, but it is better to use a tool set according to the language.

Proposal

Create e2e tests using Playwright's API testing capabilities, using TestContainers.

Explore the possibility of using SpecFlow for scenario management.

As part of this Discussion, I will create a demo of how this would look like before continuing.

[stephenfuqua]

Additional Postman cons: closed source potential vendor lock-in; now requires creating a Postman account.

For this project, I like the continued distinction between the e2e tests and demonstration code. The low-ceremony approach and simplicity of RestClient is still very attractive. My votes: 1) RestClient, 2) Bruno, 3) Postman.

The core e2e tests will be written by developers, so staying in a developer mindset with Playwright is attractive. I've used SpecFlow successfully in the past; it can be nice to have a separation of test description from test code. Allows people to contribute to the testing process without needing to become programmers, and allows others to easily review the test cases. There is a little extra work there, but it could be worthwhile. I'm interested to hear from others about this proposal.

For Testcontainers, I assume this would be Testcontainers for .NET. This is a great framework - fully supportive.

[stephenfuqua]

@andonyns can you comment further on what you meant in your remark about .NET falling behind the JavaScript version of Playwright?

[andonyns]

Thinking more about it, this might be more of personal perspective but not something critical.

Playwright was originally born as a Node library, and the basics of it were created in the JavaScript ecosystem.
After the success of the tool, this was ported into Python, Java and .NET.

The .NET Playwright Library is relatively new compared with the Node version, therefore, it's still a growing platform compared with more mature libraries like Selenium.

For instance, the Node repo has +12k commits, and the .NET has only +1200 commits.

There are more blog posts, notes and material about Playwright for Node than other languages. But I've found some examples of Playwright for .NET (most of those are for UI tests but structure is similar)

https://github.com/microsoft/playwright-dotnet/tree/main/src/Playwright.Tests
https://github.com/executeautomation/Playwrightdotnet

I don't think this is an issue with Playwright for .NET, specially since a lot of its functionality works out of the box with NUnit.

[bradbanister]

I think the non-Node ports will always be behind the Node version, but the .net version is a Microsoft project and so should be kept up well. Playwright in general is quite popular, so there's still a good community to draw from.

[bradbanister]

I think everyone knows how much I like the simplicity of RestClient for manual testing/demos. 🙂
Playwright + Testcontainers makes sense to me.

[andonyns]

The SpecFlow project is not being maintained at the moment (there haven't been updates for over two years). See this issue for more info.

Because of this issue, one of the previous maintainers created a fork Reqnroll.

After testing this approach, I consider the best approach to be NUnit + Playwright + Reqnroll + TestContainers, where:

NUnit is the test runner.
Playwright interacts with the API.
Reqnroll takes care of defining scenarios (this will help us define complex scenarios).
TestContainers sets up the environment against which one

See Ed-Fi-Alliance-OSS/Data-Management-Service#12 for a demo of the approach (still in progress)

[stephenfuqua]

@vimayya you might be interested in seeing this info about SpecFlow and Reqnroll. Please also review to see if you have any long-term concerns with this testing framework discussion.

@andonyns your PR is a nice demonstration. I like the direction here.