[Proposal] It is proposed to implement UI tests using: https://playwright.dev/

[pmario]

This is a reminder that: It is proposed to implement UI tests using: https://playwright.dev/

From: #6756 (comment)

By the way, we should probably switch to Playwright; it's built by some of the same people as Puppeteer, but works with a wider variety of browser engines, including Firefox.

https://playwright.dev/

That's interesting. I did have a look at the release history. It seems the got from beta 0.10.0 to v1.0.0 in just 3 months. .. That's impressive. Had a look at an intro-video from last month. ... The code generator and the test-debugging tools look impressive too. ... It actually looks like fun to create tests.

Coming back to the SVG test we have here. .. I can see how the visual tests can be implemented and since it seems to have all the DOM structure info, we are able to check that. ... Good

So it will complement the existing jasmine tests, and we should probably replace those tests. which need to use the fakeDOM at the moment ... right?

#6756 should be re-visited for the first experiments.

[Jermolene]

So it will complement the existing jasmine tests, and we should probably replace those tests. which need to use the fakeDOM at the moment ... right?

I suspect we should run all the tests under both Node.js and the browser for maximum coverage.

[rmunn]

I've had some experience with Playwright. The one drawback to Playwright is that it'll be significantly slower than the Jasmine tests we have already. That's because Playwright loads a fresh browser instance and a fresh copy of your page/site for every test, which is good from a test-isolation standpoint. But that means every Playwright test will load a fresh TiddlyWiki instance, and have to wait several seconds for TW to start up, before it can run its test. Each Jasmine test runs in milliseconds, but each Playwright test is going to run in seconds, and the total runtime for the tests is going to be much longer.

I definitely like Playwright — it's been a very good experience to use so far — but there is also a place for simpler tests using frameworks like Jasmine that don't need to start up a browser every time.

(NOTE: It's also possible for Playwright tests to run in milliseconds, if they don't need to start up a browser instance. For example, yesterday I was writing some Playwright tests that tested an API, using Playwright's APIRequestContext class. That one can just send HTTP requests via node.js, and those tests ran in 8-20ms. So not all Playwright tests will run in seconds, only the ones that exercise the browser. But since that's what Playwright is best at, if you're writing Playwright tests then you probably are writing mostly browser-exercising tests, and therefore most of your Playwright tests will be slow.)

[pmario]

Yea, the plan is, to use Playwrite along side the existing tests. At the moment, we don't have a possibility to check real UI handling. eg: Opening and closing a new tiddler in a new window ... and so on.

[Jermolene]

Thanks @rmunn that's interesting. We'd definitely want to optimise startup. Perhaps there's a middle ground where we can reuse the same tab for each test.

[rmunn]

We did something along those lines in our project, where we tried to re-use a browser tab between tests. Playwright doesn't make it easy. but you can do it. You start by doing something like this:

test.describe('One set of tests', () => {
  let browserTab;
  test.beforeAll(async ({ browser }) => {
    const context = await browser.newContext();
    browserTab = await context.newPage();
    const port = '8800'; // TODO: Get port from config somehow
    await browserTab.goto(`http://localhost:${port}/`);
  });
});

Then in each test, where the Playwright documentation wants you to get the { page } fixture as a parameter in your test, you instead re-use the browserTab object that you've saved. It will be torn down at the end of the describe block, after all the tests that use it have finished running.

With that approach, you get a new browser context (essentially a browser tab with attached cookie storage) for each describe block. Which is the best way to do it: you don't want to re-use a browser context between two sets of unrelated tests, as you can really pollute your testing environment badly that way. Keeping it scope-limited to everything in the same describe block limits the possible interactions tests can have between them.

Also, while I'm giving Playwright tips: by default, it runs tests in parallel on a file basis. Tests within a single file will run in order, top to bottom, but all test files will be started in parallel. (Or rather, they will all be queued at once, and then Playwright will start about as many parallel worker processes as you have processor cores in your machine, and each worker process will take one file from the queue, then grab the next available queued file when it finishes its current task). If you make each describe block be in its own file, that's usually the best way to run Playwright tests as fast as possible. (If your tests modify the environment that other tests can see, say they're creating GitHub projects via the GitHub API and experimenting with making changes to those test projects, then just make sure each describe block gets its own project name that it modifies and no other describe block is looking for that same project name, and you won't have tests able to interfere with each other).

[Jermolene]

Great stuff @rmunn. It would be great to have some experiments on this stuff. There are so many bits of core development that I would love to be able to have proper browser tests for...

[CodaCodr]

I'll just leave this here for perusal:

https://fusebit.io/blog/node-testing-comes-to-core/

[joshuafontany]

That is how C#.net NUnit + Sekenium tests are run (seperate Setup/Teardown methods per class name/fixture). Very smart @rmunn. I am writing a C# test suite for a corporate website at the day job. Would be cool to have a similar suite for TiddlyWiki.

[hoelzro]

It's not Playwright, but I just saw that @EvidentlyCube is working on some Cypress tests: #6989

[EvidentlyCube]

@hoelzro I am also planning to drop a sick album an update which includes identical tests but written in Playwright, gonna have it ready in an hour or so.

[pmario]

Just found this video: https://www.youtube.com/watch?v=lCb9JoZFpHI ... which is a great intro