test: add e2e tests for CTB restarts from file changes

[innerdvations]

What does it do?

• adds e2e test for creating a collection type and that the ctb tutorial modal appears
• adds util to watch for the server restart
• adds util to help ignore the ctb tutorial modal
• adds a very hacky test framework and util for resetting the filesystem state after changes like CTB edits, uploads, etc
• minor refactoring and docs

I'm not super happy with how hacky it is, but it seems stable enough from the time I've been working with it. The most important thing I would like to find a solution for is the delay() to wait for the server to realize files have changed and it begins its restart process. However, without access to the output of the server I can't think of anything better. Perhaps a custom winston logger could pipe it somewhere that we could watch that would be realtime?

Why is it needed?

We were limited by being unable to test the CTB because of the server restarts, filesystem changes, etc.

Roadmap

After this PR, I am planning the following changes:

• bring CLI tests from v5/main to develop (and then work from there on further changes)
• copy/paste and clean up first attempt at CTB tests from chore(e2e): CTB end-to-end tests #18065
• refactor e2e+cli tests to share code, ideally everything except the test runner itself (jest vs playwright command)
• lots of other little cleanups, like improving the restart monitoring being added here

How to test it?

Running the full test suite with multiple browsers (that is, re-using the same running instance of strapi across multiple test runs of the same suite, and re-using the test-apps between runs ie without using --setup) should work every time. And continuing to run them many times in a row, they should always succeed without flakiness.

Related issue(s)/PR(s)

DX-1166

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
contributor-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 4, 2024 10:59am

[alexandrebodin]

FYI there is also this historical PR with a bunch of good tests #18065

[innerdvations]

I'm going ahead and opening this PR as-is with the delay() giving strapi time to detect file changes and restart. While it could technically be flakey, I haven't seen it fail in about 5 runs on github so far, and never locally either with many more runs.

I'm very open to better ideas, but trying to watch the logs for the restart / started also feels overly complicated and potentially flakey as well.

[joshuaellis]

Overall looks great, well done for cracking this, super hard stuff. Just some minor comments to discuss

[joshuaellis]

why are you doing it afterAll but also in beforeEach? Maybe it should be in afterEach instead?

[innerdvations]

It's a tricky question here.

In this file, the correct thing would be only the beforeEach (since each test makes changes) and then other suites would ensure on their own it was clean with a beforeAll/beforeEach as needed. But doing the filesystem reset is slow and only a couple suites make file changes (ctb and uploading), and I didn't want to update every other test file to do that.

So I put the "bad" code (trying to clean up for other tests in afterAll is bad practice because you're not guaranteed it will run) separate even though it means we run it one extra time, because ultimately I want to remove it altogether once we have a faster method to detect/reset the files that would be ok to add universally.

[joshuaellis]

V nice well done

[innerdvations]

V nice well done

not quite, one of the runs failed 😞

I think I have another solution though that I'll try to run through the CI several times and see if it still happens