Support site-specific tests

[arjan]

It would be nice to have a way to create test cases for everything related to your site. For instance, testing the installation of the site and its domain model, whether all modules are started and configured correctly, whether ACL rules are set up properly, et cetera.

My proposal is to create a test runner
z_sitetests:run(sitename).

which does the following:

• Stop the specific site when it was running (and wait for it to be stopped)
• Create a new database schema for the test
• Start the site, overriding the site config to use this database schema
• (the z_installer now starts the site and installs everything that's configured)
• Wait for the site to be fully started
• Run all tests using eunit that conform to the pattern: sitename_*_sitetest.erl

Initially this will only work inside the Zotonic shell but later we can make it externally callable in order to connect it with a CI system.

Naming the tests *_sitetest helps in preventing conflicts with our current Zotonic-wide tests.

[ddeboer]

Yes! We definitely need this! 👍

• this database schema
• (the z_installer now starts the site and installs everything that's configured)

So the data that will be used is that from sitename:manage_schema/2, right?

[arjan]

Yes, it would use the normal startup sequence.

[ddeboer]

We may also want to exclude site tests from the full Zotonic test suite, so you can run the Zotonic tests (when doing Zotonic dev work) separately from your site’s tests.

[arjan]

With eunit you can a create setup function inside the module to add extra fixtures which are specific to your test.

[arjan]

Indeed, by naming the tests this way they are automatically excluded, that's the idea at least.

[arjan]

I'm adding a new notification for when a site is fully started and ready to go, so we can do:

z_notifier:await(#site_startup{host=Host}, Context),

[mworrell]

@arjan doesn't that await need some extra code, as I would assume it just does a kind of subscribe once, which would then be triggered for any site. Maybe add a z_sites_manager:await(Host)

[mworrell]

On a related note: I would like to change the site start sequence from a simple supervisor to a state machine, so we can better handle crashes, on-demand site starting etc. Each site would then have a single supervised state machine, which manages the site's supervisor

[arjan]

Yeah let's move the state machine thing to a different issue because we already had quite some talk about it in the past. I added a new notification at the end of z_site_startup; that's the point where the site is fully up, with modules loaded, installed datamodel etc.

[arjan]

I'll create a PR shortly.

[arjan]

Wow the site startup is one big hairy mess :'(

[arjan]

Waiting for the module_ready signal seems good enough, no separate notification needed.

[ddeboer]

This is done, thanks @arjan! 😄