feat: add tests to django example

[vinckr]

Add automated tests to the django example. The same template should then be usable for the other examples.

fyi @kevgo should we also add text-runner in this PR? it would check that all commands are correct right?

Related Issue or Design Document

Checklist

• I have read the contributing guidelines and signed the CLA.
• I have referenced an issue containing the design document if my change introduces a new feature.
• I have read the security policy.
• I confirm that this pull request does not address a security vulnerability. If this pull request addresses a security
  vulnerability, I confirm that I got green light (please contact security@ory.sh) from the
  maintainers to push the changes.
• I have added tests that prove my fix is effective or that my feature works.
• I have added necessary documentation within the code base (if appropriate).

Further comments

[kevgo]

We spoke about adding Text-Runner to this repo a while ago and I still have a backlog item for it. Will do this after I'm done implementing the licensing tickets. It likely won't happen in this PR because this PR contains no Markdown files, and I prefer that each PR does only one thing.

[vinckr]

Test is passing 🎉
Can you take a look maybe @kevgo ?

[kevgo]

Sorry for the late review, this one slipped through my notification system. Very cool that you work on this! This is the first code review with me so naturally I have a lot of comments. Don't worry about that, this is totally normal. Just go through each comment and either apply it or come up with a good rationale for keeping your version. Both are equally okay with me, code reviews are intended to be helpful, not dominating.

Once you are through all comments, please move all the files that you have created in the root directory into the django-ory-cloud directory. You are writing an automated test for that app, and tests should live in the same folder as the app they test.

I suggest moving one file at a time, then updating the references to it, then checking that things still work, then committing this. That way, if you mess something up, you can easily go back to the latest working state.

When you have time, we can pair up and go through a few things and concepts together or even do some pair programming.

[kevgo]

And yes, I will focus on adding Text-Runner here and elsewhere very soon. I was tinkering all weekend on Text-Runner to update it!

[vinckr]

Thanks for your review Kevin, really appreciate the detail!
I did not have time to look into it, but will do so now (and probably tomorrow 😁)

Good point about moving the test into the directory. I went by how it is in the ory/docs repo (https://github.com/ory/docs/tree/master/tests) but in this repo it probably makes more sense to have it in the app dir (not sure why it is this way in /docs, maybe to make the CI easier).

Another thing I am not 100% sure of is if we should run all tests for all examples in one workflow or if we should make a dedicated workflow for each example.
If we have only one workflow, its quite a long test (like in ory/docs).
If we have one workflow for each example we have a lot of workflows over time.
Intuitively I am leaning towards the second option, wdyt?

[kevgo]

The repo ory/docs contains a single codebase. It has a src folder containing the source code and a tests folder containing the tests. That makes sense.

The repo here (ory/examples) is a monorepo. It contains many different codebases (the various example apps). Each code base (example app) is pretty independent of each other. They are in different programming languages (Go, Python, TypeScript) and are run separately from each other. Each codebase (example app) in this repo should have its own src and tests folder.

So, long story short, please move the tests for the Django example into a django-ory-cloud/tests folder (after you apply my comments and suggestions).

Good question around how we set up testing. I think your second option makes more sense, especially given the fact that each codebase uses a different technology and therefore might need a different docker image to run.

We can also do your first option (test all codebases in one workflow), with the performance optimization that we only run the test of codebases that are being changed in that PR. I.e. don't run tests for codebases that don't get changed in a PR. This should keep that workflow short.

[kevgo]

Good progress so far! Please let me know if you want me to help implementing some of the suggestions I made.

[vinckr]

Thanks - lots of other topics this week but will try to finish it by EOW.
If you have bandwidth, I also appreciate the help :-)

[vinckr]

Makes sense! I wanna start with the next test next week.
I renamed this one to E2E Tests for django-ory-cloud and then we have a .yml for each.
If we can run everything as part of the official test suite that would be good, but we should make sure to prevent that a broken community example doesn't block updates in /docs.

I also added you as maintainer here, but we could also add you to the ory mod team, then you have access to moderate all discussions as well.

[vinckr]

Everything but two points is resolved 🙏

[kevgo]

Great work! This is coming along nicely! The last open item from my end are the commented out tests. Do you have plans to make them run?

[kevgo]

The finish line is in sight! This is ready to go after the last few comments are addressed. Really good work Vincent!

[vinckr]

Alright added the last changes, thanks for the patience and reviews 🙏
Maybe we can make this a template that you just get with a new example?
make example and it creates a bunch of template files for README/playwright (and possibly more?)

[kevgo]

Very cool, ship it!

[kevgo]

Maybe we can make this a template that you just get with a new example?

I propose we do a few examples and then see how much overlap there is between them. If it's just a few files (readme, playwright config), it might be easier to just copy-and-paste an existing example when one creates a new one.