Go all-in on using pytest instead of unittest in docs, examples, etc

[benhoyt]

Pytest really makes writing tests a lot lighter and easier-to-read with plain functions and assert foo == bar rather than self.assertEqual(foo, bar). Exemplify this in our SDK docs, API reference docs, charmcraft init templates examples, and so on.

[Batalex]

I chatted with Ben, and I'll get started on this issue!

[Batalex]

Progress tracker:

• https://juju.is/docs/sdk/write-unit-tests-for-your-charm Note Removed autouse
• https://juju.is/docs/sdk/write-scenario-tests-for-your-charm
• https://juju.is/docs/sdk/open-a-kubernetes-port-in-your-charm
• https://juju.is/docs/sdk/write-a-unit-test-for-a-charm Blocked by charmcraft: the document says that the code snippet comes from charmcraft init, so we should wait for the finalised PR for consistency
• charmcraft init templates PR
• juju tutorial k8s repo
  • part 8 PR
  • part 9 PR
  • part 10 PR
  • part 11 PR

[benhoyt]

Cool, looks like a good start. We'll cast our eyes over those PRs. Can you please add a note or link about why #4 is blocked by charmcraft? Or is that just referring to the charmcraft init template PR?

[sed-i]

I am not too familiar with pytest but I think there's a subtle difference we need to keep in mind:

• With unittest, by instantiating harness inside setUp, we will always have the same "state", even when we ask the test runner to run one single test method from a test class.
• With pytest and autouse fixtures this could be a tricky detail. And what would happen if a test file is taking the "rolling changes" approach, and then the user asks the test runner to run only a given test method?

[tonyandrewmeyer]

• With unittest, by instantiating harness inside setUp, we will always have the same "state", even when we ask the test runner to run one single test method from a test class.
• With pytest and autouse fixtures this could be a tricky detail.

Creating a Harness object in setUp gives a fresh object for each test, so is the same as scope='function' in pytest. If anyone is creating a Harness in setUpClass that's probably questionable but I would imagine you'd rewrite it using scope=module.

I'm not sure how this ends up being a tricky detail - it's essentially the same behaviour. Am I missing something here?

And what would happen if a test file is taking the "rolling changes" approach, and then the user asks the test runner to run only a given test method?

Can you explain what the "rolling changes" approach is?

[sed-i]

Right, fixtures have scope. In that case do we want to encourage function scope in the templates?

Module scope is the "rolling changes" approach: in one test function you add a relation, and it is still there in the next test function. Can get messy.

[benhoyt]

@sed-i It looks like function scope is the default, right, so as I understand it that's what the suggested templates propose?

[tonyandrewmeyer]

Module scope is the "rolling changes" approach: in one test function you add a relation, and it is still there in the next test function. Can get messy.

How are you doing this with unittest? With setUpClass or setUpModule? It seems like it would be just as messy with those, and some sort of "set up my relations" helper function would be much safer.

[sed-i]

• For some reason I though module scope is the default. Function scope as default is great.
• With unittest I do not use setUpClass or any equivalent of module scope, precisely because it is messier.

I'm very curious to see how utests with pytests could look like. On one hand, it's much easier to use hypothesis with pytest. On the other hand, pytest is notorious for having too much hidden magic.

[Batalex]

TBH, the hidden magic is a big part of why I like pytest. Though side effects from hidden fixtures can be a pain to debug, I agree.

[benhoyt]

@Batalex I'm with @sed-i here. I really like pytest's simple use of assert (and automatic assert introspections for good output), but I can't stand the magic. Reason: automatically wiring things up makes code hard to follow. I've been confused by this many times when looking at codebases -- I once had to debug some pytests with four or five layers of fixtures depending on each other, some of which were in conftest.py, which is imported automagically. Especially with conftest.py and autouse=True, pytests executes code implicitly without any wiring in sight.

On my "articles to write" list I have this suggestion: write about my love-hate relationship with pytest - assert and test finding, yes! Magical fixtures wired up by name, no thanks! conftest.py and autouse=True, double no thanks!

Maybe I'll write that article someday. I wish there was something like pytest but without those magical features. :-)

[tonyandrewmeyer]

Thanks for all the work on this @Batalex !