Skip to content

Enhancements for tests.helpers and our test suite#660

Merged
scragly merged 5 commits into
masterfrom
unittest-helpers-improvements
Nov 13, 2019
Merged

Enhancements for tests.helpers and our test suite#660
scragly merged 5 commits into
masterfrom
unittest-helpers-improvements

Conversation

@SebastiaanZ
Copy link
Copy Markdown
Contributor

This pull request introduces several enhancements to our testing suite to make running tests easier and reduce the number of developer surprises. It does come with a couple of breaking changes that may influence currently open PRs (I haven't checked yet, but they should be easy to resolve).

Enhancements

Stricter custom mock types

One of the reasons for using custom mock types is because they follow the specifications of the actual objects they're mocking. However, as @kwzrd remarked, the spec method we used for that only provided partial protection: spec does not prevent setting attributes that should not exist on the object we are mocking. I have solved this by using the stricter spec_set option that makes sure the validity of attributes is also checked when they are set (as opposed to only when they are accessed).

Prevent logging when running individual tests

As you may have noticed, logging messages were not suppressed when running an individual test file (instead of the entire test suite), making the output extremely messy. This is now no longer the case.

Prevent RuntimeWarning after MockBot.loop.create_task

When "scheduling" a coroutine object with MockBot.loop.create_task, the coroutine object is not actually scheduled as a task since we're mocking the create_task method and we're not actually running an event loop. While this is typically what we want, this triggers RuntimeWarnings since the coroutine object was never awaited. I've solved this by setting a default side_effect for MockBot.loop.create_task that calls stop() on the passed coroutine object.

Allow name attribute to be set with __init__

The name kwarg has a special meaning in the __init__ methods of Mock and MagicMock preventing us from setting a name attribute during initialization of our custom mock types that inherit from these classes. This is not desirable, as discord.py objects frequently have a name attribute and we want to be able to set it when initializing the mock. I've added special handling of the name kwarg to prevent it from being passed to the __init__ of the parent classes.

Note: This does prevent us from using that special name handling of Mock/MagicMock, but I don't think that option is useful for us in the first place. Being able to set the name atttribute of the mock is much more useful and actually in use a lot in our current tests.

Use actual attribute names instead of 'safe' surrogates

Previously, I used surrogates like message_id to set the id attribute of mocks in __init__ to prevent eclipsing the built-in id. Since I've now switched to kwargs-only approach in the __init__, this is no longer necessary, as no eclipsing will occur. This means we can now just pass the attributes by their actual name to prevent "developer surprise". (My apologies for those hours of frustration, @lemonsaurus...)

Allow all attributes to be set during initialization of the mock

Due to a mistake, some attributes could not be set during initialization of a custom mock type as the attributes were overwritten by default values. This is no longer the case.

Breaking changes

  • Our custom mocks can no longer be initialized with positional arguments to set attributes. This means you always have to set attributes with a keyword argument (e.g, name="Helpers").
  • All attributes must now be set using the actual attribute name. Surrogate names like user_id or message_id no longer work and will result in an AttributeError since they are not valid attributes of the types being mocked.

Previously, logging messages would output to std.out. when running
individual test files (instead of running the entire suite). To
prevent this, I've added a `for`-loop to `tests.helpers` that sets
the level of all registered loggers to `CRITICAL`.

The reason for adding this to `tests.helpers` is simple: It's the
most common file to be imported in individual tests, increasing the
chance of the code being run for individual test files.

A small downside of this way of handling logging is that when we are
trying to assert logging messages are being emitted, we need to set
the logger explicitly in the `self.assertLogs` context manager. This
is a small downside, though, and probably good practice anyway.

There was one test in `tests.bot.test_api` that did not do this, so
I have changed this to make the test compatible with the new set-up.
The `name` keyword argument has a special meaning for the default
mockobjects provided by `unittest.mock`. This means that by default,
the common d.py `name` attribute can't be set during initalization of
one of our custom Mock-objects by passing it to the constructor.

Since it's unlikely for us to make use of the special `name` feature
of mocks and more likely to want to set the d.py `name` attribute, I
added special handling of the `name` kwarg.
Our custom `discord.py` now follow the specifications of the object
they are mocking more strictly by using the `spec_set` instead of the
`spec` kwarg to initialize the specifications. This means that trying
to set an attribute that does not follow the specifications will now
also result in an `AttributeError`.

To make sure we are not trying to set illegal attributes during the
default initialization of the mock objects, I've changed the way we
handle default values of parameters. This does introduce a breaking
change: Instead of passing a `suffix_id`, the `id` attribute should
now be passed using the exact name. `id`.

This commit also makes sure existing tests follow this change.
Previously, the coroutine object passed to `MockBot.loop.create_task`
would trigger a `RuntimeWarning` for not being awaited as we do not
actually create a task for it. To prevent these warnings, coroutine
objects passed will now automatically be closed.
@SebastiaanZ SebastiaanZ added t: bug Something isn't working type: enhancement p: 1 - high High Priority a: tests Related to tests (e.g. unit tests) labels Nov 13, 2019
Copy link
Copy Markdown
Contributor

@lemonsaurus lemonsaurus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spectacular work. Let's do it.

Copy link
Copy Markdown
Contributor

@scragly scragly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a: tests Related to tests (e.g. unit tests) p: 1 - high High Priority t: bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants