StubBroker problems with Prometheus middleware.

[jenstroeger]

Issues¹

I’m trying to use the StubBroker in unit tests as described in this section. I noticed one problem and wanted to raise it for discussion: Using the default middlewares

dramatiq/dramatiq/middleware/__init__.py

Lines 44 to 47 in a104a0b

	#: The list of middleware that are enabled by default.
	default_middleware = [
	Prometheus, AgeLimit, TimeLimit, ShutdownNotifications, Callbacks, Pipelines, Retries
	]

for the StubBroker includes the Prometheus middleware which, when run in a pytest context, leaves the following errors after testing:

Exception ignored in: <_io.FileIO name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/gauge_livesum_25420.db' mode='ab+' closefd=True>
ResourceWarning: unclosed file <_io.BufferedRandom name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/gauge_livesum_25420.db'>
Exception ignored in: <_io.FileIO name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/histogram_25420.db' mode='ab+' closefd=True>
ResourceWarning: unclosed file <_io.BufferedRandom name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/histogram_25420.db'>
Exception ignored in: <_io.FileIO name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/counter_25420.db' mode='ab+' closefd=True>
ResourceWarning: unclosed file <_io.BufferedRandom name='/var/folders/2r/yyz1661s4j91vhdkzktg66z80000gn/T/dramatiq-prometheus/counter_25420.db'>

I’ve not yet dug into this issue, but could probably take a look.

For the time being I work around this by removing the Prometheus middleware from the StubBroker² as we discussed here:

_, *middleware = default_middleware
broker = StubBroker(middleware=[m() for m in middleware])

I’m unsure if that’s related to the mentioned Prometheus Gotchas.

What OS are you using?

macOS Ventura 13.2

What version of Dramatiq are you using?

>>> dramatiq.__version__
'1.14.0'

What did you do?

See above.

What did you expect would happen?

No warnings from the Prometheus middleware.

What happened?

Warnings from the Prometheus middleware.

Footnotes

1. I think the template needs updating as it still references the retired Reddit instead of the Groups.io link. ↩

2. It might be a good idea to document the middleware arg for the StubBroker here? ↩

[jenstroeger]

Note that the Prometheus middleware relies on various middleware hooks to set up and tear down — make sure that those hooks execute in the pytest context!