Skip to content
A system to Instrument running Python code
Branch: master
Clone or download
Latest commit c1534ba Jul 11, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
src/diagnose Version bump. Jul 11, 2019
tests Version bump. Jul 11, 2019
.gitignore Initial commit. Jul 3, 2019
LICENSE Create LICENSE Jul 3, 2019 Oops. Jul 11, 2019 Version bump. Jul 11, 2019


A library for instrumenting Python code at runtime.


Structured logs and metrics and observability are great, but almost always require you to alter your code, which interrupts the flow when reading code. It also typically requires a build cycle to alter; it's no fun making a ticket, getting review, waiting for a build and deploy, and then doing it all again to back out your temporary additions. This is doubly true when doing research, where you might perform a dozen small experiments to measure your live code.

This library allows you to dynamically add probes at runtime instead. Probes are:

  • reliable: errors will never affect your production code
  • ephemeral: set a "lifespan" (in minutes) for each instrument
  • comprehensive: all references to the target function are instrumented
  • fast: measure most functions with fast local lookups; uses hunter (in Cython) for more invasive internal probes.

Individual probes can be created directly by calling attach_to(target):

>>> from import myclass
>>> myclass().add13(arg=5)
>>> p = diagnose.probes.attach_to("")
>>> p.instruments["foo"] = diagnose.LogInstrument("foo", "arg")
>>> p.start()
>>> myclass().add13(arg=5)
Probe (foo) = 5

Instruments aren't limited to recording devices! Use probes to fire off any kind of event handler. Instruments are free to maintain state themselves, or read it from somewhere else, to control their own behavior or even implement feedback mechanisms. A truly evil instrument could even alter the args/kwargs passed to a function on the fly, or call arbitrary Python code to do any number of crazy things. Consequently, it's up to you to govern what instruments are added to your environment.


In a running system, we want to add, remove, start, and stop probes and instruments without having to code at an interactive prompt or restart the system; we do this with an InstrumentManager. Start by configuring the global diagnose.manager:

>>> diagnose.manager.instrument_classes = {
    "log": LogInstrument,
    "hist": MyHistogramInstrument,
    "incr": MyIncrementInstrument,
>>> diagnose.manager.global_namespace.update({"foo": foo})

Later, you can define instruments:

>>> diagnose.manager.specs["instr-1"] = {
    "target": "myapp.module.file.class.method",
    "instrument": {
        "type": "log",
        "name": "myapp.method",
        "value": "result",
        "event": "return",
        "custom": {},
    "lifespan": 10,
    "lastmodified": datetime.datetime.utcnow(),
    "applied": {},

Then call diagnose.manager.apply(), either when you add an instrument, or on a schedule if your store is in MongoDB and the process defining probes is not the target process.

The applied dictionary will be filled with information about which processes have applied the probe, and whether they encountered any errors.

You can’t perform that action at this time.