Profiling Framework

[heikoheiko]

Add support for easy profiling
ideally the continuous integration would notify if a commit slows the system down

[hackaugusto]

The easiest for now is to set a hard limit in individual tests, and fail the unit test if the perf is unacceptable. This has a couple of disadvantages like, not knowing about how much we improved/regressed, and not having plot of the performance changes.

Perf dashboards that I know of:

• catapult dashboard [example]
  • This is an appengine app, but we can gather the data from a build and publish to a "local" server
• perf-o-matic [example, alerts, ref]
  • Confusing project, really
• codespeed [example]

I don't know if there is any continuous integration as a service that also has perf dashboards built-in, so we might need to host our own server.

[hackaugusto]

c82bc42 adds an experiment to profile greenlets in isolation (preserving the timming per greenlet with the stack trace)

[heikoheiko]

@hackaugusto what's the status here? what is missing to close this?

[hackaugusto]

@heikoheiko no support in the CI

[hackaugusto]

Another dashboard option: http://asv.readthedocs.io/en/latest/

[hackaugusto]

closing in favor of #61

[hackaugusto]

I found a really neat approach in the deno project:

• there is a benchmark script, which is executed by the CI after a successful deploy, and saves the data into a json file.
• the results are published as a github page using the branch gh-page, so there is no need for a server to store the data besides git itself

[Dominik1999]

as discussed yesterday, should we reopen?