Arc Lamp helps gather stack traces from a running PHP 7 application and publish them in the form of flame graphs and trace logs. See performance.wikimedia.org for a live example.
- Your PHP application, with php-excimer and php-redis.
- A Redis server.
- Python, with python-redis and python-yaml.
ifnecommand from moreutils.
require_once 'ArcLamp.php'; Wikimedia\ArcLamp::collect( [ 'redis-host' => '127.0.0.1' ] );
To automatically enable this for all web requests and entry points, you can use the PHP
option. See Wikimedia's own configuration for example.
How it works
The Arc Lamp pipeline comprises of three stages:
- Capture stack traces.
- Create trace logs.
- Generate flame graphs.
Capture stack traces
The php-excimer extension is used to periodically collect a backtrace. It has no run-time overhead by default. When enabled from a web request, it periodically schedules a graceful interrupt at which point it captures a backtrace.
These traces can be collected from a PHP callback and dispatched to a socket or file as-needed.
The default in
ArcLamp.php is to send the trace to a Redis pubsub channel.
Arc Lamp was originally created in 2014 for Xenon, a sampling profiler native to the HHVM engine for PHP. To use Arc Lamp with HHVM Xenon, see arc-lamp v1.0 instead.
Create trace logs
In Wikimedia's production cluster, Arc Lamp's Redis server resides in the high availability "Tier 1" zone, and thus kept separate from offline performance, research, and statistics services ("Tier 2").
Each of the production web servers uses the
ArcLamp::collect client in PHP to send traces to Redis.
arclamp-log service subscribes to the Redis pubsub channel, normalizes the stack traces and
write them to the relevant trace log files. The example configuration creates
a trace log file for each hour and each day.
Within those two time periods, it segregates the trace logs by entry point of the PHP application.
For example, the MediaWiki application has
rest.php (web) and
entry points. This results in the following trace logs:
arclamp-log service is also responsible for pruning trace logs older than the configured
Generate flame graphs
arclamp-generate-svgs script runs at a regular interval and creates or updates the flame graph
associated with each trace log file. It also maintains a reverse version of each flame graph.
arclamp-generate-svgs script also removes graphs for which a trace log no longer exists.
Flamegraphs are generated using Brendan Gregg's flamegraph.pl.
See performance.wikimedia.org for a live example.