Skip to content

Stoplight 6 πŸš€

Latest

Choose a tag to compare

@bolshakov bolshakov released this 02 Oct 09:42
· 7 commits to main since this release
91d51b8

Stoplight 6.0 is built around a single idea: a light is a named thing your application declares once, not an object
you rebuild at every call site.
Almost everything else in this release follows from that β€” a real registry, an
authoritative inventory for the Admin dashboard, a telemetry bus, and a Redis layer that finally agrees with itself
about what time it is.

The most visible consequence is speed. The way most applications call Stoplight β€” Stoplight("name").run { ... } β€”
is 5.6x faster with the in-memory data store and 2.2x faster with Redis, and a Redis sliding window now costs the
same at 10 requests per second as at 10,000.

This is a major release with breaking changes. They are listed at the end of these notes, and UPGRADING.md
walks through every one of them, roughly ordered from "needs some thought" to "find and replace". For most applications
the upgrade is smaller than the list suggests.


Faster by default

Most applications use Stoplight the same way, and have since the beginning:

Stoplight("Payment Service").run { payment_gateway.charge(order) }

We benchmarked that pattern in three configurations, on the previous release and on 6.0, with both data stores. Each
row is the same code called in a tight loop with benchmark-ips, reported as the median of five runs next to the
number of Ruby objects allocated per call. Every light is green and has no telemetry subscribers. Timings came from an
Apple M1 Max on Ruby 3.2.4 with a local Redis. Absolute numbers vary by machine; the ratios are the stable part.

Per call, in-memory data store

Call 5.8.3 6.0 Change
Stoplight("name").run { } 20.9 Β΅s, 65 allocations 3.7 Β΅s, 6 allocations 5.6x faster, 91% fewer allocations
Stoplight("name", window_size: 300).run { } 21.6 Β΅s, 66 allocations 6.9 Β΅s, 17 allocations 3.1x faster, 74% fewer allocations
Stoplight("name", window_size: 300, traffic_control: :error_rate).run { } 22.1 Β΅s, 67 allocations 7.5 Β΅s, 18 allocations 3.0x faster, 73% fewer allocations

Passing settings to Stoplight() costs a little more than passing none, because 6.0 checks them against the
registered configuration on every call. Registering at boot and calling Stoplight.light("name") at the call site
is the cheapest path of all: it is the same lookup as the first row, whatever the light's settings.

Per call, Redis data store

Call 5.8.3 6.0 Change
Stoplight("name").run { } 765 Β΅s, 774 allocations 349 Β΅s, 71 allocations 2.2x faster, 91% fewer allocations
Stoplight("name", window_size: 300).run { } 777 Β΅s, 780 allocations 357 Β΅s, 86 allocations 2.2x faster, 89% fewer allocations
Stoplight("name", window_size: 300, traffic_control: :error_rate).run { } 776 Β΅s, 781 allocations 359 Β΅s, 87 allocations 2.2x faster, 89% fewer allocations

Every #run makes the same number of Redis round trips while the circuit is closed as before: one to read the state
and one to record the result.

Memory

A sliding window in Redis no longer grows with traffic. This is a light with window_size: 300 after five seconds
of load, measured as the bytes Redis holds for it:

Requests in the window 5.8.3 6.0
500 60 KB 1.2 KB
5,000 667 KB 1.2 KB
18,500 2.4 MB 1.2 KB

5.8.3 keeps roughly 130 bytes per request until the window slides past it. 6.0 keeps one small entry per second, so
a window costs the same at 10 requests per second as at 10,000. The in-memory store already bucketed by the second,
and its window footprint is unchanged.

A registered light costs more at rest than it used to:

Per registered light 5.8.3 6.0
In-memory data store, heap per light 1.8 KB 3.4 KB
Redis data store, bytes per light 200 B 640 B
Redis data store, 10,000 lights 2.7 MB 7.2 MB

Traffic does not grow memory in either version. One default light run at 100 to 100,000 requests per second for
three seconds showed no growth in the process or in Redis.

The sections below explain where each of these numbers comes from.


Register once, look up by name

In 5.x, every Stoplight("Payment Service", threshold: 5) call built a brand-new light. Nothing tied the call sites
together, so two files could configure the same name differently and both would quietly work β€” which settings a
process ended up with depended on which file it happened to execute first.

6.0 makes registration explicit. Declare your lights at boot:

# config/initializers/stoplight.rb
Stoplight.register("Payment Service", threshold: 5, cool_off_time: 60)
Stoplight.register("Search", threshold: 10, window_size: 300)

Then look them up by name, with no settings list to keep in sync:

Stoplight.light("Payment Service").run { payment_gateway.charge(order) }

Stoplight.light takes a name and nothing else. If that name was never registered it raises
Stoplight::Error::UnregisteredLightError instead of silently creating a light with default settings β€” a typo in a
light name is now a loud failure rather than a circuit breaker that never trips the way you expected.

Stoplight() still works everywhere it used to. It registers on first call and returns the cached instance
afterwards. What it no longer does is let two call sites configure the same name differently β€” that now raises. The
error message names the file and line where the light was first registered, and its backtrace starts at the
conflicting call in your own code rather than inside Stoplight, so the disagreement is obvious. Dynamic names
(Stoplight("api-#{endpoint}")) still work: caching is per name, so each distinct name registers once.

This is where the per-call numbers above come from. Each Stoplight("name") call used to build a light from scratch:
merge the configuration, construct the storage wrappers, wire the notifiers, and with Redis, load the Lua script for
the new instance in an extra round trip. Everything was discarded when the block returned. Now the call is a lookup
in the registry, and when you pass settings, a digest of them compared against the registered ones.

The registry also writes each light's configuration to your data store, which is what finally gives the Admin
dashboard an authoritative list of every light your application defines β€” including ones that have never run. It
is also why a registered light costs more at rest than before: 6.0 keeps each light's configuration digest and
registration site in memory to detect conflicting call sites, and a registry entry in Redis for the dashboard.

β€” @bolshakov (#681, #733, #734, #737, #739, #811, #826, #828, #928, #931)


Telemetry: subscribe to what your circuit breakers are doing

Stoplight has always had notifiers, but they only fired on state transitions and only told you the color changed.
6.0 adds a proper in-process event bus. Every light publishes what it does β€” runs, trips, recovery attempts, manual
locks, registration β€” and any part of your application can subscribe without touching the light's own code.

Stoplight.telemetry.subscribe(Stoplight::Telemetry::TrafficBreached) do |envelope|
  logger.warn("#{envelope.light_name} tripped: #{envelope.payload.failure&.exception&.message}")
end

Stoplight.telemetry.subscribe(Stoplight::Telemetry::RecoverySucceeded) do |envelope|
  logger.info("#{envelope.light_name} recovered")
end

Every event arrives in an Envelope carrying system_name, light_name, occurred_at, and the typed payload.
The payloads are Data objects, so they destructure cleanly and fail loudly on a typo:

Stoplight.telemetry.subscribe(Stoplight::Telemetry::RunCompleted) do |envelope|
  run = envelope.payload
  statsd.timing("stoplight.run", run.duration_ms, tags: [
    "light:#{envelope.light_name}",
    "outcome:#{run.outcome}",
    "color:#{run.color}",
    "fallback:#{run.fallback_used}"
  ])
end

The full event set: RunCompleted, TrafficBreached, RecoveryStarted, RecoveryProbeCompleted,
RecoverySucceeded, RecoveryFailed, LockChanged, and LightRegistered. Subscribe with no filter for the
firehose, with a specific class for one event, or with a module for a family of them.

Two properties matter for production use. Event payloads are built lazily β€” if nothing is subscribed to
RunCompleted, nothing is allocated for it, so the bus costs you nothing when you aren't using it. And a raising
subscriber cannot break your application
: exceptions from handlers are routed to your error_notifier, never back
into the protected block.

The notifiers option is now implemented as a telemetry subscriber itself, so anything a notifier can do, a
subscription can do with more event types and more detail. The stoplight-statsd gem is built exactly
this way. See the Telemetry guide for the complete reference.

This is what it looks like once the events reach Grafana β€” every light's run rate by outcome, with a circuit
blocking traffic while it is red, and run duration percentiles across the fleet:

Stoplight metrics in Grafana

β€” @bolshakov (#721, #723, #724, #726, #741, #742, #854, #878, #881, #907, #909, #910, #911), @Zainulhassan01 (#852)


One dashboard, every system

Named systems β€” isolated registries with their own data store, notifiers, and configuration β€” are now a first-class
public API via Stoplight.register_system:

Payments  = Stoplight.register_system("Payments",  threshold: 3, cool_off_time: 30, data_store: payments_redis)
Analytics = Stoplight.register_system("Analytics", threshold: 5, cool_off_time: 60, data_store: analytics_redis)

Payments.register("stripe")
Analytics.register("amplitude")

Payments.light("stripe").run { charge_card }

The same circuit name in two systems is two completely independent circuits, which is what makes this work for
multi-tenancy and for separating failure domains with different SLOs.

The Admin dashboard now understands all of them. Register your systems with it and a switcher appears in the top
navigation:

Stoplight::Admin.configure do |config|
  config.add_system Payments
  config.add_system Analytics
end

mount Stoplight::Admin => "/stoplights"

Underneath, lights are addressed by stable IDs through system-scoped RESTful routes. Previously the dashboard used
light names as URL identifiers, which broke on names containing slashes or spaces and collided across systems.

There's also a read-only mode, which is what you want if you're exposing the dashboard more widely than your
on-call rotation:

Stoplight::Admin.configure do |config|
  config.add_system Payments
  config.read_only = true
end

Everything stays visible; lock, unlock, and remove are hidden and disabled, and any request that would change state
returns 403. For the pre-built Docker image, set STOPLIGHT_ADMIN_READ_ONLY=true.

β€” @bolshakov (#712, #719, #803, #808, #809, #859, #861, #863, #864, #867, #868, #869, #912)


Redis tells the time now

Stoplight 5.0 shipped clock skew detection: a probabilistic check that warned you when your application servers and
Redis disagreed about the current time. It was a useful diagnostic, but it only told you that you had a problem.

6.0 removes the problem. Every time-dependent decision β€” state transitions, cool-off expiry, metrics bucketing,
recovery timing β€” is now made from Redis's own clock, read inside the same Lua script that performs the write.
Every instance reads the same time no matter what its host believes, so a drifting server can no longer trip a
circuit early, hold one red past its cool-off, or corrupt a sliding window.

It is also cheaper. Every write used to carry a timestamp computed in Ruby, and window writes carried a random request
id as well, both generated on every call. Now the script reads the time itself, so the client sends less and allocates
less per call. That, and the removal of the compatibility wrappers that sat between a light and its storage, is why a
run on Redis allocates fewer objects even after the cost of building the light is taken out of the picture.

Because the coordination problem is solved at the source, there's nothing left to warn about, and
warn_on_clock_skew: has been removed.

β€” @bolshakov (#917, #918, #921, #925, #926)


Sliding-window metrics that scale with the window, not with traffic

5.8.3 stored one sorted-set member in Redis per request and counted them on every read, so a busy light carried a large
and growing set. 6.0 keeps per-second success and failure counters, running totals, and an index of bucket timestamps,
and evicts expired buckets in batches inside the same Lua script that records the result.

That is where the flat 1.2 KB column above comes from: memory is a function of your configured window size, not
of how much traffic flows through the circuit. Writes stay amortized O(1), and the old keys expire naturally after you
upgrade β€” there's no migration step.

β€” @bolshakov (#904)


Features and improvements

  • Override tracked_errors or skipped_errors for a single call without changing the registered configuration:
    light.run(skipped_errors: [ActiveRecord::RecordNotFound]) { find_customer }. β€” @oiahoon (#758)
  • Faster light creation and lookup: reduced lock contention, no mutex acquisition on the read path, and optimized
    in-memory stores. β€” @bolshakov (#681, #733, #734, #737)
  • In-memory sliding windows no longer rescan for expired buckets on every write within the same second.
    β€” @bolshakov (#930)
  • Red-light errors now carry a descriptive default message identifying the stopped circuit. β€” @Lokideos (#870)
  • Admin dashboard: refined layout (#668), locally served assets instead of a CDN (#676), and application icons (#685).
    β€” @Lokideos
  • Admin dashboard: recovery progress, recovery timing, and last-check information are now shown for yellow lights.
    β€” @bolshakov (#888, #890, #895, #896)
  • A clearer empty-state message in the Admin dashboard. β€” @nebiyuelias1 (#897)
  • A new README state diagram that actually explains the transitions. β€” @Lokideos (#903)
  • Rewritten README and expanded guides for telemetry, systems, and
    admin. β€” @bolshakov (#916, #924)

Fixes

  • Deduplicate recovery notifications when another process already completed the transition. β€” @pollychen-lab (#749)
  • Per-call bookkeeping failures are no longer misclassified as failures of your protected code. β€” @snowyukitty (#836)
  • Persistent Redis NOSCRIPT responses no longer cause unbounded, backoff-free retries. β€” @arvindpz (#846)
  • A raising error_notifier can no longer escape Stoplight's fail-safe storage paths. β€” @johnsonsirv (#843)
  • Telemetry subscription filters are validated before the subscriber registry is mutated. β€” @ankuanku1 (#810)
  • Filters that cannot match any event class are rejected outright. β€” @SAY-5 (#834)
  • Fixed in-memory metrics bucket alignment at hour boundaries. β€” @ankuanku1 (#825)
  • In-memory metrics now age on a monotonic clock, so wall-clock adjustments cannot corrupt window eviction.
    β€” @OursCodeur (#873)
  • Expired in-memory metric buckets are pruned during writes, so memory stays bounded even if snapshots are never
    read. β€” @henriquejsza (#887)
  • Fixed yellow-state color calculation when clock skew placed the recovery timestamp in the future.
    β€” @nebiyuelias1 (#877)
  • Corrected and consolidated monotonic clock handling in memory sliding windows. β€” @nebiyuelias1 (#893)
  • Fixed a race when reading the last failure time from an unbounded in-memory window. β€” @bolshakov (#735)
  • Traffic-control strategy objects now compare consistently by value. β€” @mausamp (#827)
  • Failures behave consistently as hash keys and set members. β€” @johnsonsirv (#840)
  • Non-numeric error-rate thresholds are rejected with a configuration error. β€” @johnsonsirv (#804)
  • Consecutive-success and consecutive-error configuration types are validated before their ranges.
    β€” @johnsonsirv (#805, #806)
  • Invalid error-rate minimum-request settings are validated consistently during compatibility checks.
    β€” @johnsonsirv (#837)
  • Generic notifiers receive LightInfo rather than internal configuration objects. β€” @shkrt (#792)
  • The Digest dependency used to build Redis key spaces is now loaded explicitly. β€” @shkrt (#789)
  • Corrected the generic notifier documentation example. β€” @johnsonsirv (#790)
  • Corrected tracker RBS signatures that referenced a nonexistent superclass. β€” @johnsonsirv (#801)

Breaking changes

This is the price of the above. Each item links to its section in UPGRADING.md, which has the
before/after code and the reasoning. Most of them are a find and replace; the first two need a look at how your
application configures its lights.


Thank you

This release had more new contributors than any before it. Thank you to @Lokideos, @johnsonsirv, @nebiyuelias1,
@ankuanku1, @shkrt, @OursCodeur, @henriquejsza, @snowyukitty, @arvindpz, @pollychen-lab, @Zainulhassan01, @mausamp,
@oiahoon, and @SAY-5.

Routine dependency updates, release bookkeeping, CI configuration, and internal tooling changes are intentionally
omitted from this list.

Full Changelog: v5.8.3...v6.0.0