Q&A: PrioritisedReplayBuffer config — capacity, alpha, beta, update_priorities

[web3guru888]

Q&A: Configuring and tuning PrioritisedReplayBuffer

Common questions about the Phase 7.3 replay buffer — feel free to add more.

---

Q1: What values should I use for capacity?

Agent type	Recommended capacity
Toy / test environment	1,000 – 10,000
Standard RL benchmark	50,000 – 100,000
Large observation spaces	10,000 – 50,000 (memory-limited)
Multi-agent (shared buffer)	200,000+

A good starting point is capacity = 50_000. Monitor asi_replay_buffer_size / asi_replay_buffer_capacity — if the buffer fills in < 1 epoch, increase capacity.

---

Q2: What is alpha and how does it affect training?

alpha controls prioritisation strength:

alpha	Effect
0.0	Fully uniform sampling (equivalent to UniformReplayBuffer)
0.4	Mild prioritisation — default for most benchmarks
0.6	Standard PER (Schaul et al. 2016 recommendation)
1.0	Fully greedy — only highest-priority episodes sampled

Start with alpha=0.6. If training is unstable, reduce to 0.4.

---

Q3: What is beta_start and does it need tuning?

beta_start is the initial IS correction strength (0 = no correction, 1 = full correction). The standard recommendation (Schaul et al.) is beta_start=0.4 annealed to 1.0.

You rarely need to change beta_start. However, set beta_anneal_steps to match your expected total training steps so β reaches 1.0 at the end of training:

buffer = PrioritisedReplayBuffer(
    capacity=50_000,
    alpha=0.6,
    beta_start=0.4,
    beta_anneal_steps=1_000_000,   # match total env steps
)

---

Q4: When should I use UniformReplayBuffer instead?

Use UniformReplayBuffer when:

• Running ablation studies to isolate the effect of prioritisation
• Debugging: uniform sampling is deterministic and easier to reason about
• Very small capacity (< 1,000): prioritisation overhead isn't worth it
• Your environment has near-uniform TD errors (little benefit from prioritisation)

Use build_replay_buffer(strategy="uniform") to switch without code changes.

---

Q5: How does update_priorities() work and when should I call it?

After a training step, compute new TD-errors (or losses) for the replayed episodes and call:

await buffer.update_priorities(
    task_ids=["task_a", "task_b"],       # episodes to update
    new_priorities=np.array([0.8, 0.3]), # new |δ| values
)

Call this after every training step where you used replay samples. Stale priorities cause the buffer to over-sample old hard episodes. If you can't compute TD-errors cheaply, use loss values as a proxy.

---

Q6: Can PrioritisedReplayBuffer be shared across multiple agents?

Not directly — the current implementation is in-process. For multi-agent sharing:

1. Snapshot the buffer periodically: snap = await buffer.snapshot()
2. Serialise with pickle or msgpack and store in Redis
3. Other agents call await buffer.restore(snap) to load

A shared Redis-backed buffer backend is planned but not in Phase 7.3 scope.

---

Q7: How do I monitor replay health via Prometheus?

# Is the buffer filling up?
asi_replay_buffer_size / asi_replay_buffer_capacity

# How fast are episodes being added vs sampled?
rate(asi_replay_episodes_added_total[1m]) /
rate(asi_replay_episodes_sampled_total[1m])

# Is mean priority rising? (sign of increasing difficulty)
deriv(asi_replay_mean_priority[10m])

A rising mean priority indicates the agent is encountering harder episodes — expected during curriculum advancement. A falling mean priority suggests the agent is mastering tasks.