[Bug] examples/multi_agent can crash with NoneType in GRPO advantage computation because old logprobs are missing

[yy5h4n]

Bug Description

The default examples/multi_agent rollout can produce training batches without any old-policy log probabilities. In this case GRPO advantage computation may crash with:

TypeError: 'NoneType' object is not iterable

The root cause appears to be that examples/multi_agent/agent_system.py requests SGLang generation with return_logprob=True, but only extracts generated token ids from output["meta_info"]["output_token_logprobs"]. The corresponding token log probabilities are discarded and never written to Sample.rollout_log_probs.

Because of that, the training batch has no rollout_log_probs. Under the default GRPO path, if the training engine also skips recomputing actor logprobs and there is no critic, all fallback sources can be None:

log_probs = None
rollout_log_probs = None
values = None

Then compute_advantages_and_returns() reaches:

xs = log_probs or rollout_log_probs or values
kl = [torch.zeros_like(x, dtype=torch.float32, device=x.device) for x in xs]

and crashes because xs is None.

Steps to Reproduce

Run the multi-agent example with its custom rollout function, for example:

bash examples/multi_agent/run-qwen3-30B-A3B-multi-agent.sh

Expected Behavior

The multi-agent custom rollout should preserve token-level rollout log probabilities whenever it requests return_logprob=True, matching the default SGLang rollout implementation.

The default SGLang rollout already does this:

# slime/rollout/sglang_rollout.py
new_response_tokens = [item[1] for item in output["meta_info"]["output_token_logprobs"]]
new_response_log_probs = [item[0] for item in output["meta_info"]["output_token_logprobs"]]
...
sample.rollout_log_probs += new_response_log_probs

The multi-agent rollout should do the same.

Actual Behavior

The multi-agent code currently does:

# examples/multi_agent/agent_system.py
payload = {"input_ids": prompt_token_ids, "sampling_params": current_sampling_params, "return_logprob": True}

if "output_token_logprobs" in output["meta_info"]:
    new_response_tokens = [item[1] for item in output["meta_info"]["output_token_logprobs"]]
else:
    new_response_tokens = []

sample.tokens = sample.tokens + new_response_tokens
sample.response_length += len(new_response_tokens)

It never extracts item[0] or sets sample.rollout_log_probs.

Environment

• slime version: source checkout, commit 09c70450
• Python version: 3.12.3
• PyTorch version: 2.9.1+cu128
• CUDA version: CUDA 12.8 runtime
• GPU: NVIDIA A800-SXM4-80GB
• SGLang version: 0.5.12.post1

Logs

Additional Context

In examples/multi_agent/agent_system.py, extract and store log probabilities alongside token ids:

if "output_token_logprobs" in output["meta_info"]:
    new_response_tokens = [item[1] for item in output["meta_info"]["output_token_logprobs"]]
    new_response_log_probs = [item[0] for item in output["meta_info"]["output_token_logprobs"]]
else:
    new_response_tokens, new_response_log_probs = [], []

sample.tokens = sample.tokens + new_response_tokens
sample.response_length += len(new_response_tokens)

if sample.rollout_log_probs is None:
    sample.rollout_log_probs = []
sample.rollout_log_probs += new_response_log_probs

assert len(sample.rollout_log_probs) == sample.response_length

Then the multi-agent script can use:

--use-rollout-logprobs

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and my issue is not addressed there.
• I have searched for existing issues and this is not a duplicate.
• I have provided a minimal, reproducible example.