Rewards spec design philosophy

[Darktex]

Leaving here some notes based on feedback I'm receiving for broader circulation and transparency.

The current spec is very lightweight on rewards -- to be honest, intentionally: as we wrote in RFC 0, we started elsewhere and we are going to look at rewards after we consolidate the bases on sandboxing, binary distribution, etc.

An upside of this is that we get some space to take in feedback from folks.

Here's some feedback I got from a researcher that's worth thinking about:

• High-level API makes sense: {reset, step, state}
• Reward as part of the observation may not be the best place for these reasons:
  1. What if reward calculation is super slow? Reward may not be available for a while and may come later (think a super large reward model)
  2. How do we handle scenarios where you may have multiple rewards, e.g. code style + accuracy + safety + something else

My thoughts: 2 should require a simple extension of our dataclass. Right now we do enforce rewards to be scalars.

When you are training, rewards are scalars of course, but you could indeed have multiple trainers optimize for different things. For example, you may decide to do a training step against just the code style reward, and then the next training step can optimize for accuracy. You could also just do a weighted sum of all possible rewards so that you are back to a single scalar, but 1) these two things are very likely not equivalent and 2) as providers of standards, I think we need to be neutral wrt these design choices and allow both options.

[AlekseyKorshuk]

I would love to highlight some other aspects of rewards design philosophy. In lots of agentic and cooperative environments there is a scenario where the delayed reward has to be implemented.

Current Gymnasium-like environment with step returning the reward for this specific action will not yield the best performance and flexibility.

On top of my head there are two examples:

1. Agent that write a bullet-point plan for the exact same agent to implement it -- Cursor Plan Mode
   The reward for the trace of writing the plan can consists of multiple sub-rewards:

• Response format
• Plan correctness
• Was the agent later able to implement the system based on the plan -- the main objective, since the plan could be theoretically good and human-readable (developer can understand it and approve), but not descriptive enough or not aligned with the agent itself to implement it correctly

2. The game Codenames
   Two teams are playing the game, each team has Spymaster and Operative. Spymaster give a clue and number for Operative to guess. And the Spymaster reward consists of:

• Response format
• Valid / invalid action
• How many words the Operative guessed -- the main objective is based on next moves (probably of the exact same agent, since we train the model to play both roles at the same time)

And the problem with Gymnasium-like environment is quite straightforward: ideally we want to build rewards not at every step, but based on the full episode played.

Would love to hear other on this topic as well!
cc @Darktex @jspisak @pseudo-rnd-thoughts