Add Structured Output Support to Policy #46
Conversation
Jack-Khuu
requested review from
allenwang28,
ebsmothers,
joecummings and
pbontrager
meta-cla
bot
added
the
CLA Signed
| sampling_params = None | ||
| if guided_decoding: | ||
| # Add config for structured output | ||
| vllm_args = await policy_actor.get_vllm_args.choose() |
There was a problem hiding this comment.
noob q: why are we running .choose here? It seems to me the more idiomatic thing to do here would be .call but maybe I'm missing the point (also I assume calling all actors is not a bottleneck here anyways as this just returns an instance variable)
There was a problem hiding this comment.
You use choose when you just want one of the actors to do something. Basically like "if rank == 0: do"
There was a problem hiding this comment.
From my understanding call also triggers the entire Mesh, while like @pbontrager mentioned choose and call_one are more direct (or in the case of no arg it just picks a random)
| sampling_params = None | ||
| if guided_decoding: | ||
| # Add config for structured output | ||
| vllm_args = await policy_actor.get_vllm_args.choose() |
There was a problem hiding this comment.
You use choose when you just want one of the actors to do something. Basically like "if rank == 0: do"
| print("Model running") | ||
|
|
||
| prompt = "Tell me a joke" | ||
| prompt = "What is 3+5?" if guided_decoding else "Tell me a joke" |
There was a problem hiding this comment.
Can you spin these things off as proper tests at some point? @ebsmothers is running something like this too big for our current test setup?
There was a problem hiding this comment.
We can throw them on the H100s and it should be fine once we get things more fleshed out
| router = await router_mesh.spawn("policy_router", PolicyRouter, policy=policy_actor) | ||
|
|
||
| sampling_params = None | ||
| if guided_decoding: |
There was a problem hiding this comment.
We need to think of how we want to do this from the config in the future. Maybe some sampling param builders that the user can call?
There was a problem hiding this comment.
I'm kinda down to just lean into the vllm convention to start (Read from generation.json to populate, then let users override the kwargs)
This is probably something we ask for upstream
Add structured output (aka guided decoding) example to policy
python src/forge/actors/policy.py(With structured output toggled)