Repository navigation
Replies: 9 comments 4 replies
|
Thanks @rk042. I want to share a result that surprised me, and ask for ideas. What I ran (#32): the same 100 questions (TriviaQA, TruthfulQA-MC, GSM8K) through a plain model call and BOOTH, on What happened: BOOTH scored 42.0% against 82.0% for the plain call. The code worked (0 parse or call failures), so I added two arms to find out why:
Most of the gap comes from BOOTH's prompt, not the gate. "Output ONLY the JSON object, nothing else" leaves a 7B model no room to reason (about 36 tokens per GSM8K answer, against 261 plain). The gate then cost about 8 more points, mostly TruthfulQA questions flagged AMBIGUOUS. This is one model and 100 questions, so it's a lead, not a verdict. Full write-up: Ways we could improve it (nothing decided, and nothing in core until others have replicated):
If you have a design of your own, please share it. Replications on other models are the most useful thing right now: |
Design ideas for fixing the prompt problem (#33), feedback wantedContext: in #32/#33 BOOTH's default prompt cut a 7B model's GSM8K accuracy from 97.5% to 35.0%, because the suffix tells it to output only JSON. Letting it reason first brought GSM8K back to 92.5%, but didn't help TriviaQA or TruthfulQA, and was ~1.8x slower than a plain call. This is one model and 100 questions, so everything below is a hypothesis to test, not a plan. Here are a few directions, from smallest to biggest. Please pick holes in them. 1. Fix the contradiction in the suffix (wording only). 2. A 3. Separate answering from checking (two calls). 4. Gate on something other than self-reported confidence. 5. Make AMBIGUOUS a flag, not a rejection. What each one needs before we decide:
How to help: run |
Proposal: v0.5.9 — Parser Improvements, Regression Tests, and DocumentationHi everyone! Based on the recent discussions in #32 and #33, I'd like to propose a small set of improvements for BOOTH v0.5.9. The goal is to address some of the issues identified during benchmarking while keeping this release focused on small, backward-compatible improvements without introducing new public API changes. Proposed changes for v0.5.91. Improve JSON parsing Update This should make parsing more reliable when models provide reasoning, intermediate JSON drafts, or additional text before their final answer. 2. Add regression tests Add tests covering:
3. Update documentation Update the following:
4. Evaluate default prompt wording (benchmark-dependent) I'd also like to investigate whether softening the final instruction in However, I don't think we should change the default prompt without sufficient evidence. Before considering this change for v0.5.9, I'd like to see results across multiple models, with particular attention to GSM8K, TriviaQA, TruthfulQA, and latency. If the results aren't convincing, we'll keep the existing default prompt. What about v0.6.0?I'd prefer to keep v0.6.0 for more substantial improvements that need additional experimentation and benchmarking. Some possibilities include:
These ideas need more benchmarking and contributor feedback before we decide on their implementation. Feedback wanted!Before I start implementing the v0.5.9 changes, I'd really appreciate feedback from contributors. A few questions:
I'm happy to discuss alternatives and adjust the proposal based on feedback. My intention is to keep v0.5.9 focused, reliable, and backward-compatible, while using further benchmarking to guide the direction of v0.6.0. Thanks everyone! Looking forward to your thoughts and suggestions. 🚀 |
|
Hey @Vedantgitbot Sorry, I was on holiday for the last few days, but I’m back now. Thanks for sharing the proposal. I like keeping v0.5.9 focused on parser behavior, regression tests, and documentation, while gathering results from more models before changing the default prompt. For the parser, could we clarify what should happen when the final JSON object is unrelated, malformed, or missing required fields? The current parser tries several candidates and returns the first one that passes validation. Preferring a later object could therefore change existing results, so it would be useful to agree on those cases and add regression tests. I’d be happy to help with that. For v0.6.0, a caller-supplied suffix sounds useful as an opt-in. One detail is that the parse-failure retry also contains its own JSON-only wording, so we would need to check the complete initial and retry prompts for conflicting instructions. For agreement, our held-out experiment caught two wrong first answers but also blocked one correct answer among eight eligible questions. It’s a small result, so I agree that more comparisons against a plain model baseline would be useful before adding a new public API. |
|
Thanks @rk042, welcome back, and good questions. I tested the parser change against the 0.5.8 code on 13 edge cases. Results:
Your point about silent fallback is a real policy question. If the final object is broken, should BOOTH quietly use an earlier draft, or treat it as a parse failure and retry? I'd keep today's behaviour for 0.5.9 and discuss the stricter version separately, since it would change existing results. Would you like to write or review the regression tests? I'd include the cases above plus yours. For v0.6.0: agreed, we'll check the first prompt and all three retry prompts for conflicting instructions, including the "no commentary before or after" line in the parse-failure retry. Also, a second-model run from @camerontjs-dot on gemma3:12b (see #33) didn't show the same prompt effect as Qwen, and the reasoning arm wasn't clearly better, so the default wording stays unchanged in 0.5.9. Thanks for the agreement result too. Two wrong first answers caught and one correct answer blocked is small, but useful for the comparison. |
|
@camerontjs-dot, since your Gemma run is part of why the plan changed, I'd like your view too, if you have time:
No pressure on any of it, and thanks again for the careful run. |
|
v0.5.9 is on PyPI (pip install -U boothpy). This is a small parser release: when a reply contains several JSON objects, BOOTH now uses the last valid top-level one, and nested objects are no longer treated as candidates. The default prompt is unchanged. Details are in the changelog. Thank you @rk042 for the parser regression tests (#35) and for the nested-object questions that shaped the rule, and @camerontjs-dot for the Gemma replication, which showed that the token collapse I saw on Qwen 7B does not appear on every model. Also new: a troubleshooting entry for accuracy on reasoning-heavy tasks, with a workaround, and the benchmark example in examples/Vedantgitbot_latency_accuracy. Anything that changes the default prompt or the API (a caller-supplied suffix, the AMBIGUOUS gate, on_ambiguous) is planned for 0.6.0, after more benchmarking on more models. If you can run the benchmark on a model you use, results are very welcome, including bad ones. |
|
Hey @Vedantgitbot, I wanted to suggest an improvement to our current workflow. Can we consider switching to GitHub Projects and Milestones for better task tracking and project management? I've noticed that several issues remain open for a long time. While we're making progress, it's difficult to track what's actually completed, what's still pending, and what our priorities are. I suggest we introduce a more structured workflow with:
This would give contributors better visibility into priorities, responsibilities, and overall project progress. For reference, we could look at how Godot organizes its work: https://github.com/godotengine/godot/projects What do you think? I believe this could help us collaborate more effectively and maintain a clearer development roadmap. |
|
Thanks @rk042, agreed on the goal: it’s hard to see what’s done, pending and next. Godot’s setup is built for thousands of issues and many teams, so I’d start smaller for BOOTH: Milestones: v0.6.0 (the next release, after 0.5.9) and Later (ideas not scheduled yet). Every open issue goes into one. Docs and example work that lands on main ships with 0.6.0. I’d revisit a Projects board once there are enough open issues to need one. Since you’ve worked on most of these, could you suggest which open issues belong in v0.6.0 and which in Later? |
Uh oh!
There was an error while loading. Please reload this page.
General Discussion
Welcome! This is our space to discuss product releases, bug fixes, improvements, new ideas, and development plans.
Feel free to ask questions, share feedback, suggest improvements, or discuss anything related to the project.
For specific bugs or tasks, please create a separate GitHub Issue.
Let's collaborate and make the project better together!
All reactions