Repository navigation
Qwen3.5-4B + LoRA — Advice on reliable multi-turn constraint tracking and entity grounding #1930
nicolasmagny03-dot
started this conversation in
General
Replies: 1 comment
|
I work on CloudGPU. For question 2, I would test capacity before adding more supervision: run your 11 synthetic conversations unchanged against a larger model with no LoRA, with the same two passes and validators. If it still returns an empty constraint list for "Ne déplacez aucun de mes deux rendez-vous", the limit is more likely the schema or missing context than model size. If it passes, you know what the LoRA has to close on the 4B. I haven't tried this on your task. For the identical-label case, pass the record IDs into extraction B so the target is never inferred. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Hi Qwen team and community,
I'm building a French-language B2B assistant using Qwen3.5-4B with LoRA, running locally on an RTX 2060 SUPER (8 GB VRAM).
The assistant must understand professional requests, track conditions and prohibitions across multiple turns, identify the correct appointment or business record, and distinguish binding instructions from preferences, uncertainty, and quoted speech.
Current architecture
We use two Qwen-based semantic extraction passes:
Their outputs feed deterministic memory, entity-resolution, and permission checks. No real-world operations are executed automatically during these experiments.
What we're struggling with
After multiple LoRA and evaluation campaigns, we still observe:
For example, a client may say:
"Ne déplacez aucun de mes deux rendez-vous."
The model sometimes returns an empty list of constraints, even though this restriction should apply to both appointments.
We've also demonstrated cases where two appointments have identical labels but belong to different business records. When those relationships are absent from the model's input, the correct target cannot be determined reliably.
On one recent evaluation of 11 new multi-turn conversations, our baseline completed 0/11 conversations fully correctly, with only 1/12 legitimate requests accepted by an experimental alternative. We therefore do not consider the system production-ready.
We'd appreciate advice on three questions
Architecture: Would you recommend a different extraction or state-tracking architecture for this kind of multi-turn task, particularly one that separates language understanding from deterministic business state?
Fine-tuning: Is Qwen3.5-4B with LoRA a realistic choice for reliably handling these complex relationships, or should we benchmark a larger model before investing further in supervision?
Grounding and evaluation: What approaches would you recommend for preserving entity identity, evidence provenance, and binding constraints without excessive false refusals?
We're not looking for a simple prompt adjustment. We're trying to identify whether our main limitation is model capacity, training methodology, missing context, or the architecture itself.
We can provide a small fully synthetic, reproducible evaluation set with expected outputs and failure traces if that would help.
Any technical guidance or references to similar projects would be greatly appreciated.
Thank you!
All reactions