Feature/query engine improvements #713
Conversation
collindutter
force-pushed
the
feature/query-engine-improvements
branch
3 times, most recently
from
0b905ee
to
6bb2735
Compare
collindutter
force-pushed
the
feature/query-engine-improvements
branch
from
6bb2735
to
b49cad3
Compare
|
|
||
| for artifact in artifacts: | ||
| text_segments.append(artifact.value) | ||
|
|
||
| message = self.template_generator.render( | ||
| user_message = self.user_template_generator.render( | ||
| metadata=metadata, | ||
| query=query, | ||
| text_segments=text_segments, | ||
| rulesets=J2("rulesets/rulesets.j2").render(rulesets=rulesets), |
There was a problem hiding this comment.
Should remove rulesets from here.
|
|
||
| for artifact in artifacts: | ||
| text_segments.append(artifact.value) | ||
|
|
||
| message = self.template_generator.render( | ||
| user_message = self.user_template_generator.render( | ||
| metadata=metadata, |
There was a problem hiding this comment.
Metadata should be in the system message.
|
|
||
| for artifact in artifacts: | ||
| text_segments.append(artifact.value) | ||
|
|
||
| message = self.template_generator.render( | ||
| user_message = self.user_template_generator.render( | ||
| metadata=metadata, | ||
| query=query, | ||
| text_segments=text_segments, |
There was a problem hiding this comment.
Segments can be moved into the system message as well, so the LLM is more "grounded."
collindutter
force-pushed
the
feature/query-engine-improvements
branch
3 times, most recently
from
fec940e
to
fa11f43
Compare
collindutter
force-pushed
the
feature/query-engine-improvements
branch
from
fa11f43
to
de2dc6b
Compare
collindutter
force-pushed
the
feature/query-engine-improvements
branch
from
3a04f78
to
b78d009
Compare
There was a problem hiding this comment.
Across my own testing, I'm seeing a significant improvement.
For my own 106 test cases, I see 46 are just as good, 25 are just as bad, 31 that were bad are now good, and only 2 that were good are now bad. (Bad just means failing to provide an answer, saying something like "I don't know").
One side observation about a qualitative difference in the answers that the prompt now mentions the "context information" pretty often.
For example:
Before: "I could not find an answer."
After: "The context information does not provide any information about XYZ".
Before: "There are 23 things."
After: "There are 23 things listed in the provided context information."
As most things LLM, it doesn't always mention "context information" all the time, but way more often then before (which was basically never).
Approving because aside from mentioning "context information" subjectively too much, the retrieval seemed to improved quite a bit.
| text_segments=text_segments, | ||
| rulesets=J2("rulesets/rulesets.j2").render(rulesets=rulesets), | ||
| ) | ||
| user_message = self.user_template_generator.render(query=query, text_segments=text_segments) |
There was a problem hiding this comment.
Don't need to pass text_segments here.
No description provided.