Skip to content

Guardrails: Topic relevance evaluation #102

@AkhileshNegi

Description

@AkhileshNegi

Is your feature request related to a problem?
We need to ensure that the assistant provides relevant responses in SNEHA's program domains and avoids off-topic queries. This is crucial for maintaining the quality of interactions in real-world scenarios.

Describe the solution you'd like

  • Run the topic relevance guardrail from kaapi-guardrails on a randomly sampled subset of ~14,605 conversations of SNEHA.
  • Start with 100 conversations to refine the prompt, then 500, and finally ~1.5k (~10% sampling).
  • Document results since we don't yet have annotations or a known on-topic/off-topic ratio to stratify on.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions