Skip to content

Conversation

@ssbushi
Copy link
Contributor

@ssbushi ssbushi commented Feb 18, 2025

Checklist (if applicable):

@github-actions github-actions bot added the docs Improvements or additions to documentation label Feb 18, 2025
@ssbushi ssbushi requested review from pavelgj and thedmail February 18, 2025 15:46
harm, or exploit
For more details on how each metric is computed, please refer to the
[RAGAS metrics documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/){: .external}.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at the page, there are many more metrics listed there... do we want to link to specific metrics instead?
Ex: https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought of that, but RAGAS change their metrics frequently, I was afraid we might have to keep up with them so I used the parent page instead.

I can switch to using specific metrics if you feel that's preferred.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if they change the metrics frequently then it means that whatever we have may not be up-to-date, right? what's the point of linking to their docs if it may or may not be accurate?

Copy link
Contributor Author

@ssbushi ssbushi Feb 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we should link is because they have detailed explanation on how a metric is computed. Some of the feedback we received for evals is a scenario like: 'user expects relevancy to be 1.0 but it is only 0.5, why is that?'

The reason I am hesitant to link to individual docs for each metric is because RAGAS seem to change their metrics collection (either removing them or renaming them). E.g.: ANSWER_RELEVANCY in genkitEvals is now called "Response Relevancy" in RAGAS.

Catching up to RAGAS is a different problem altogether.

You are right though... taking them to a top level directory does not guarantee that the metric details would be available. The way I see it, both approaches have their cons.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, should we just bring the explanations of the metrics into our docs so we don't need to link our to RAGAS docs?

I believe our metrics are already just "RAGAS-inspired", so it's okay if we deviate as long as we explain what they mean well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense to me... but there's going to be more metrics in this plugin and we should move that into it's own documentation?

How about including a summary in the JSDoc for these metrics and linking to the JS API ref?

@ssbushi ssbushi removed the request for review from thedmail February 22, 2025 02:50
@ssbushi ssbushi closed this Apr 28, 2025
@ssbushi ssbushi deleted the ssbushi-patch-1 branch June 2, 2025 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Improvements or additions to documentation

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants