🗺️ Human / Programatic Feedback

[mikeldking]

Add the ability to add human feedback via an API call

In many applications, but even more so for LLM applications, it is important to collect user feedback to understand how your application is performing in real-world scenarios. The ability to observe user feedback along with trace data can be very powerful to drill down into the most interesting datapoints, then send those datapoints for further review, automatic evaluation, or even datasets.

Phoenix should make it easy to attach user feedback to traces. It's often helpful to expose a simple mechanism (such as a thumbs-up, thumbs-down button) to collect user feedback for your application responses. The phoenix SDK or API should support sending feedback.

Use-Cases

• Creating an LLM judge to align with human feedback on a dataset

Milestone 1

• As a user of the phoenix UI I can add / edit / view / delete human annotations
• As a user of phoenix I can use annotations to drive dataset curation
• As a developer I can send annotations on traces and spans

client

• [client] move the client (sdk) into its own sub-module #2518
• [client] add streaming single log endpoint for human feedback

REST

• [feedback] REST endpoint to set a trace level annotation #3826
• [feedback] REST endpoint to set span level annotation #3827
• [feedback] in-memory backup queue processing of annotations #3828

GraphQL

• [feedback] createSpanAnnotation mutation #3769
• [feedback] updateSpanAnnotation mutation #3770
• [feedback] deleteSpanAnnotation mutation #3824
• [feedback] filter on Annotations using DSL #3815
• [annotations][graphql] annotation names resolver on project #3830
• [annotations][graphql] annotation summary #3829
• [gql][annotations] resolve trace annotations off of span
• [gql][annotations] resolve span annotations off of a span #3913
• [annotations][gql] annotation sort dir parameter #3962
• [annotations][gql] allow the user to query after the mutation for cache invalidation #3963
• [annotations][gql] contained_in_dataset boolean #3992
• [gql][annotations] filter by spanKind on annotations #4028

Datasets

• [annotations] trace / span annotations to datasets should be added to dataset examples #3912

UI

• [ui] ability to copy / paste span / trace ids from slideover #3894
• [feedback] [UI] Add the ability to add annotations to a span #3825
• [feedback] [UI] Add the ability to add annotations to a trace #3831
• [ui][annotations] consolidate annotations and evaluations under a single tab #3995
• [ui][annotations] ability to update human annotations via slide-over #3996
• [ui][annotations] typeahead support for annotations #3997
• [ui][annotations] color code annotations in the table #3998
• [ui][annotations] unify annotations and evaluations under a single "feedback" column #3999
• [annotations][ui] condensed trace tree #4027
• [annotations][ui] extend span filter DSL magic to include annotations #4000
• [annotations][ui] add annotations to the quick filter dropdown #4001
• [annotations][UI] place annotations and other info in a slide over on span info #4029
• [annotations][UI] display in the table whether or not a span has ben added to a dataset #4030
• [annotations][UI] show annotations under a feedback column #4074
• [annotations][UI] show annotations under the span details, make refetchable
• [annotations][UI] lazily load annotation explanations in tooltip #4073
• [annotations][UI] Edit human annotations and save #4076
• [annotations][UI] make the explanation collapsed by default #4075
• [annotations][ui] column selector for annotations
• [annotations][ui] annotation summaries in the header
• [annotations][ui] color code metrics in the header using the alphabetic color #4077
• [annotations][ui] make document evaluations have the same style as annotations #4078
• [annotations][ui] call to actions in the feedback tables on how to add annotations #4142
• [ui] add the ability to hide metrics in the trace tree #4145

Documentation

• [annotations] full-stack example of feedback (thumbs-up) #3892
• [annotations] [docs] feedback / annotations overview docs #4134
• [annotations][ts] typescript full-stack example of how to add annotations #4140
• [annotations][docs] video on annotations #4156
• [annotations][docs] video on human feedback #4157
• [annotations][docs] python notebook tutorial on how to send feedback #4155
• [annotations][documentation] langchain on how to send feedback #4158
• [docs][annotations] annotations notebook and nbconvert #4197

Client

• [annotations] client method to post span annotation #4178

Cleanup

• cleanup and remove evaluation resolvers #4180

Readings

• Human Feedback for RLHF (https://arxiv.org/pdf/2404.13895)

[dosubot]

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[stdweird]

@mikeldking besides human thumbs up/down, there is also the other cases like "normal" ragas usage, where you (re)run same set of questions with grounds thruths as a baseline, and then want to add the evaluation scores to the trace.

the main issue is that there is no way that eg openinference callback communicates the span_id or trace_id back to the framework client in anyway (in our case llama-index). it guess the framework should provide a hook where eg the callback can set the id, and then somehow client code can retrieve it.
i can guess the id based on the response text and the timing, but that ivolves getting dataframes back from phoenix, inspecting them with sole purpose to find the correct id.

once the id is found, afaik, i only have to construct a single row evaluation dataframe and then do a log_evaluations

[mikeldking]

@stdweird yes exactly. We can definitely make an ID be available in the application so that subsequent feedback and evaluations can be programmatically logged to the phoenix server.

From a roadmap perspective we do need to tackle two key things - #2340 and persistence so that we can make phoenix scale forward. We will be unblocked to work on this after.

Thanks for your insight. appreciate it!

[mikeldking]

Closing for now as MVP is complete