Annotation Queues and Session IDs and Datasets and LLM Security

[paolotamag]

Hello there,
I am playing a bit with Langfuse (FOSS version) to understand the flow for:

• step 1: selecting a batch of inputs (don't think chat instances, rather chains of prompts for different data points in my dataset)
• step 2: tracing on langfuse with a unique session id using the LLM Security feature
• step 3: providing a link to domain experts to label the quality
• step 4: see the outcome of the evaluation on the dashboard

I was already playing with this and it is working great.
A unique link is created from the unique session ids.
I can see how the evaluation scores available to the user can be defined to me and they appear in the order I want.

Here are my questions regarding such a system:

• can this work (unique link creation with session id) if I use Langfuse datasets (for the first step) and run an experiment (for the second step) as described here?
• can the above work and keep the data secure similarly to how the LLM Security works for traces?
• is there any plan to set up dependencies between different annotations? for example: if the first annation gives a positive outcome, no need to go to the next annotation for the same trace.
• I can see that the Annotation Queues feature was recently released for paying self-hosting customers, how is this better than the current system I have in place with unique session ids links? Is it to let multiple domain experts label the same batch without overwriting each other values?

[dosubot[bot]]

Ciao @paolotamag! I'm here to help you with your Langfuse questions. Let's dive into it!

1. Unique Link Creation with Session IDs and Datasets: Yes, you can use Langfuse datasets and run experiments while maintaining unique link creation with session IDs. Langfuse allows you to add a sessionId when creating or updating a trace, which ensures that each session can be uniquely identified and linked. This functionality is supported across different platforms and use cases, including when using datasets and running experiments ^[1][2].

2. Data Security with Datasets and Experiments: Unfortunately, I couldn't find specific information on how Langfuse ensures data security for datasets and experiments in a manner similar to the LLM Security feature for traces. You might want to explore the security documentation further or test the security features in your setup.

3. Dependencies Between Annotations: There is no available information on existing functionality or planned development in Langfuse to set up dependencies between different annotations, such as conditional annotation flows. This might be a feature to look out for in future updates.

4. Annotation Queues vs. Unique Session ID Links: The Annotation Queues feature allows you to manage and prioritize annotation tasks in a structured way, which is particularly useful for large-scale projects. It ensures concurrency safety and efficient processing of tasks, allowing multiple domain experts to annotate without overwriting each other's values. This feature is designed to be first-in-first-out (FIFO) and is fully mutable, meaning you can add or remove tasks at any time. This structured approach might offer more flexibility and control compared to using unique session ID links alone ^[3][4].

To continue talking to Dosu, mention @dosu.

[paolotamag]

some questions were not answered by your bot.
This is getting very meta lol.

New question on top:

• can i assign annotation queues to selected users that get an email? or is it again via a link I am sharing to them?

[dosubot[bot]]

Langfuse does not provide specific information on whether you can assign annotation queues to selected users who then receive an email notification. The available documentation does not mention a built-in notification system for this purpose, nor does it specify if email notifications are sent when users are assigned to annotation queues. It seems that sharing links might be the primary method for sharing annotation tasks with users.

To continue talking to Dosu, mention @dosu.

[jannikmaierhoefer]

Hi @paolotamag, you can invite human annotator to your project (which triggers and email invite) and send them the link to the annotation queue you created. Implementing email alerts for users assigned to an annotation queue is currently on our roadmap.

[paolo-tamagnini-tr]

[marcklingen]

seems like you were able to resolve the issue, that's great! check out the example datasets notebook to get started: https://langfuse.com/docs/datasets/python-cookbook