Support Phoenix Evals in threads when run_sync=True, support Evals in thread callbacks #3240
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a fairly small set of edits to allow Phoenix Evals to run in async io threads, in run_sync=True mode. It allows phoenix to run synchronously in async callbacks. I also added some more clear error handling so people know you can't run Phoenix Evals Asynchronously in a thread, you must be in the main.
Huge value to people who want to just run Evals as they are sending data versus manage chron jobs.
This is needed to support "Inline Evals" as part of the Span processor. The spans get sent to Phoenix in a normal fashion, but in a non-blocking way, a separate thread calculates the Eval and sends to Phoenix.
This notebook pulls the branch and is working! Note the Eval is added to the LLM span and does not show up in the Trace view:
https://colab.research.google.com/drive/14C6PQ_MdQXUtKkjY7NxKFuTqkS1xsKy1?usp=sharing
I did add the threading library for this check. The main code logic checks if we are in the "main" thread or not, and doesn't set up the signal.signal calls (needed for async) if we are running synchronously.
##Main line added ##
if threading.current_thread() is threading.main_thread():
Logical Description
Purpose:
This check is used to determine if the current code is being executed in the main thread of the application. The main thread is the thread that is initially started when the Python program begins execution. It is often where the primary control flow and main application logic run.
How It Works
threading.current_thread():
This function returns a reference to the current thread object, representing the thread in which the current code is being executed.
threading.main_thread():
This function returns a reference to the main thread object, representing the thread that was the initial entry point of the program.
is operator:
The is operator checks for object identity, meaning it checks if both references point to the same object (i.e., the same memory location).
Combined Logic
threading.current_thread() is threading.main_thread():
This condition evaluates to True if the current thread (in which the code is running) is the main thread of the application.
It evaluates to False if the current thread is any other thread (e.g., a worker thread, background thread, etc.).
------ NOTE on THREADING ------
We could use the OS library and not have to install a new library like "threading", this felt less standard but would work, let me know:
import _thread as thread # Standard library module for low-level thread operations
def is_main_thread():
# The main thread's identifier is the same as the process ID
return thread.get_ident() == os.getpid()
The notebook uses the SPANLEVEL hallucination detection template will do a separate PR to get those done but wanted to get this in to unblock.