Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Phoenix Evals in threads when run_sync=True, support Evals in thread callbacks #3240

Closed
wants to merge 1 commit into from

Conversation

jlopatec
Copy link
Collaborator

@jlopatec jlopatec commented May 18, 2024

This is a fairly small set of edits to allow Phoenix Evals to run in async io threads, in run_sync=True mode. It allows phoenix to run synchronously in async callbacks. I also added some more clear error handling so people know you can't run Phoenix Evals Asynchronously in a thread, you must be in the main.

Huge value to people who want to just run Evals as they are sending data versus manage chron jobs.

This is needed to support "Inline Evals" as part of the Span processor. The spans get sent to Phoenix in a normal fashion, but in a non-blocking way, a separate thread calculates the Eval and sends to Phoenix.
This notebook pulls the branch and is working! Note the Eval is added to the LLM span and does not show up in the Trace view:
https://colab.research.google.com/drive/14C6PQ_MdQXUtKkjY7NxKFuTqkS1xsKy1?usp=sharing

I did add the threading library for this check. The main code logic checks if we are in the "main" thread or not, and doesn't set up the signal.signal calls (needed for async) if we are running synchronously.

##Main line added ##
if threading.current_thread() is threading.main_thread():

Logical Description
Purpose:

This check is used to determine if the current code is being executed in the main thread of the application. The main thread is the thread that is initially started when the Python program begins execution. It is often where the primary control flow and main application logic run.

How It Works
threading.current_thread():

This function returns a reference to the current thread object, representing the thread in which the current code is being executed.
threading.main_thread():

This function returns a reference to the main thread object, representing the thread that was the initial entry point of the program.
is operator:

The is operator checks for object identity, meaning it checks if both references point to the same object (i.e., the same memory location).

Combined Logic
threading.current_thread() is threading.main_thread():

This condition evaluates to True if the current thread (in which the code is running) is the main thread of the application.
It evaluates to False if the current thread is any other thread (e.g., a worker thread, background thread, etc.).

------ NOTE on THREADING ------
We could use the OS library and not have to install a new library like "threading", this felt less standard but would work, let me know:
import _thread as thread # Standard library module for low-level thread operations
def is_main_thread():
# The main thread's identifier is the same as the process ID
return thread.get_ident() == os.getpid()

The notebook uses the SPANLEVEL hallucination detection template will do a separate PR to get those done but wanted to get this in to unblock.

@dosubot dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label May 18, 2024
@jlopatec jlopatec closed this May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant