Phidata support

[michaelpoluektov]

Describe the feature or potential improvement

phidata is a framework to build "assistants" with custom function calls.

I believe the only monitoring framework they support is their own proprietary one, but I would rather avoid vendor lock-in.

Is it at all viable to integrate langfuse with it? Are there any plans to integrate other framworks at all?

Additional information

No response

[marcklingen]

happy to have a look. can you share docs on the phidata observability? could not find a link in the project docs

[michaelpoluektov]

Thanks! I don't think they have any docs on observability at all: right now I just log all the events to a plaintext log file by overriding the _api_log_assistant_event method of the Assistant class. If you just want to print it:

from phi.tools.duckduckgo import DuckDuckGo
from phi.llm.groq import Groq
from phi.assistant import Assistant

class LoggingAssistant(Assistant):
    def _api_log_assistant_event(self, event_type: str = "run", event_data: dict[str, ...] | None = None) -> None:
        print(f"Event type: {event_type}")
        print(f"Event data: {event_data}")
        
assistant = LoggingAssistant(
    llm=Groq(),  # whatever you have installed, they have an Ollama integration
    run_id="test_run",  # or None
    user_id="test_user",  # or None
    markdown=True,
    add_chat_history_to_messages=True,
    monitoring=True,  # tells it to actually log events
    tools=[DuckDuckGo(news=False)],
    description="You are a helpful assistant with internet access. Always use the DuckDuckGo tool.",
)

assistant.cli_app(markdown=True)

This method is being called in _run, where event_data has the following format:

        event_data = {
            "run_type": "assistant",
            "user_message": message,
            "response": llm_response,
            "response_format": llm_response_type,
            "messages": llm_messages,
            "metrics": self.llm.metrics if self.llm else None,
            "functions": functions,
            # To be removed
            "llm_response": llm_response,
            "llm_response_type": llm_response_type,
        }

[marcklingen]

thanks for sharing. should be fairly easy for you to use the low-level python sdk to create a trace for a run and add each _api_log_assistant_event to it as a generation if you want to store/view/evaluate in langfuse.

[michaelpoluektov]

Cheers, I've given it a quick go but decided that it's just easier to reimplement their Assistant API in langchain instead.

The rough way of doing this would be to iterate through messages from the last user message, matching each message with assistant role to a generation and each message with tool role to a span (with the timings from metrics["response_times"] and metrics["tool_call_times"] respectively.)

The problem is that I want to nest Assistants as tool calls and that's a bit less trivial. I might give it a look over the weekend if I get the time.

[marcklingen]

sounds good, let me know in case you make any progress on this. A cookbook on how to best integrate with Phidata could be interesting to help others to integrate.

reopening this thread to keep it open for further ideas and questions