Alter generation names with LangChain integration

[nic-asnes-klaviyo]

👋
What I'm trying to do: trace several generations through a series of function calls. Afterwards, I'd like to run model-based evaluationwith the LangChain integration.

My hope is to use a different custom criterion for different generations. As a result, I need to filter the call to get_generations, e.g. langfuse.get_generations(name=SOME_SPECIFIC_GENERATION_NAME, limit=limit, user_id=user_id, page=page).

My integration of Langfuse for tracing generations looks something like this:

@observe()
async def llm_ainvoke(chain: LLMChain, params):

    # get the langchain handler for the current trace
    langfuse_handler = langfuse_context.get_current_langchain_handler()

    # use the handler to trace langchain runs within this function
    return await chain.ainvoke(params, config={"callbacks": [langfuse_handler]})

What I'm hoping to do is something like this:

@observe()
async def llm_ainvoke(chain: LLMChain, params, generation_name=None):

    # get the langchain handler for the current trace
    langfuse_handler = langfuse_context.get_current_langchain_handler()

    # TODO: use generation_name 

    # use the handler to trace langchain runs within this function ...
    return await chain.ainvoke(params, config={"callbacks": [langfuse_handler]})

In my current setup, all generation names are ChatOpenAI, so any attempts at filtering are very difficult. Am I missing some piece of the integration that could make this easier?

[marcklingen]

Sorry for slow response here during launch week.

Afaik you can add custom run_names to eg the ChatOpenAI class within your langchain chains: https://docs.smith.langchain.com/cookbook/tracing-examples/runnable-naming#example-2-runnable-lambda

Happy to have a quick chat about this to figure out the best solution in case this does not work: https://cal.com/marc-kl/10

[marcklingen]

@nic-asnes-klaviyo, did this solve this for you?