Retrieve prompts based on versioning rather than times for caching

[ajram23]

Describe the feature or potential improvement

Prompts don't change in seconds or minutes. And I don't want to fetching the prompt from your server for every call. What would be be better is for the prompts to be cached and provide a mechanism for looking for updates(for newer versions) and updating the cache. And not make this part of the execution pipeline.

Additional information

Submitting as requested by Clemens from Langfuse based on our conversation on Intercom!

[marcklingen]

Thanks for the suggestion!

Current behavior: prompts are refetched (blocking) after the configured cache TTL, they are not fetched on every call

Suggested behavior: prompt are refetched after configured cache ttl but stale cache is served. I'd probably rather not have prompts refetch after first use on scheduled interval in background, this has the risk of refetching prompts which are not frequently used

What do you think about this @ajram23?

[ajram23]

@marcklingen To be honest, cache seems like a hack when you have versions. Curious why would you refetch prompts which are not frequently used if their versions don't change? May be I am missing something.

[marcklingen]

Thank you, short term you could add a very long TTL. Improvement: add "infinite" cache TTL?

[ajram23]

perhaps. I am hoping for a solution where I don't need to deploy for tweaking or iterating on prompts. and thank you for your engaging and congratulations on your week 1 of launch!

[marcklingen]

perhaps. I am hoping for a solution where I don't need to deploy for tweaking or iterating on prompts. and thank you for your engaging and congratulations on your week 1 of launch!

Sorry, then I did not understand you correctly. With the SDK-level cache, Langfuse does not "fetching every time is delay/introducing another point of failure while answering a user". What do you suggest to change here?

thank you for your engaging and congratulations on your week 1 of launch!

Thanks and no problem! I want to make prompt management in Langfuse awesome and any feedback is highly appreciated

[ajram23]

I made a suggestion at the beginning of the thread but I am empathetic to your other concerns. At the end of the day, want to be able to deploy the latest prompts to the user with minimal mucking around and performance impact. For now I am going with once a day as a cache. BTW one related question - generations always seem 0, do I have to trigger something for them to increment?

[marcklingen]

For now I am going with once a day as a cache

Sounds good

BTW one related question - generations always seem 0, do I have to trigger something for them to increment?

you'd need to link traces to prompt versions, see prompt management docs for details on this as it depends on how you integrate for tracing in langfuse