Org. or project level langfuse workers

[bharath-muthineni]

Describe the feature or potential improvement

Hey langfuse team,
Just dropping a thought here about having separate worker deployments for some specific organisations or projects.
I'll try explaining my use case and if this is something which makes sense, I'd suggest we try doing this:
So, I have a few online applications that produce traces which I would like to see them on my dashboard almost instantaneously while there are other offline jobs which product insane amount of traces compared to my online use cases and I don't mind seeing them even after over a day.
Can I have some workers dedicated for my online applications - based on projects?
I'm happy to contribute if this is something that is going to help the community.

Additional information

No response

[CodeCLS]

Hi @bharath-muthineni, could you explain this in greater detail? Why would you want to slow down seeing the other traces if there is no cost-downside for you if they arrive immediately? Are you refering to something else perhaps? Thanks!

[bharath-muthineni]

@CodeCLS , what I'm trying to achieve is if there is a way to prioritize certain projects' events' processing over others? Probably have priority queue in terms of workers.

[CodeCLS]

Ok thanks for adding this. I understand it better now :).

[bharath-muthineni]

Bumping this up to understand if this is something that could fit in langfuse's roadmap. @marliessophie @Steffen911

[Steffen911]

@bharath-muthineni We would recommend that you scale the platform such that it can keep up with the regular load. Otherwise, one also runs risks of using redis capacity to the limits.
RIght now, separating the workloads is not on the roadmap and given the potential complexity we're also unlikely to accept community PRs on it.