Upcoming architecture changes for Langfuse 3.0

[maxdeichmann]

Hi all,

Langfuse is growing a lot, both in feature scope as well as in usage on single instances. Thus we plan for a couple of changes that will be released in Langfuse v3.

We are currently required to mature our architecture as we are working on the following challenges:
✅ Building model-based evals, which requires us to run asynchronous tasks, rate limited, with failover capabilities.
🧑‍🍳 Improve performance as instances scale out.

I wanted to give you a heads up on upcoming changes which are required to make these features work. Currently, Langfuse contains a single Docker container, which takes care of everything we do. This was fast to set up Langfuse initially, but we need more technical capabilities now. In addition to the existing components (Docker container + Postgres database), we will add the following:

• Asynchronous processing in a dedicated container
• Redis container (for queues)
• OLAP storage (likely Clickhouse) to enable lower latencies on analytical queries and scale out storage for large and multi modal inputs outputs

If you self-host Langfuse, this means that we will likely advise to change to the following setup to be able to benefit from new infra changes easily. We are happy to hear your thoughts on this:

• Migrate from a container service (e.g. GCP Cloud Run) to a Virtual Machine to run docker-compose from there. Docker-compose helps to quickly bootstrap the new components.
• Either provide connection strings to the docker-compose for Redis, Postgres, and Clickhouse for managed databases or run all of them on the virtual machine as well.

Feel free to share your thoughts below on these topics:

• Tool choice: do you have experience with the tools mentioned above and have advise for us?
• Do you have ideas on how to make self-hosting easier in a multi-container, multi-database setup than docker compose?
• Do you have ideas on how a great migration path could look like?

Find more context in the last Langfuse Townhall meeting. We will provide an easy to follow upgrade path for self-hosters once v3 is generally available. The infrastructure change does not affect public APIs, thus, users of Langfuse Cloud will not be affected by this change. Currently we pilot the async container & queue for the evals feature which is currently in public beta on Langfuse Cloud.

We plan to release v3 in July/August and will post updates here.

UPDATE JULY 22nd

A more detailled overview of planned changes:

Architectures v2 vs. v3

• Containers
  • web container: hosts public api, and all resources for the user interface
  • worker container: asynchronous processes, no exposed ports
• Databases
  • Redis used as cache and queue
  • Postgres stores transactional data such as projects or API keys
  • Clickhouse stores tracing data generated by the SDKs. This database will do most of the processing as our server will insert all the SDK data and read it for tables and dashboards.

Next to the core application, an application load balancer for TLS termination and routing of requests to the Web container is necessary. We use nginx but you can also use the fully managed AWS load balancer.

Upgrade path from v2 to v3

Thousands of teams run on Langfuse (~400k docker pulls)
→ we aim to offer the easiest migration experience that is automated and documented

• Guidance on scalability of dockerized datastores, when are fully-managed datastores necessary
• Guidance of reliable auto-scaling configuration of the application and databases
• Migration script to move data from Postgres to Clickhouse

Application deployment

As there is no unique complexity, deployment on ECS follows the instructions of deploying on K8S.

For low-volume/non-production deployments, dockerized DBs + docker compose is a sensible option to keep complexity low. We will publish guidance on when options 3 and 4 are necessary.

DB deployment

Databases (see above): redis, postgres, clickhouse

Low-volume

• All datastores are available as single docker containers
• Can be bundled in docker compose or EKS/ECS/K8S deployment

High-volume / fully-managed → databases external of application cluster

• Redis
  • No special requirements
  • AWS: we chose ElastiCache (Redis OSS) for a fully managed Redis instance
• Postgres
  • No special requirements compared to other applications.
  • AWS: we chose RDS for a fully-managed Postgres instance
• Clickhouse
  • Main data store, thus this database will scale the most.
  • AWS:
    • There is no RDS-equivalent for Clickhouse on AWS
    • We use Clickhouse Cloud which is the fully managed Clickhouse database managed by the Clickhouse team. This can be run on an AWS region of your choice with VPC peering for private access and data security.
    • There are potentially additional vendors for managed Clickhouse such as DoubleCloud or Altinity.
    • Alternatively, you can run Clickhouse yourself in ECS, e.g. by using this template.

We will provide guidance at which scale high-volume/fully-managed clickhouse is necessary. As discussed, we expect dockerized Clickhouse to perform well at least up to 50 GB of tracing data (10s of millions of traces)

[MarkEdmondson1234]

As requested from Discord, my comment:

I really do not want to move off serverless infra to a dedicated VM, I'd say a major reason I chose Langfuse was its Cloud Run deployment I could couple with my existing AlloyDB. And perhaps AlloyDB is quick enough it doesn't need help with analytical queries.

Cloud Run recently introduced side car containers so perhaps that is an option?
https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments

There is a managed Redis option too but its a bit more pricy.

My serverless deployment at the moment I don't pay for Langfuse until I'm browsing the UI or its capturing traces, aside the already sunk cost of the database.

[noorvir]

The current docs advise not to use Docker Compose for production. I'm guessing that will change for v3? I would be great to have a ready to deploy docker compose that just needs an env file to get started. I'd use the cloud hosted service but have relatively strict data privacy requirements.

[marcklingen]

awesome, thank you @mautini 🙏 We are currently finalizing many infra pieces, will definitely reach out!

[benobytes]

Our current stack uses langfuse in Kubernetes so I will be happy to assist as well!

[verdverm]

We use k8s operators for the redis, postgres, clickhouse components. Would want a way to have develop vs production setups, which is probably easy from the chart perspective, more about docs / examples

[marcklingen]

I think it's sensible to do this similar as the current chart, i.e. provide two options:

1. dev/staging, low-volume prod: deploy all components within k8s
2. high-volume prod: add api keys for dedicated/managed redis, clickhouse, postgres

[DevBey]

big up vote, k8s is the way to go.

[Manouchehri]

I'm hesitant on a more complicated docker-compose setup. One of the reasons we were able to open to using Langfuse to begin with, was how easy it was to deploy on a serverless platform.

[nikcaryo-super]

copying from discord, GCP offers: https://cloud.google.com/blog/products/serverless/cloud-run-now-supports-multi-container-deployments but I'm not sure exactly how those would work for asynchronous tasks (e.g. does the sidecar running an async task continue running if the request finishes and the cpu is unallocated?)

maybe there's an architecture where async tasks are just sent as webhooks to whatever "langfuse-worker" url people have setup? and then GCP people can just have a langfuse-worker serverless instance / pubsub task queue or whatever.

[MarkEdmondson1234]

That’s how the rest of my infra works, pubsub events to Cloud Run scale to 0 workers

[marcklingen]

agree, would love to figure out together how to best run this on Google Cloud Run to potentially scale to 0. The worker is necessary to power async workloads (evals, tokenization) but could also scale to zero if the queue could be in pub/sub (GCP specific). I/m unsure how to find a generalizable solution here but we'll figure it out

[fernandocamargoai]

It feels weird to deploy using docker-compose. For me, it's always have been a good tool to use during development, but not really for production.

My team is currently planning to deploy it. We haven't decided if we'll deploy to kubernetes or Cloud Run though. We'll stay tuned to see what will work best for us.

[Manouchehri]

As a team with k8s experience.. we try to avoid it at all costs. 😬 We like Langfuse enough that we'd probably cave and do it if needed, but not our first choice.

[marcklingen]

Agree that we’d want to make it work on GCP/CloudRun and comparable solutions as it is just much easier. There’s no way really around it as without an additional container, it’s seems close to impossible to run async workloads

[MarkEdmondson1234]

There are a few options for async tasks, like Batch, cloud run jobs, workflows? GCP specific though. k8s is most generic but yeah, I’d prefer that only as a last option

[zoltan-fedor]

We do run Langfuse today on k8s (our team has lots of k8s experience - we run everything there with the exception of databases).
I wrote our own Helm chart for Langfuse (includes HPA (autoscaling), ingress, etc), which we use for about 1/2 a year with no issues both in prod and preprod.
I would advise that Langfuse includes a Helm chart for production-level deployments next to the docker compose option, especially once it become multi-container with v3.0

Also Redis should be allowed to be external (running Redis in a container in K8s is not really a good architecture) and connection to it parametrized just like Postgres is. I am sure I will be able to make it so in our Helm charts, but it could be useful for those who are less experienced. We already run Redis external to K8s.
We have no Clickhouse, but I am sure we will be able to set that up on K8s if required.

EDIT: I just saw that further own @marcklingen has already described basically the same I was recommending here (external database connection option + official helm chart offered by Langfuse). See https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

[marcklingen]

thanks for sharing you thoughts @zoltan-fedor 🙏

[chai3]

I want Radis and OLAP Clickhouse to be keep opt-in.

I am deploying LangFuse on AWS with a simple architecture.
Just AWS AppRunner and RDS.
AWS AppRunner only supports a single Docker image and cannot use docker compose and sidecar.
ECS, EC2, Radis, and ALB are expensive, but AppRunner only needs to pay the CPU cost when there is a request by user, and if there is no request, it is very cheap at 0.007 USD / GB-hour(If 2GB memory, 5 USD/month)

https://github.com/AI4Organization/langfuse-ecr-ecs-deployment-cdk/blob/f21d237324f9c5727227915a57523a7ead779720/lib/langfuse-ecr-apprunner-deployment-cdk-stack.ts

[qymab]

Any specific reason why not providing arm based images ?

[marcklingen]

See self-hosting docs

Are there prebuilt ARM images available? No, currently we do not publish official ARM images. However, you can build your own ARM images using the Dockerfile in the Langfuse repository.

No specific reason, open for contributions on this!

[marcklingen]

First version to include arm image was just released: https://github.com/langfuse/langfuse/releases/tag/v2.58.0
Thanks again @rennokki for the contribution

[newcworld]

when clickhouse, I think it's great

[aiakubovich]

I'm not a big fan of docker-compose and we unable to use docker-compose files for deployments. Our requirements dictate that we first build an image using an ARM template, scan it with an infosec tool like Aqua in the CI/CD pipeline, and then deploy it. It would be great if there is option to deploy each container as stand along app.

[marcklingen]

makes sense, thanks for sharing!

[tiagoantao]

We are currently very happy users of Langfuse 2 (massive internal adoption success).
There are a few of issues for us with the new architecture:

• Maintaining a new OLAP solution. We would prefer not to have it or to be able to integrate it with our existing OLAP solution
• We would prefer a container only solution vs a VM based solution. This is purely because our current deployment is container based and we have no VMs
• Better yet, containers without nothing attached (e.g. Clickhouse would require attaching volumes).

Thank you for considering our requirements

[marcklingen]

Thanks for you feedback here!

The v3 infrastructure will most likely include clickhouse as it is a very solid yet OSS OLAP db that works well at high throughput and size yet has support for upserts which are required in some scenarios.

we are currently aiming for these 4 deployment options:

data stores (postgres, redis, clickhouse)	docker compose (vm)	helm chart (k8s)
dockerized	1, low volume	2, low-mid volume
external, connection provided via env	3, low-mid volume	4, high volume

We'd encourage deployments with external data stores (3&4) for improved scalability and less ops overhead (no additional storage attached to containers) as there are managed services for all 3 data stores, postgres/redis offered by all major cloud vendors, and clickhouse e.g. via Clickhouse Cloud. At low volume, deployment via docker should work as well and we cross-checked this with many teams who run low-volume clickhouse deployments in docker.

The only alternative would be to maintain an additional postgres-only flavor of Langfuse. However, we do not want to go this route as this would slow down development a lot and would make the two versions drift in performance. Context: many analytics use cases right now are blocked by this change as the postgres instance is limited by IOPS for many teams who deploy langfuse at scale.

[verdverm]

I'd love to see this integrate with my existing log/mon stack (grafana lgtm, but there are others)

Having to fuse traces & logs across them is not ideal

[mrout94]

Will there be a way to migrate existing traces, prompts and datasets from v2 to v3?

We have massive internal success and adoption rate for this and there are a lot of data, prompts and traces stored in the containers. We are really excited about v3 but also do not want to redo these steps again.

Wishing the entire team good luck for the v3 release. Looking forward to this.

[marcklingen]

There will be a migration guide and script that will bring the migration effort down to a minimum. We are very much aware that you and many other teams depend on Langfuse for their workflows.

[snikch]

Hey team. Thanks for the updates in the newsletter.

I agree on the general comments that docker compose isn't a great production solution, however I generally see docker compose as a great way to document the required setup for self hosters. It's simple to grok the required services and interactions / configuration.

We will continue to use a k8s deployment, and are happy to integrate into our existing Redis and Clickhouse services / spin up new instances. Bringing up Clickhouse and Redis shouldn't be considered a large operational hurdle IMO for people wanting to self host. With the reality of services becoming more complex over time, and features / scale are added, it is a reasonable expectation (Posthog is a good example).

As long as the migration path is documented and clear this all sounds great! 👏

[marcklingen]

Thanks for sharing and your trust in this general path! We’ll follow up with drafts of these docs in this thread when we have a stable end-to-end deployment running across all features in order to get feedback and help folks test v3 ahead of a stable release.

We talked a lot about this and are convinced that this change is necessary to serve all teams that want to scale with Langfuse. If your product is doing well and Langfuse breaks as the infra was kept as simple as possible, that’s not great. Agree that Posthog is a great example as some other OSS analytics projects deliberately run on Postgres and thus can’t keep up with Posthog from my point of view.

[marcklingen]

Update to the above: we plan to release v3 in July (no strict ETA yet) as we are currently going though many optimization steps to make the new setup as performant as possible. We will post an update here once there is documentation and a pre-release version to try.

[4t8dd]

Looking forward to the migration to clickhouse.

[marcklingen]

me too!

[pamelafox]

I've received an inquiry as to how we'd be able to deploy your changes to Azure:
Azure-Samples/langfuse-on-azure#14

We can set up a Redis connection to Azure Cache for Redis. I'm not familiar with Clickhouse. Do you already have an idea about how we could get that working on Azure?

[marcklingen]

The architecture is not yet finalized but I'd suggest to deploy as mentioned here via either fully-managed services (as the one for Redis) or dockerized versions (e.g. Clickhouse): https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

This should work on Azure as will work on other cloud providers as well and we will share drafts of the docs on this ahead of the v3 release here. Would love to chat about potential adaptations to the Azure example once this is finalized.

[indranilr]

Would one then need to use managed services from AWS for Postgres, Redis and Clickhouse if hosting Langfuse as ECS containers ?

[marcklingen]

You will not need to use managed services. For low-mid volume deployments, you cam use dockerized versions of Redis, Postgres and Clickhouse or run them all on a single VM via docker compose. See this comment for more details: https://github.com/orgs/langfuse/discussions/1902#discussioncomment-9729809

[indranilr]

Any specific benchmarks/guidance around s to what is the criteria (numbers range) to classify as low, mid and high volume (I assuming this is Query TPS) ?

[marcklingen]

Not yet, we aim to provide guidance together with the v3 migration docs.

Core metrics:

• TPS of API
• Overall ingestion volume and order of magnitude of objects stored -> relevant for read-heavy APIs, can be approximated by #(Traces/Observations/Scores)/Timeframe

[luciowu]

Happy to see Langfuse is keep evolving!
We deployed Langfuse in k8s cluster in GCP. And in order to meet our OLAP requirements, we build our own ETL pipeline (PG -> BigQuery) to extract the data to BigQuery.
Hope to make Clickhouse as a plugin so that we could keep our current deployment architecture without change.

[clemra]

We've just updated the initial post with more information and guidance on the envisaged changes. Here attached in PDF format.
Langfuse_V3_Architecture.pdf