Langfuse v4 self-hosting preview

[Steffen911]

Hi all,
In https://github.com/orgs/langfuse/discussions/12518 we've shared out plans for a new data model and subsequently published https://langfuse.com/blog/2026-03-10-simplify-langfuse-for-scale with more implementation details.

Today, we want to share how to run a Langfuse v4 preview in self-hosted environments.

Recap / What is new?

We are moving to an observation-centric data model based on a new, wide, (mostly) immutable ClickHouse table. This eliminates joins and deduplication at read-time and optimizes for the most performant ClickHouse access patterns.

Who can use this?

At this time, you can use a Langfuse v4 setup for completely new deployments only and you must use the Langfuse Python v4 or JavaScript v5 SDK versions or any external OpenTelemetry-based instrumentation.

Additionally, you must use the following minimum versions of the Langfuse infrastructure components:

• ClickHouse version: 25.12 (minimum) / 26.4 (recommended) (Necessary to support new lightweight updates, JSON type, and full text search)
• Postgres version: 16 (recommended)
• Redis version: 7.2 (recommended)

How to use this?

Start with a regular Langfuse v3 deployment using release v3.179.1 or later.

1. ClickHouse Schema Migrations

Connect to your ClickHouse warehouse and execute the following migrations.

Initial Migration Statements

On non-cloud, multi-node clusters, tables must be created as `ReplicatedReplacingMergeTree` instead of `ReplacingMergeTree`. Replace those below. Also add `ON CLUSTER ‘default’` to the first row of each statement except the `MATERIALIZED VIEW`.

-- Staging table for the dual-write pipeline that populates events_full.
-- Ingestion writes every observation (and trace-as-synthetic-observation) here
-- with an s3_first_seen_timestamp; the periodic event-propagation job in the
-- worker reads completed 3-minute partitions, joins them with `traces`, and
-- inserts the result into `events_full`.
--
-- Partitions are automatically expired after 48 hours via TTL. The 48h window
-- (vs. the 12h used internally for Langfuse Cloud) gives self-hosters a
-- multi-day grace period to recover the propagation job after an incident
-- without losing staging data. `ttl_only_drop_parts = 1` ensures only complete
-- partitions are dropped, never individual rows.
CREATE TABLE IF NOT EXISTS observations_batch_staging
(
    id String,
    trace_id String,
    project_id String,
    type LowCardinality(String),
    parent_observation_id Nullable(String),
    start_time DateTime64(3),
    end_time Nullable(DateTime64(3)),
    name String,
    metadata Map(LowCardinality(String), String),
    level LowCardinality(String),
    status_message Nullable(String),
    version Nullable(String),
    input Nullable(String) CODEC(ZSTD(3)),
    output Nullable(String) CODEC(ZSTD(3)),
    provided_model_name Nullable(String),
    internal_model_id Nullable(String),
    model_parameters Nullable(String),
    provided_usage_details Map(LowCardinality(String), UInt64),
    usage_details Map(LowCardinality(String), UInt64),
    provided_cost_details Map(LowCardinality(String), Decimal64(12)),
    cost_details Map(LowCardinality(String), Decimal64(12)),
    total_cost Nullable(Decimal64(12)),
    usage_pricing_tier_id Nullable(String),
    usage_pricing_tier_name Nullable(String),
    tool_definitions Map(String, String),
    tool_calls Array(String),
    tool_call_names Array(String),
    completion_start_time Nullable(DateTime64(3)),
    prompt_id Nullable(String),
    prompt_name Nullable(String),
    prompt_version Nullable(UInt16),
    created_at DateTime64(3) DEFAULT now(),
    updated_at DateTime64(3) DEFAULT now(),
    event_ts DateTime64(3),
    is_deleted UInt8,
    s3_first_seen_timestamp DateTime64(3),
    environment LowCardinality(String) DEFAULT 'default'
)
ENGINE = ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toStartOfInterval(s3_first_seen_timestamp, INTERVAL 3 MINUTE)
PRIMARY KEY (project_id, toDate(s3_first_seen_timestamp))
ORDER BY (
    project_id,
    toDate(s3_first_seen_timestamp),
    trace_id,
    id
)
TTL toDateTime(s3_first_seen_timestamp)+ INTERVAL 48 HOUR
SETTINGS ttl_only_drop_parts = 1;

-- events_full is the immutable, full-fidelity event table that will eventually
-- replace observations. It is populated by the worker propagation job that
-- joins observations_batch_staging with traces and dataset_run_items and new
-- OTel based SDKs.
CREATE TABLE IF NOT EXISTS events_full
(
    project_id String,
    trace_id String,
    span_id String,
    parent_span_id String,

    start_time DateTime64(6),
    end_time Nullable(DateTime64(6)),

    -- Core properties
    name String,
    type LowCardinality(String),
    environment LowCardinality(String) DEFAULT 'default',
    version String,
    release String,
    trace_name String,
    user_id String,
    session_id String,
    tags Array(String),
    level LowCardinality(String),
    status_message String,
    completion_start_time Nullable(DateTime64(6)),
    is_app_root Bool DEFAULT false,

    -- Updateable properties
    bookmarked Bool DEFAULT false,
    public Bool DEFAULT false,

    -- Prompt
    prompt_id String,
    prompt_name String,
    prompt_version Nullable(UInt16),

    -- Model
    model_id String,
    provided_model_name String,
    model_parameters String,

    -- Usage and Cost
    provided_usage_details Map(LowCardinality(String), UInt64),
    usage_details Map(LowCardinality(String), UInt64),
    provided_cost_details Map(LowCardinality(String), Decimal(18,12)),
    cost_details Map(LowCardinality(String), Decimal(18,12)),
    calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),
    calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
    calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
    total_cost Decimal(18, 12) ALIAS cost_details['total'],

    usage_pricing_tier_id Nullable(String),
    usage_pricing_tier_name Nullable(String),

    -- Tools
    tool_definitions Map(String, String),
    tool_calls Array(String),
    tool_call_names Array(String),

    -- I/O
    input String CODEC(ZSTD(3)),
    input_length UInt64 MATERIALIZED lengthUTF8(input),
    output String CODEC(ZSTD(3)),
    output_length UInt64 MATERIALIZED lengthUTF8(output),

    -- Metadata
    metadata_names Array(String),
    metadata_values Array(String),

    -- Experiment properties
    experiment_id String,
    experiment_name String,
    experiment_metadata_names Array(String),
    experiment_metadata_values Array(String),
    experiment_description String,
    experiment_dataset_id String,
    experiment_item_id String,
    experiment_item_version Nullable(DateTime64(6)),
    experiment_item_expected_output String,
    experiment_item_metadata_names Array(String),
    experiment_item_metadata_values Array(String),
    experiment_item_root_span_id String,

    -- Source metadata (Instrumentation)
    source LowCardinality(String),
    service_name String,
    service_version String,
    scope_name String,
    scope_version String,
    telemetry_sdk_language LowCardinality(String),
    telemetry_sdk_name String,
    telemetry_sdk_version String,

    -- Generic props
    blob_storage_file_path String,
    event_bytes UInt64,
    created_at DateTime64(6) DEFAULT now(),
    updated_at DateTime64(6) DEFAULT now(),
    event_ts DateTime64(6),
    is_deleted UInt8,

    -- Indexes
    INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,
    INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,
    INDEX idx_fts_input_low lower(input) TYPE text(tokenizer = splitByNonAlpha),
    INDEX idx_fts_output_low lower(output) TYPE text(tokenizer = splitByNonAlpha),
    INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
    INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
)
ENGINE = ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toYYYYMM(start_time)
PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
SAMPLE BY xxHash32(trace_id)
SETTINGS
    index_granularity_bytes = '64Mi', -- Default 10MiB. Prevents very small granules due to large rows.
    merge_max_block_size_bytes = '64Mi',
    enable_block_number_column = 1,
    enable_block_offset_column = 1,
    prewarm_mark_cache = 1,
    prewarm_primary_key_cache = 1,
    enable_full_text_index = 1;

-- events_core is the lightweight, query-optimized projection of events_full
-- with truncated input/output/metadata. It is populated via the events_core_mv
-- materialized view defined below.
CREATE TABLE IF NOT EXISTS events_core
(
    project_id String,
    trace_id String,
    span_id String,
    parent_span_id String,

    start_time DateTime64(6),
    end_time Nullable(DateTime64(6)),

    -- Core properties
    name String,
    type LowCardinality(String),
    environment LowCardinality(String) DEFAULT 'default',
    version String,
    release String,
    trace_name String,
    user_id String,
    session_id String,
    tags Array(String),
    level LowCardinality(String),
    status_message String,
    completion_start_time Nullable(DateTime64(6)),
    is_app_root Bool DEFAULT false,

    -- Updateable properties
    bookmarked Bool DEFAULT false,
    public Bool DEFAULT false,

    -- Prompt
    prompt_id String,
    prompt_name String,
    prompt_version Nullable(UInt16),

    -- Model
    model_id String,
    provided_model_name String,
    model_parameters String,

    -- Usage
    provided_usage_details Map(LowCardinality(String), UInt64),
    usage_details Map(LowCardinality(String), UInt64),
    provided_cost_details Map(LowCardinality(String), Decimal(18,12)),
    cost_details Map(LowCardinality(String), Decimal(18,12)),
    calculated_input_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0, cost_details))),
    calculated_output_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
    calculated_total_cost Decimal(18, 12) MATERIALIZED arraySum(mapValues(mapFilter(x -> positionCaseInsensitive(x.1, 'input') > 0 OR positionCaseInsensitive(x.1, 'output') > 0, cost_details))),
    total_cost Decimal(18, 12) ALIAS cost_details['total'],

    usage_pricing_tier_id Nullable(String),
    usage_pricing_tier_name Nullable(String),

    -- Tools
    tool_definitions Map(String, String),
    tool_calls Array(String),
    tool_call_names Array(String),

    -- I/O
    input String,
    input_length UInt64 MATERIALIZED lengthUTF8(input),
    output String,
    output_length UInt64 MATERIALIZED lengthUTF8(output),

    -- Metadata
    metadata_names Array(String),
    metadata_values Array(String),

    -- Experiment properties
    experiment_id String,
    experiment_name String,
    experiment_metadata_names Array(String),
    experiment_metadata_values Array(String),
    experiment_description String,
    experiment_dataset_id String,
    experiment_item_id String,
    experiment_item_version Nullable(DateTime64(6)),
    experiment_item_expected_output String,
    experiment_item_metadata_names Array(String),
    experiment_item_metadata_values Array(String),
    experiment_item_root_span_id String,

    -- Source metadata (Instrumentation)
    source LowCardinality(String),
    service_name String,
    service_version String,
    scope_name String,
    scope_version String,
    telemetry_sdk_language LowCardinality(String),
    telemetry_sdk_name String,
    telemetry_sdk_version String,

    -- Generic props
    blob_storage_file_path String,
    event_bytes UInt64,
    created_at DateTime64(6) DEFAULT now(),
    updated_at DateTime64(6) DEFAULT now(),
    event_ts DateTime64(6),
    is_deleted UInt8,

    -- Indexes
    INDEX idx_span_id span_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_trace_id trace_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_user_id user_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_session_id session_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_created_at created_at TYPE minmax GRANULARITY 1,
    INDEX idx_updated_at updated_at TYPE minmax GRANULARITY 1,
    INDEX idx_provided_model_name provided_model_name TYPE bloom_filter(0.01) GRANULARITY 2,
    INDEX idx_experiment_id experiment_id TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_metadata_names metadata_names TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_fts_metadata_values metadata_values TYPE text(tokenizer = splitByNonAlpha),
    INDEX idx_fts_metadata_names metadata_names TYPE text(tokenizer = splitByNonAlpha)
)
ENGINE = ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toYYYYMM(start_time)
PRIMARY KEY (project_id, toStartOfMinute(start_time), xxHash32(trace_id))
ORDER BY (project_id, toStartOfMinute(start_time), xxHash32(trace_id), span_id, start_time)
SAMPLE BY xxHash32(trace_id)
SETTINGS
    enable_block_number_column = 1,
    enable_block_offset_column = 1,
    prewarm_mark_cache = 1,
    prewarm_primary_key_cache = 1,
    enable_full_text_index = 1;

-- Materialized view to populate events_core from events_full.
CREATE MATERIALIZED VIEW IF NOT EXISTS events_core_mv TO events_core AS
SELECT
    project_id,
    trace_id,
    span_id,
    parent_span_id,
    start_time,
    end_time,
    name,
    type,
    environment,
    version,
    release,
    trace_name,
    user_id,
    session_id,
    tags,
    level,
    status_message,
    completion_start_time,
    is_app_root,
    bookmarked,
    public,
    prompt_id,
    prompt_name,
    prompt_version,
    model_id,
    provided_model_name,
    model_parameters,
    provided_usage_details,
    usage_details,
    provided_cost_details,
    cost_details,
    usage_pricing_tier_id,
    usage_pricing_tier_name,
    tool_definitions,
    tool_calls,
    tool_call_names,
    leftUTF8(input, 200) as input,
    leftUTF8(output, 200) as output,
    metadata_names,
    arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
    experiment_id,
    experiment_name,
    experiment_metadata_names,
    experiment_metadata_values,
    experiment_description,
    experiment_dataset_id,
    experiment_item_id,
    experiment_item_version,
    experiment_item_expected_output,
    experiment_item_metadata_names,
    experiment_item_metadata_values,
    experiment_item_root_span_id,
    source,
    service_name,
    service_version,
    scope_name,
    scope_version,
    telemetry_sdk_language,
    telemetry_sdk_name,
    telemetry_sdk_version,
    blob_storage_file_path,
    event_bytes,
    created_at,
    updated_at,
    event_ts,
    is_deleted
FROM events_full;

Schema Updates

For any updates to the schema, we will publish dedicated ALTER statements here including the release that they need to be applied for. Make sure to check this discussion for any upgrade as the application may break otherwise.
We recommend to apply all of the given upgrades in order before upgrading to the version being referenced.

v3.184.0

ALTER TABLE events_full ADD INDEX IF NOT EXISTS idx_fts_metadata_values_ngram arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;

ALTER TABLE events_core ADD INDEX IF NOT EXISTS idx_fts_metadata_values_ngram arrayStringConcat(metadata_values) TYPE ngrambf_v1(4, 32000, 3, 0) GRANULARITY 2 SETTINGS enable_full_text_index = 1;

v3.204.0

ALTER TABLE observations_batch_staging
  ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
  SETTINGS enable_full_text_index = 1;

ALTER TABLE observations_batch_staging
  ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

ALTER TABLE observations_batch_staging
  ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_full
  ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_full
  ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_full
  ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_core
  ADD COLUMN IF NOT EXISTS ingestion_api_key String DEFAULT ''
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_core
  ADD COLUMN IF NOT EXISTS ingestion_sdk_name LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

ALTER TABLE events_core
  ADD COLUMN IF NOT EXISTS ingestion_sdk_version LowCardinality(String) DEFAULT 'unknown'
  SETTINGS enable_full_text_index = 1;

-- Update events MV
ALTER TABLE events_core_mv MODIFY QUERY
SELECT
    project_id,
    trace_id,
    span_id,
    parent_span_id,
    start_time,
    end_time,
    name,
    type,
    environment,
    version,
    release,
    trace_name,
    user_id,
    session_id,
    tags,
    level,
    status_message,
    completion_start_time,
    is_app_root,
    bookmarked,
    public,
    prompt_id,
    prompt_name,
    prompt_version,
    model_id,
    provided_model_name,
    model_parameters,
    provided_usage_details,
    usage_details,
    provided_cost_details,
    cost_details,
    usage_pricing_tier_id,
    usage_pricing_tier_name,
    tool_definitions,
    tool_calls,
    tool_call_names,
    leftUTF8(input, 200) as input,
    leftUTF8(output, 200) as output,
    metadata_names,
    arrayMap(v -> leftUTF8(v, 200), metadata_values) as metadata_values,
    experiment_id,
    experiment_name,
    experiment_metadata_names,
    experiment_metadata_values,
    experiment_description,
    experiment_dataset_id,
    experiment_item_id,
    experiment_item_version,
    experiment_item_expected_output,
    experiment_item_metadata_names,
    experiment_item_metadata_values,
    experiment_item_root_span_id,
    source,
    service_name,
    service_version,
    scope_name,
    scope_version,
    telemetry_sdk_language,
    telemetry_sdk_name,
    telemetry_sdk_version,
    blob_storage_file_path,
    event_bytes,
    created_at,
    updated_at,
    event_ts,
    is_deleted,
    ingestion_api_key,
    ingestion_sdk_name,
    ingestion_sdk_version
FROM events_full
SETTINGS enable_full_text_index = 1;

ClickHouse Cloud Settings

ClickHouse Cloud implements specific settings that are not available for open source projects. We use the following settings on top of the data schema declared above:

• cache_populated_by_fetch = 1: Add this to the settings for the events_core table.

ClickHouse User Permissions

In addition to the standard ClickHouse User Permissions the user should receive:

GRANT SELECT(database, partition, table, active) ON system.parts TO 'user';
GRANT SELECT(database, table, is_done) ON system.mutations TO 'user';
GRANT SYSTEM SYNC REPLICA ON default.observations_batch_staging TO 'user';
GRANT READ ON REMOTE TO 'user';
GRANT SELECT ON system.query_log TO 'user';

2. Environment Variable Changes

To switch an existing Langfuse v3 deployment with a suitable ClickHouse schema (see previous point) to a Langfuse v4 preview, set the following environment variables:

• LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only: Configures the backend to push all tracing information into the new events tables.
• LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true: Forces the UI to read from the new events tables.
• LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct: Write all OpenTelemetry spans directly into the events tables without attribute propagation as this is not supported in the new mode.
• CLICKHOUSE_DISABLE_LAZY_MATERIALIZATION=true: Needs to be disabled due to ongoing compatibility issues.

Limitations

The following endpoints and functionality will not be present on a Langfuse v4 preview deployment. Support for those features and endpoints may become available for the full Langfuse v4 rollout.

Endpoints

Replaced with OpenTelemetry-based ingestion:

• POST /api/public/spans: Returns 404.
• POST /api/public/ingestion: Keeps 207 status code, but returns 400 for every event type that is not a score-create, sdk-log, or dataset-run-item-create.
• POST /api/public/generations: Returns 404.
• POST /api/public/events: Returns 404.
• POST /api/public/traces: Returns 404.

Replaced with GET /api/public/v2/observations [link]:

• GET /api/public/spans: Returns 404.
• GET /api/public/generations: Returns 404.
• GET /api/public/traces: Returns 404.
• GET /api/public/traces/:id: Returns 404.
• GET /api/public/sessions: Returns 404.
• GET /api/public/sessions/:id: Returns 404.
• GET /api/public/observations: Returns 404.
• GET /api/public/observations/:id: Returns 404.

Replaced with GET /api/public/v3/scores [link]:

• GET /api/public/scores: Returns 404.
• GET /api/public/scores/:id: Returns 404.
• GET /api/public/v2/scores: Returns 404.
• GET /api/public/v2/scores/:id: Returns 404.

Replaced with GET /api/public/v2/metrics [link]:

• GET /api/public/metrics: Returns 404.
• GET /api/public/metrics/daily: Returns 404.

Features

Trace-based & legacy-dataset-based LLM as a Judge evaluations:

• Currently, not supported as they perform lookups on the traces/observations tables. We’re investigating how to port them over to the events tables.

Batch exports / integrations:

• The mixpanel, posthog and blob storage integrations must be configured for events tables [changelog]. The classical TRACE_OBSERVATIONS export option won’t produce any data.

FAQ & Troubleshooting

Please ask any questions within this thread or contact the maintainers. We will publish guides or frequently appearing questions here.

What is coming next?

Internally, we work on migrations for existing Langfuse v3 deployments including historic data migrations.
We will provide updates once we have more things to share.

Changelog

• 2026-06-08: Initial version.
• 2026-06-11: Add v3.184.0 migration.
• 2026-06-29: Add ClickHouse user permissions.
• 2026-07-03: Add v3.204.0 migration.

[YeshwanthSadum]

If I'm using langfuse v4 SDK, latest langfuse self-hosted version.
Will all the methods in sdk work? Also does the UI will not have separate tabs for traces and observation once the above changes are done?

[Steffen911]

@YeshwanthSadum Yes, you will only see the Observations view within the UI, similar to Langfuse Cloud when the "Fast (Preview)" mode is enabled.

I'll add @hassiebp to cover the SDK question, but my expectation is that you'd see broad compatibility across most SDK functions unless they call the endpoints listed above.

[danielstankw]

@Steffen911 thank you for the update,
Regarding the docker image, when can we expect a stable v4 docker image version tag? - from what I see latest is still 3.X.X

[Steffen911]

Right now, we cannot provide a timeline for a v4-tagged image yet. We recommend to follow the path above to rollout a Langfuse v4 state.

[kbumsik]

Thanks for the continued work on v4, and especially for committing to support it on the OSS. I've been looking forward to this update!

I have two questions about the preview:

1. I am a bit confused on the migration. The post says the v4 setup is for "completely new deployments only," but the instructions start with "Start with a regular Langfuse v3 deployment".
   So does this mean I should stand up a fresh v3 deployment with no existing data, and then apply the ClickHouse migrations and environment-variable changes to switch it into v4 preview mode? Or am I misunderstanding the flow?

2. I wonder if it is safe to apply v4 preview now. I'm keen to try the preview, but I'd like to know whether a preview deployment is likely to break (or require a DB rebuild) when the stable version ships. I am fine with the API compatibility issues, but DB schema compatibility is a thing.

Thanks again!

[Steffen911]

@kbumsik

1.: Yes. With this setup, no data migration happens, i.e. if you would apply the changes to an existing deployment, the old data wouldn't be visible within the UI anymore. Therefore, we recommend to start from a fresh setup with no data.

2.: We're running the given database schema for months on Langfuse Cloud and have confidence in the setup. We call it a preview as we want to wait for a firm timeline on migrations behaviours before producing a stable Langfuse v4 release. At this time, we don't expect any backwards incompatible changes to the database schema and think it's safe for adoption on fresh installations.

[andrey-pipedrive]

If we have this deployment, will we be able to continue with in future, e.g get updates to GA version, or it will be a new deployment again?

[Steffen911]

@andrey-pipedrive We've run the given schemata on Langfuse Cloud for multiple weeks and do not expect breaking changes. At this point, we expect that deployments with this preview can be converted into a full GA version without further changes.

[ZPerling]

It currently shows:

- name: LANGFUSE_MIGRATION_v4_WRITE_MODE

but it should be:

- name: LANGFUSE_MIGRATION_V4_WRITE_MODE

(V4 needs to be uppercase.)

Since environment variables are case-sensitive, using the lowercase v4 causes Langfuse to ignore the setting and fall back to the default legacy mode, which then results in the following startup error:

Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}.

It took me a while to spot this, so hopefully this helps anyone else who copies the example.

[Steffen911]

@ZPerling Great catch and thank you! I've patched the original discussion to make it easier for anyone new who comes across this 🙏

[Millmer]

We're really looking forward to updating to v4, thanks for all the work to make it available to OSS as well 😄

We're running in production (v3.178.0) and would prefer to keep our existing deployment rather than spin up a new one if we can. While we wait for a migration path for existing deployments, what's the latest version we can safely update to?

To confirm my understanding: can we update to the latest version and as long as we leave the new environment variables unset it will stay stable until the migrations for existing data are available?

[Steffen911]

@Millmer Yes, all v3 major releases are fully compatible with each other. You can safely upgrade in the meantime as long as you keep the environment variables to their default settings.

[Millmer]

Thanks @Steffen911, I just upgraded to v3.221.1 and unfortunately all the traces have dissapeared from our UI. Am I missing something in the configuration to tell the UI to still process things in a "v3 way"? I looked in our Clickhouse and I see that all the data is still there, which is good, so I believe this is just an issue with the way the data is being processed and returned to the UI. I didn't set any of the new v4 related env vars.

I see there are a few pending migrations (like createRootSpansFromTraces, rewriteObservationsToPidTidSorting, backfillEventsFullFromObservations) but they are gated behind the LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL env var so I don't think they will run (and I'm not sure if they are necessary to run for us atm)

Any guidance here would be appreciated :)

False alarm. I was being impatient...