ducklake_flush_inlined_data does not respect table partitioning when called from a fresh DuckDB connection

[michael-layfer]

What happens?

When calling ducklake_flush_inlined_data() from a fresh DuckDB connection (without any prior queries against the table), the resulting Parquet files are written to the table's root directory in S3, ignoring the configured Hive-style partition structure.

The same command executed from a DuckDB CLI session that has previously queried the table correctly respects partitions and writes files to the proper partition paths.

Environment
DuckDB version: 1.4.3
DuckLake version: (latest)
Metadata catalog: PostgreSQL
Data storage: S3

To Reproduce

Table is partitioned using:

ALTER TABLE my_ducklake.main.table 
SET PARTITIONED BY (year(_ingest_date), month(_ingest_date), day(_ingest_date));

Scenario A: Working (DuckDB CLI with prior queries)

-- Establish connection and attach catalog
ATTACH 'ducklake:postgres:postgresql://user:pass@host:5432/ducklake_catalog' AS my_ducklake
    (DATA_PATH 's3://bucket/data/');

-- Any query that touches the table schema (forces metadata load)
SELECT * FROM my_ducklake.main.table LIMIT 1;
-- OR
DESCRIBE my_ducklake.main.table;

-- Now flush respects partitions ✓
CALL ducklake_flush_inlined_data('my_ducklake');

Result: Files written to s3://bucket/data/year=2025/month=01/day=15/ducklake-xxx.parquet

Scenario B: Not working (Fresh connection, immediate flush)

import duckdb

# Fresh connection
conn = duckdb.connect()
conn.execute("INSTALL ducklake; LOAD ducklake;")
conn.execute("INSTALL postgres; LOAD postgres;")

# Attach catalog
conn.execute("""
    ATTACH 'ducklake:postgres:postgresql://user:pass@host:5432/ducklake_catalog' AS my_ducklake
        (DATA_PATH 's3://bucket/data/');
""")

# Immediately flush without any prior table access
conn.execute("CALL ducklake_flush_inlined_data('my_ducklake');")

Result: Files written to s3://bucket/data/ducklake-xxx.parquet (root directory, no partitions!)

Workaround
Before calling ducklake_flush_inlined_data, query the partition metadata to force it to be loaded:

-- Force partition metadata load
SELECT * FROM __ducklake_metadata_my_ducklake.public.ducklake_partition_column 
WHERE table_id = (
    SELECT table_id FROM __ducklake_metadata_my_ducklake.public.ducklake_table 
    WHERE table_name = 'table' AND end_snapshot IS NULL
);

-- Now flush works correctly
CALL ducklake_flush_inlined_data('my_ducklake');

OS:

aarch64

DuckDB Version:

1.4.3

DuckLake Version:

0.3

DuckDB Client:

Python

Hardware:

AWS Lambda function

Full Name:

Michael Layfer

Affiliation:

CISCO

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• Yes, I have

[pdet]

Thanks, Ive managed to reproduce!

[pdet]

Hi @michael-layfer I thought I had managed to reproduce this issue, but I had made a mistake of inserting the data before altering the table, which, of course, does not creat partitions when flushing.

Could you send a full reproducer? where you create the tables, insert the data and so on?

For reference, here was my attempt to reproduce it, but the test actually produces the partitions. Can you also confirm if the problem only happens with Postgres as the catalog, or with the data in S3?

require ducklake

require parquet

test-env DUCKLAKE_CONNECTION __TEST_DIR__/{BASE_TEST_NAME}.db

test-env DATA_PATH __TEST_DIR__

statement ok
ATTACH 'ducklake:${DUCKLAKE_CONNECTION}' AS ducklake (DATA_PATH 'partition_metadata_load', DATA_INLINING_ROW_LIMIT 1000 )

statement ok
USE ducklake

statement ok
create table t (a integer);

statement ok
ALTER TABLE t SET PARTITIONED BY (a);

statement ok
insert into t values (0), (1), (0), (1), (2);

statement ok
use memory

statement ok
DETACH ducklake

statement ok
ATTACH 'ducklake:${DUCKLAKE_CONNECTION}' AS ducklake (DATA_PATH 'partition_metadata_load', DATA_INLINING_ROW_LIMIT 1000 )

statement ok
USE ducklake

statement ok
CALL ducklake_flush_inlined_data('ducklake');

[duckdblabs-bot]

Thanks for opening this issue in the DuckLake issue tracker! To resolve this issue, our team needs a reproducible example. This includes:

• A source code snippet which reproduces the issue.
• The snippet should be self-contained, i.e., it should contain all imports and should use relative paths instead of hard coded paths (please avoid /Users/JohnDoe/...).
• A lot of issues can be reproduced with plain SQL code executed in the DuckDB command line client. If you can provide such an example, it greatly simplifies the reproduction process and likely results in a faster fix.
• If the script needs additional data, please share the data as a CSV, JSON, or Parquet file. Unfortunately, we cannot fix issues that can only be reproduced with a confidential data set. Support contracts allow sharing confidential data with the core DuckDB team under NDA.

For more detailed guidelines on how to create reproducible examples, please visit Stack Overflow's “Minimal, Reproducible Example” page.

[michael-layfer]

I have tried reproducing it locally with no success.
I kept the same versions for everything, I guess the main difference is that the initial failure happened while running on a Lambda function.

[pdet]

I see, @michael-layfer, I will be closing this issue for now, then. If you manage to reproduce it, feel free to reopen it with the reproducer! Many thanks,