Possible race-condition when using COPY with PARTITION_BY

[colinbreame]

What happens?

When running:

import duckdb
input = duckdb.sql('SELECT year(time) as year, month(time) as month FROM "updates.parquet"')
duckdb.sql('''COPY input TO 'output' (FORMAT PARQUET, COMPRESSION ZSTD, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE 1)''')

Results in:

duckdb.duckdb.IOException: IO Error: Could not create directory: 'output\year=2023'

To Reproduce

It is difficult to reproduce with static data and seems to only happen with a specific input.

import duckdb
input = duckdb.sql('SELECT year(time) as year, month(time) as month FROM "updates.parquet"')
duckdb.sql('''COPY input TO 'output' (FORMAT PARQUET, COMPRESSION ZSTD, PARTITION_BY (year, month), OVERWRITE_OR_IGNORE 1)''')

It only happens if the directory has not previously been created.

Saving the input variable into a new parquet file, and then using it as the source of COPY does not trigger the error.

OS:

Windows

DuckDB Version:

0.9.1

DuckDB Client:

Python

Full Name:

Colin Breame

Affiliation:

Statkraft Germany GmbH

Have you tried this on the latest main branch?

I have tested with a main build

Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?

• Yes, I have