Skip to content

How to write csv file to disk from a empty dataframe? #16240

@mmooyyii

Description

@mmooyyii

I want write a csv file only include headers;

use datafusion::config::CsvOptions;
use datafusion::dataframe::DataFrameWriteOptions;
use datafusion::error::Result;
use datafusion::prelude::*;
use std::path::Path;

#[tokio::main]
async fn main() -> Result<()> {
    let ctx = SessionContext::new();

    let df = ctx.sql("SELECT 1 limit 0").await?;

    let mut cfg1 = DataFrameWriteOptions::new();
    cfg1 = cfg1.with_single_file_output(true);

    let mut cfg2 = CsvOptions::default();
    cfg2.has_header = Some(true);

    df.write_csv("/tmp/t.csv", cfg1, Some(cfg2)).await?;
    assert_eq!(Path::new("/tmp/t.csv").exists(), true);  // panic!
    Ok(())
}

Activity

mmooyyii

mmooyyii commented on Jun 3, 2025

@mmooyyii
Author

I want to do something like materialized view. I have to deal with this special case if datafusion can't write empty data frame;

alamb

alamb commented on Jun 3, 2025

@alamb
Contributor

I am not sure -- it seems like maybe we should have an option in DataFrameWriteOptions that will force an empty file to be written even if there are no rows 🤔

chenkovsky

chenkovsky commented on Jun 4, 2025

@chenkovsky
Contributor

take

reopened this on Jul 8, 2025
alamb

alamb commented on Jul 8, 2025

@alamb
Contributor

Unfortunately, we had to revert the original fix, see

So I am reopening the ticket to track

vadimpiven

vadimpiven commented on Jul 9, 2025

@vadimpiven
Contributor

I can add that for Arrow format the empty DataFrame also does not create a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @alamb@chenkovsky@mmooyyii@vadimpiven

    Issue actions

      How to write csv file to disk from a empty dataframe? · Issue #16240 · apache/datafusion