[RUST][Datafusion] What causes "Error: Execution("file size of 4 is less than footer")" error?

[Miyake-Diogo]

When I Try to read a CSV and Write as Parquet, compiler raises this errror: Error: Execution("file size of 4 is less than footer").
I didn't found anything on docs and internetm, about it ...
My code:

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
  let ctx: SessionContext = SessionContext::new();
  let raw_covid_path: &str = "data/minilake/raw/covid/owid-covid-data.csv";
  
  let covid_df = ctx.read_csv(raw_covid_path,  
                                  CsvReadOptions::new()).await?;
  covid_df.write_parquet(stage_covid_path,None).await?;
  Ok(())
  }

Covid Dataset is located in this link

[westonpace]

I see that you are reading raw_covid_path but you are writing stage_covid_path (which I don't see defined anywhere). Is this just a typo in your example?

[Miyake-Diogo]

I read from a Raw Path and Write in another path in this case stage as a parquet type.

[andygrove]

working version of code:

use datafusion::prelude::*;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx: SessionContext = SessionContext::new();
    let raw_covid_path: &str = "/mnt/bigdata/covid/owid-covid-data.csv";
    let stage_covid_path = "/tmp/covid";
    let covid_df = ctx.read_csv(raw_covid_path,
                                CsvReadOptions::new()).await?;
    covid_df.write_parquet(stage_covid_path,None).await?;
    Ok(())
}

confirmed that this produces a 4 byte parquet file (with DataFusion 13)

[andygrove]

The plot thickens... I ran this unit test in df:

#[tokio::test]
async fn main() -> datafusion::error::Result<()> {
    let ctx: SessionContext = SessionContext::new();
    let raw_covid_path: &str = "/mnt/bigdata/covid/owid-covid-data.csv";
    let covid_df = ctx.read_csv(raw_covid_path,
                                CsvReadOptions::new()).await?;
    let batches = covid_df.collect().await?;
    for batch in &batches {
        println!("{}", batch.num_rows());
    }
    Ok(())
}

fails with:

Error: ArrowError(InvalidArgumentError("Column 'population' is declared as non-nullable but contains null values"))

so it appears we have 2 separate issues:

• schema inference on partial scan says field is non-null but there are nulls .. maybe that is fine
• errors seem to be consumed and ignored when reading csv and writing to another format

[Dandandan]

On the null issue, I think schema inference should default to assuming everything is nullable?

[andygrove]

On the null issue, I think schema inference should default to assuming everything is nullable?

That would seem safer to me

[andygrove]

@Miyake-Diogo as with #3783, if you specify the schema rather than inferring the schema you will get past this issue.