Optimizer `push_down_projection` failed due to different schemas

[SteveLauC]

Describe the bug

The query SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field'; panicked when being executed against an empty parquet file, below is the error message given by datafusion-cli:

The parquet file is empty in data, not schema, it has a field field.

push_down_projection
caused by
Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, 

original schema: DFSchema { fields: [], metadata: {} },

new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. 

To Reproduce

1. Use the following rust code to create an empty parquet file, data is empty but the fields in schema is NOT empty.

   fn main() {
      let file = OpenOptions::new()
          .write(true)
          .create(true)
          .open("test.parquet")
          .unwrap();
      let writer = ArrowWriter::try_new(
          file,
          Arc::new(Schema::new(vec![Field::new("field", DataType::Utf8, true)])),
          None,
      )
      .unwrap();
   
      writer.close().unwrap();
   }
   

2. Run the following query in datafuion-cli

   Or a Rust program using the datafusion library, you will get the same result

   $ ls -l test.parquet
   .rw-r--r--@ 263 steve  9 Aug 13:40 test.parquet
   
   $ datafusion-cli
   ❯ SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field';
   push_down_projection
   caused by
   Internal error: Optimizer rule 'push_down_projection' failed, due to generate a different schema, original schema: DFSchema { fields: [], metadata: {} }, new schema: DFSchema { fields: [DFField { qualifier: Some(Bare { table: "test.parquet" }), field: Field { name: "field", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} } }], metadata: {} }. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker

Expected behavior

The query can be successfully executed.

Additional context

• parquet library version: 43
• datafusion-cli version: 28.0.0

[SteveLauC]

BTW, in my code using the datafusion library, I try to skip failed optimizer rules using the following code snippet:

    let mut optimizer_cfg = OptimizerOptions::default();
    optimizer_cfg.skip_failed_rules = true;
    let mut cfg_opt = ConfigOptions::default();
    cfg_opt.optimizer = optimizer_cfg;
    let session_cfg = SessionConfig::from(cfg_opt);

    let ctx = SessionContext::with_config(session_cfg);

But I can still get this error :(

[smiklos]

I'm not entirely sure what the result of selecting no columns should be?
The current behavior is the same for csv files or any table source. Also it's independent of whether the file is empty

Imo datafusion should reject queries that don't select anything or alternatively act as if 'SELECT NULL' was issued

[SteveLauC]

I'm not entirely sure what the result of selecting no columns should be?

Agree, you reminds me that my example is actually bad...

Let's say we have an empty parquet file whose scheme is

1. optional field string
2. optional another_field string

Then run the query "SELECT * EXCEPT (field) FROM 'test.parquet' WHERE field = 'field'", I guess you can still get the error. I haven't checked out what is the behavior against parquet, for my custom data source, the error can still be triggered

[smiklos]

As long as you have one col left in the select, it works. The error only occurs when except excludes all columns.

[SteveLauC]

My pc is not at my hand, will give it a test tomorrow:)

[smiklos]

Another thing to point out is that it's not the rule that fails but the fact that it transforms the plan's schema.

It seems strange that a rule changes the schema when it's explicitly not allowed by any optimization rules

[SteveLauC]

As long as you have one col left in the select, it works. The error only occurs when except excludes all columns.

You are right, it works with parquet and my custom data source. Thanks for your help.

---

In my perspective, this issue can be closed, so feel free to do so.

[jackwener]

Thank you @smiklos .

I prepare to improve error-message to make easier to understand when meet this problem

[Dandandan]

I think it's still a bug worth addressing.
Likely it has to do with that projection push down will add the first field for empty projections, thereby altering the schema.

[smiklos]

Yeah, I'm looking into that rn . Also don't you think we should also catch the error thrown for changed schemas and allow users to opt out from that rule? Right now it blows up the entire optimizer flow