Fix ambiguous reference error in filter plan

[jonmmease]

Which issue does this PR close?

Closes #1411.

cc @kszucs

Rationale for this change

This PR adds an initially failing test in a060b40 that reproduces the behavior described in #1411.

The change in 2042fbb may not be correct overall, but it addresses this particular failure.

The core issue seems to be that filter optimization examines fields across all plans and then fails because there is an ambiguity. At least in the DataFrame context, I would expect there to be no ambiguity since only one of the columns is projected prior to filtering.

[jonmmease]

unsurprisingly, this change causes a failure in the execution::context::tests::unprojected_filter test.

I'm not sure what the correct behavior here is. Should the projected schema take precedence to resolve ambiguity and then fall back to the combined schema if a column reference is not found?

[alamb]

I'm not sure what the correct behavior here is. Should the projected schema take precedence to resolve ambiguity and then fall back to the combined schema if a column reference is not found?

That sounds reasonable to me at a high level

@houqp wrote up the desired output name semantics here: https://arrow.apache.org/datafusion/specification/output-field-name-semantic.html

which might serve to guide you in your question

[alamb]

(thank you @jonmmease for raising this PR)

[houqp]

thank you @jonmmease for taking on this, I agree with @alamb that the order you outlined in your comment sounds like a good order to me.

[jonmmease]

Thanks for taking a look @alamb and @houqp! I made the proposed change.

This error poppup up in https://github.com/apache/arrow-datafusion/runs/5470235871?check_suite_focus=true:

 failures:

---- datasource::file_format::avro::tests::test stdout ----
thread 'datasource::file_format::avro::tests::test' panicked at 'Local file metadata: Os { code: 2, kind: NotFound, message: "No such file or directory" }', datafusion/src/datasource/object_store/local.rs:178:40


failures:
    datasource::file_format::avro::tests::test

Maybe something flaky?

[alamb]

Maybe something flaky?

I think this is related to the fact that somehow this PR has changes to the testing submodule which I suspect was not intended:

[jonmmease]

Thanks @alamb, not sure how that got in there 🤦 Should be reverted now

[jonmmease]

tests now passing

[houqp]

Thanks @jonmmease

[alamb]

Thanks @jonmmease