Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify schema usage in Datafusion #10442

Open
comphead opened this issue May 10, 2024 · 1 comment
Open

Unify schema usage in Datafusion #10442

comphead opened this issue May 10, 2024 · 1 comment

Comments

@comphead
Copy link
Contributor

Sometimes DF uses 2 schema sources, which is schema from previous node and current like in example(s) below.

Would be nice to check if we can use only 1 schema, multiple schemas are usually confusing

          I agree it doesn't make sense -- however, it is the same logic as in `with_new_exprs`:

LogicalPlan::Union(Union { schema, .. }) => {
let input_schema = inputs[0].schema();
// If inputs are not pruned do not change schema.
let schema = if schema.fields().len() == input_schema.fields().len() {
schema.clone()
} else {
input_schema.clone()
};
Ok(LogicalPlan::Union(Union {
inputs: inputs.into_iter().map(Arc::new).collect(),
schema,
}))
}

Originally posted by @alamb in #10405 (comment)

@yyy1000
Copy link
Contributor

yyy1000 commented May 14, 2024

When I try to fix this, I found that there may be a latent issue.
I want to open #10494 to discuss it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants