Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

support read meta columns in staged scan #8872

Merged
merged 4 commits into from
Nov 16, 2023
Merged

support read meta columns in staged scan #8872

merged 4 commits into from
Nov 16, 2023

Conversation

zinking
Copy link
Contributor

@zinking zinking commented Oct 19, 2023

like in query file meta column,

it is convenient to support the following

spark.read \
  .format("csv") \
  .schema(schema) \
  .load("dbfs:/tmp/*") \
  .select("_metadata.file_name", "_metadata.file_size")

@github-actions github-actions bot added the spark label Oct 19, 2023
@zinking
Copy link
Contributor Author

zinking commented Oct 19, 2023

link the issue #8866

@aokolnychyi
Copy link
Contributor

I should be able to take a look this week.

aokolnychyi
aokolnychyi reviewed Nov 16, 2023
View reviewed changes
Copy link
Contributor

@aokolnychyi aokolnychyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good. I left some comments to address.

spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScanBuilder.java Show resolved Hide resolved
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScan.java Show resolved Hide resolved
...src/test/java/org/apache/iceberg/spark/extensions/TestMetaColumnProjectionWithStageScan.java Show resolved Hide resolved
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScanBuilder.java Outdated Show resolved Hide resolved
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScanBuilder.java Outdated Show resolved Hide resolved
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkStagedScanBuilder.java Outdated Show resolved Hide resolved
@aokolnychyi
Copy link
Contributor

One minor comment and should be good to go.

@aokolnychyi aokolnychyi merged commit bfe1d03 into apache:main Nov 16, 2023
35 checks passed
@aokolnychyi
Copy link
Contributor

Thank you, @zinking! Would you be interested to cherry-pick this change to other Spark versions?

@zinking
Copy link
Contributor Author

zinking commented Nov 16, 2023

Thank you, @zinking! Would you be interested to cherry-pick this change to other Spark versions?

sure, I will follow up.

zinking added a commit to zinking/iceberg that referenced this pull request Nov 17, 2023
@zinking
Spark 3.4 3.3: Support metadata columns in staged scan (apache#8872)
5313edd
@zinking zinking mentioned this pull request Nov 17, 2023
Merged
zinking added a commit to zinking/iceberg that referenced this pull request Nov 17, 2023
@zinking
Spark 3.4 3.3: Support metadata columns in staged scan (apache#8872)
582bcad
aokolnychyi pushed a commit that referenced this pull request Nov 18, 2023
@zinking
Spark 3.4, 3.3: Support metadata columns in staged scans (#9098)
abbfdae 
This change cherry-picks PR #8872 to Spark 3.4 and 3.3.
devangjhabakh pushed a commit to cdouglas/iceberg that referenced this pull request Apr 22, 2024
@zinking @devangjhabakh
Spark 3.5: Support metadata columns in staged scan (apache#8872)
3b4da15
devangjhabakh pushed a commit to cdouglas/iceberg that referenced this pull request Apr 22, 2024
@zinking @devangjhabakh
Spark 3.4, 3.3: Support metadata columns in staged scans (apache#9098)
b214263 
This change cherry-picks PR apache#8872 to Spark 3.4 and 3.3.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aokolnychyi aokolnychyi aokolnychyi approved these changes

Assignees
No one assigned
Labels
spark
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@zinking @aokolnychyi