Skip to content

Conversation

@MonkeyCanCode
Copy link
Contributor

Description

This is for adding doc around which operations are not supported by credential vending.

Fixes # (390)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • Documentation update
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

Please delete options that are not relevant.

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue


- **remove_orphan_files:** This operation does not currently support
credential vending when utilizing Apache Spark as the compute engine.
- For more details, please refer to [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be using the new Hugo-style links? I am honestly not clear on that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eric-maynard is right, I think it's better to use Hugo style link to have a clean documentation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the info. Will finish validations later today then update the PR with suggestions above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From some quick read, Hugo style link is mainly for internal links within the site and external links seems to be preferred as markdown's style instead. Also, there are many places we are using markdown style links for external reference across current site. If that is more preferred, should we do it in a different PR instead?

Then in terms of table maintenance operations validation with credential vending, following are tested:

Trino:

  • optimize: passed
  • expire_snapshots: passed
  • remove_orphan_files: passed
  • drop_extended_stats: passed

Spark:

  • expire_snapshots: passed
  • remove_orphan_files: failed (not supported as mentioned in this PR)
  • rewrite_data_files: passed
  • rewrite_manifests: passed
  • rewrite_position_delete_files: passed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MonkeyCanCode thanks a lot for validating maintenance procedures. It's a good callout, but it is essentially an Spark issue. Maybe we can reword to something like this:

- **remove_orphan_files:** Apache Spark can't use credential vending for this due to a known issue. See [apache/iceberg#7914](https://github.com/apache/iceberg/pull/7914) for details. 

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anytime. Updated.

@flyrain flyrain enabled auto-merge (squash) October 24, 2024 20:18
@flyrain flyrain merged commit 301dc14 into apache:main Oct 24, 2024
@MonkeyCanCode MonkeyCanCode deleted the 390 branch December 12, 2024 04:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants