Skip to content

[MINOR] Use parent as the glob path when full file path specified#11150

Merged
danny0405 merged 2 commits intoapache:masterfrom
the-other-tim-brown:glob-paths-usage
May 8, 2024
Merged

[MINOR] Use parent as the glob path when full file path specified#11150
danny0405 merged 2 commits intoapache:masterfrom
the-other-tim-brown:glob-paths-usage

Conversation

@the-other-tim-brown
Copy link
Contributor

Change Logs

  • Fix usages of the glob paths to take in partition level paths instead of file level paths in clustering and metadata writing

Impact

  • Fixes a bug where we see listing calls per file instead of per partition

Risk level (write none, low medium or high below)

low

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

  • The config description must be updated if new configs are added or the default value of the configs are changed
  • Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
    ticket number here and follow the instruction to make
    changes to the website.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:XS PR with lines of changes in <= 10 label May 4, 2024
@the-other-tim-brown the-other-tim-brown marked this pull request as ready for review May 4, 2024 19:34
@hudi-bot
Copy link
Collaborator

hudi-bot commented May 4, 2024

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

// Building HoodieFileIndex needs this param to decide query path
params.put("glob.paths", readPathString);
params.put("glob.paths", globPathString);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have any test cases?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find a test class matching this class name. Is there a clustering test suite I should look in?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure whether TestHoodieSparkMergeOnReadTableClustering is the candidate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it is already covered there

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine, let's merge it first.

@danny0405 danny0405 merged commit 519b78d into apache:master May 8, 2024
TheR1sing3un pushed a commit to TheR1sing3un/hudi that referenced this pull request Feb 12, 2025
…ry-pick

1. introduce bucket-resizing procedure
2. optimize the configuration content when performing clustering with row writer, from apache#11614
3. use parent as the glob path when full file path specified, from apache#11150
4. resolve conflicts after cherry-pick

Signed-off-by: liuchaoyang03 <liuchaoyang03@kuaishou.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-0.15.0 size:XS PR with lines of changes in <= 10

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants