Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

always set the filter to skip navigating #12984

Merged
merged 1 commit into from Aug 17, 2021
Merged

always set the filter to skip navigating #12984

merged 1 commit into from Aug 17, 2021

Conversation

harshavardhana
Copy link
Member

Description

always set the filter to skip navigating

Motivation and Context

baseDir is empty if the top level prefix does not
end with / this causes large recursive listings
without any filtering, to fix this filtering make
sure to set the filter prefix appropriately.

also do not navigate folders at top level that do
not match the filter prefix, entries don't need
to match prefix since they are never prefixed
with the prefix anyways.

How to test this PR?

Create various prefixes at top level and observe
the directories that get skipped which do not match
the expected prefix.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Documentation updated
  • Unit tests added/updated

@harshavardhana
always set the filter to skip navigating baseDir
cca16a4 
baseDir is empty if the top level prefix does not
end with `/` this causes large recursive listings
without any filtering, to fix this filtering make
sure to set the filter prefix appropriately.

also do not navigate folders at top level that do
not match the filter prefix, entries don't need
to match prefix since they are never prefixed
with the prefix anyways.
harshavardhana
harshavardhana commented Aug 17, 2021
View reviewed changes
cmd/metacache-walk.go Show resolved Hide resolved
cmd/metacache-walk.go Show resolved Hide resolved
cmd/metacache-server-pool.go Show resolved Hide resolved
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh ✔️
Deleting image on docker hub
Deleting image locally

klauspost
klauspost approved these changes Aug 17, 2021
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼

vadmeste
vadmeste approved these changes Aug 17, 2021
View reviewed changes
Copy link
Member

@vadmeste vadmeste left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit 654a6e9 into minio:master Aug 17, 2021
@harshavardhana harshavardhana deleted the fix-filter branch August 17, 2021 14:43
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 18, 2021
@harshavardhana
fix: all levels deep flat key match
6de0bc0 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
@harshavardhana harshavardhana mentioned this pull request Aug 18, 2021
Merged
7 tasks
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 18, 2021
@harshavardhana
fix: all levels deep flat key match
91b45e2 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
harshavardhana added a commit to harshavardhana/minio that referenced this pull request Aug 18, 2021
@harshavardhana
fix: all levels deep flat key match
0670ba2 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
harshavardhana added a commit that referenced this pull request Aug 18, 2021
@harshavardhana
fix: all levels deep flat key match (#12996)
9c65168 
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
harshavardhana added a commit that referenced this pull request Sep 22, 2021
@harshavardhana
fix: all levels deep flat key match (#12996)
2894299 
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
harshavardhana added a commit that referenced this pull request Sep 22, 2021
@harshavardhana
fix: all levels deep flat key match (#12996)
a3c3659 
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
gergan pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 1, 2021
@harshavardhana @gergan
fix: all levels deep flat key match (minio#12996)
5c84341 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
# Conflicts:
#	cmd/metacache-walk.go
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 1, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
aa08650 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
7898fee 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
a2ba0fa 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
55453d5 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 11, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
1fc3d63 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 13, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
792ff35 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 13, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
ce9baf5 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
rluetzner pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 14, 2021
@harshavardhana @rluetzner
fix: all levels deep flat key match (minio#12996)
a7d6801 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vadmeste vadmeste vadmeste approved these changes

@klauspost klauspost klauspost approved these changes

Assignees
No one assigned
Labels
priority: high
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@harshavardhana @minio-trusted @vadmeste @klauspost