Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: all levels deep flat key match #12996

Merged
merged 1 commit into from
Aug 18, 2021
Merged

fix: all levels deep flat key match #12996

merged 1 commit into from
Aug 18, 2021

Conversation

harshavardhana
Copy link
Member

Description

fix: all levels deep flat key match

Motivation and Context

this addresses a regression from #12984
which only addresses flat keys from single
level deep at the bucket level.

added extra tests as well to cover all
these scenarios.

How to test this PR?

Create the following hierarchy

~ mc tree -f myminio/testbucket/x/
myminio/testbucket/x/
├─ 201910_x
├─ 201911_x
└─ 201910
   ├─ 1220
   ├─ 1221
   ├─ 2221
   └─ x
      └─ 1

Then attempt to list with flat key match

~ aws s3api list-objects-v2 --bucket testbucket --prefix x/201910/1 --profile minio --endpoint-url http://localhost:9000

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

harshavardhana
harshavardhana commented Aug 18, 2021
View reviewed changes
Copy link
Member Author

@harshavardhana harshavardhana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@klauspost PTAL

cmd/metacache-walk.go
if len(prefix) > 0 && !strings.HasPrefix(entry, prefix) {
// Do do not retain the file, since it doesn't
// match the prefix.
entries[i] = ""
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This felt to be right since we do not need to stack this up again in dirStack() as we do not need to go further down if there is no prefix match

cmd/metacache-walk.go
if len(forward) > 0 && entry < forward {
// Do do not retain the file, since it doesn't
// is lexically smaller than 'forward'
entries[i] = ""
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This felt to be right since we do not need to stack this up again in dirStack() as we do not need to go further down if lexically smaller than forward

@harshavardhana
fix: all levels deep flat key match
0670ba2 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
@minio-trusted
Copy link
Contributor

Mint Automation

Test Result
mint-large-bucket.sh ✔️
mint-fs.sh ✔️
mint-gateway-s3.sh ✔️
mint-erasure.sh ✔️
mint-dist-erasure.sh ✔️
mint-zoned.sh ✔️
mint-gateway-nas.sh ✔️
mint-compress-encrypt-dist-erasure.sh ✔️
Deleting image on docker hub
Deleting image locally

klauspost
klauspost approved these changes Aug 18, 2021
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@harshavardhana harshavardhana merged commit 9c65168 into minio:master Aug 18, 2021
@harshavardhana harshavardhana deleted the fix-metacache branch August 18, 2021 14:40
harshavardhana added a commit that referenced this pull request Sep 22, 2021
@harshavardhana
fix: all levels deep flat key match (#12996)
2894299 
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
harshavardhana added a commit that referenced this pull request Sep 22, 2021
@harshavardhana
fix: all levels deep flat key match (#12996)
a3c3659 
this addresses a regression from #12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
gergan pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 1, 2021
@harshavardhana @gergan
fix: all levels deep flat key match (minio#12996)
5c84341 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.
# Conflicts:
#	cmd/metacache-walk.go
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 1, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
aa08650 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
7898fee 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
a2ba0fa 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 7, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
55453d5 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 11, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
1fc3d63 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 13, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
792ff35 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
tristanessquare pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 13, 2021
@harshavardhana @tristanessquare
fix: all levels deep flat key match (minio#12996)
ce9baf5 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
rluetzner pushed a commit to iternity-dotcom/minio that referenced this pull request Oct 14, 2021
@harshavardhana @rluetzner
fix: all levels deep flat key match (minio#12996)
a7d6801 
this addresses a regression from minio#12984
which only addresses flat key from single
level deep at bucket level.

added extra tests as well to cover all
these scenarios.

---

Additionally had to update prefix handling in fs-v1-storage.go in order to make the new test cases run in nas mode.
Prefix-check in fsStorage.WalkDir is now identical to xlStorage.WalkDir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@klauspost klauspost klauspost approved these changes

@vadmeste vadmeste Awaiting requested review from vadmeste

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@harshavardhana @minio-trusted @klauspost