Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stat: Fix output when stat a prefix which has other prefixes #4501

Merged
merged 2 commits into from
Mar 23, 2023
Merged

stat: Fix output when stat a prefix which has other prefixes #4501

merged 2 commits into from
Mar 23, 2023

Conversation

vadmeste
Copy link
Member

@vadmeste vadmeste commented Mar 14, 2023 •
edited
Loading

Description

mc stat <alias>/<bucket>/<dir>/ can show bucket meta-information
when <dir>/ contains a prefix as well.

Bucket stat info is called inside the listing loop code, which is wrong
and causes bad performance issue which some prefixes are found
during stat command.

Motivation and Context

Fix stat output in some cases

How to test this PR?

mc mb play/testbucket/
mc cp file play/testbucket/dir1/
mc cp file play/testbucket/dir1/dir2/
mc stat play/testbucket/dir1/

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

@klauspost
Copy link
Contributor 

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc mb play/testbucket2/
Bucket created successfully `play/testbucket2/`.

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc cp 1b.xyz play/testbucket2/dir1/
...thub.com\minio\mc\1b.xyz: 1 B / 1 B [==========================================================================================================] 2 B/s 0s e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc cp 1b.xyz play/testbucket2/dir1/dir2/
...thub.com\minio\mc\1b.xyz: 1 B / 1 B [==========================================================================================================] 2 B/s 0s e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc stat play/testbucket2/dir1/
mc: <ERROR> Unable to stat `play/testbucket2/dir1/`. Target `play/testbucket2/dir1/` not found.

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ
`λ mc stat --debug play/testbucket2/dir1/` output 
mc: <DEBUG> GET /testbucket2/?location= HTTP/1.1
Host: play.min.io
User-Agent: MinIO (windows; amd64) minio-go/v7.0.49 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230321/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230321T145002Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 137
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 21 Mar 2023 14:50:02 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 174E76AE7F1E1C83
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2023-06-18 23:03:53 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2025-09-15 16:00:00 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2024-09-30 18:14:03 +0000 UTC
mc: <DEBUG> Response Time:  542.176ms

mc: <DEBUG> GET /testbucket2/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=dir1%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (windows; amd64) minio-go/v7.0.49 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230321/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230321T145002Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 716
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 21 Mar 2023 14:50:03 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 174E76AE87E946FA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2023-06-18 23:03:53 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2025-09-15 16:00:00 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2024-09-30 18:14:03 +0000 UTC
mc: <DEBUG> Response Time:  149.8447ms

mc: <DEBUG> GET /testbucket2/?continuation-token=ZGlyMS8xYi54eXpbbWluaW9fY2FjaGU6djIscmV0dXJuOl0%3D&delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&max-keys=1&prefix=dir1%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (windows; amd64) minio-go/v7.0.49 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230321/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230321T145002Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 447
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 21 Mar 2023 14:50:03 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 174E76AE90E58166
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2023-06-18 23:03:53 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2025-09-15 16:00:00 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2024-09-30 18:14:03 +0000 UTC
mc: <DEBUG> Response Time:  150.5422ms

mc: <DEBUG> GET /testbucket2/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=dir1%2F HTTP/1.1
Host: play.min.io
User-Agent: MinIO (windows; amd64) minio-go/v7.0.49 mc/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=**REDACTED**/20230321/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230321T145002Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 685
Accept-Ranges: bytes
Connection: keep-alive
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Tue, 21 Mar 2023 14:50:03 GMT
Server: nginx/1.18.0 (Ubuntu)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Request-Id: 174E76AE99C71BFF
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2023-06-18 23:03:53 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Internet Security Research Group
mc: <DEBUG>  >> Expires: 2025-09-15 16:00:00 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2024-09-30 18:14:03 +0000 UTC
mc: <DEBUG> Response Time:  298.8499ms

mc: <ERROR> Unable to stat `play/testbucket2/dir1/`. Target `play/testbucket2/dir1/` not found.
 (0) e:/gopath/src/github.com/minio/mc/cmd/stat.go:272 cmd.statURL(..) Tags: [play/testbucket2/dir1/1b.xyz, play\testbucket2\dir1\]
 Commit:DEVELOPMENT. | Release-Tag:DEVELOPMENT.GOGET | Host:DESKTOP-DM27GLF | OS:windows | Arch:amd64 | Lang:go1.19.5 | Mem:3.5 MiB/18 MiB | Heap:3.5 MiB/7.4 MiB.

stat: Fix output when stat a prefix which has other prefixes
e48c403 
mc stat <alias>/<bucket>/<dir>/ can show bucket meta-information when
<dir>/ contains a prefix as well.

Bucket stat info is called inside the listing loop code, which is wrong
and causes bad performance issue which some prefixes are found during
stat command.
@vadmeste
Copy link
Member Author

vadmeste commented Mar 21, 2023 •
edited
Loading

@klauspost I am not able to reproduce the issue. It looks like it is a Windows thing. I rebased the PR with master, can you try again ?

@klauspost
Copy link
Contributor

@vadmeste Great success!

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ SET BUCKET=testbucket3

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc mb play/%BUCKET%/
Bucket created successfully `play/testbucket3/`.

e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc cp 1b.xyz play/%BUCKET%/dir1/
...thub.com\minio\mc\1b.xyz: 1 B / 1 B [==========================================================================================================] 2 B/s 0s e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc cp 1b.xyz play/%BUCKET%/dir1/dir2/
...thub.com\minio\mc\1b.xyz: 1 B / 1 B [==========================================================================================================] 2 B/s 0s e:\gopath\src\github.com\minio\mc (stat -> vadmeste)
λ mc stat play/%BUCKET%/dir1/
Name      : 1b.xyz
Date      : 2023-03-22 11:00:00 CET
Size      : 1 B
ETag      : 0cc175b9c0f1b6a831c399e269772661
Type      : file
Metadata  :
  Content-Type: chemical/x-xyz

Name      : dir2/
Date      : 2023-03-22 11:00:06 CET
Type      : folder

klauspost
klauspost approved these changes Mar 22, 2023
View reviewed changes
Copy link
Contributor

@klauspost klauspost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Tested on Windows.

@harshavardhana harshavardhana merged commit 81453d7 into minio:master Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@harshavardhana harshavardhana harshavardhana approved these changes

@klauspost klauspost klauspost approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vadmeste @klauspost @harshavardhana