This repository has been archived by the owner on Oct 6, 2020. It is now read-only.
pulling in upstream #3
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Streaming signature V4 calculation is implemented as an io.Reader. It reads at most 64KB (library constant) at a time. Also, added PutObjectStreaming API in minio-go SDK to upload an object using streaming signature v4.
This is implemented to address the problems of applications wanting to validate the entire bucket policy in a custom manner. Fixes #659 Refer minio/minio#4131
- Detect size automatically like other PutObject() operations. - Allow progress bar to be passed into PutObjectStreaming(). - Allow also metadata to be passed into PutObjectStreaming(). - Rename NewStreamingV4 to just StreamingV4(). Keeping it consistent with other signature methods.
Additionally this PR cleans up spaces.
GetBucketPolicy should return exact error received from the server and let the caller manage it. Fixes #664
Implements a new API to provide a way to set headers for GetObject(), StatObject() request such as to - read partial data starting at offsets. - read only if etag matches. - read only if modtime matches. - read only if etag doesn't match. - read only if modtime doesn't match. Fixes #669
Thanks @krishnasrinivas for the simple approach. - Add a functional test case to confirm the fix.
Also adds tests
This PR adds a new API
- NewWithCredentials()
Internally NewWithCredentials is now used with
all APIs such as New(), NewV4(), NewV2() and NewWithRegion.
Also brings a new package called `credentials` to manage
various credentials type, currently the credentials
package supports
- Reading file from `.aws/credentials`, `.mc/config.json`
- Reading env variables for AWS*, MINIO*
- Fetching from IAM roles assigned to an EC2 instance.
- Static credentials which is the current default behavior.
Example code using IAM.
```go
iam := credentials.NewIAM("")
s3Client, err := minio.NewWithCredentials("s3.amazonaws.com", iam, true, "")
if err != nil {
log.Fatalln(err)
}
buckets, err := s3Client.ListBuckets()
if err != nil {
log.Fatalln(err)
}
for _, bucket := range buckets {
log.Println(bucket)
}
```
Fixes #643
Avoid using random source when generating large data. This will heavily reduces functional tests time.
In situations where a file has been Seeked, we need to start reading from the offset which it indeed happens. But our reader size calculation needs to honor this to be accurate. Fixes #680
This PR adds an example to showcase how one can use AWS S3 to enable Server-Side Encryption with Customer-Provided Encryption Keys (SSE-C)
…encrypted-object (#697)
Getting an object and reading it doesn't work as expected when S3 server doesn't return the object content length, which happens sometimes with Google Cloud Service.
We should implement `io.Closer` for the encryption materials to close the underlying reader whenever a caller requests. GetEncryptedObject() should have returned `io.ReadCloser` in the first place.
This code simplification also addresses bugs which can arise from issues like #702 where error handling in multiple conditions is cryptic. Benefit of this fix is also that we are not going to buffer memory if there are lot of copy operations, since our min-part size has increased to 64MB using bytes buffer may cause each file copy to use 128MB worth of memory due to `bytes.Buffer` growth strategy used in go standard library. Incidentally this change fixes #702
Current code was returning without validating te origAuth value correctly. Fixes #705
HTTP clients like browsers or curl automatically strip port 80 and 443 in Host header. This PR makes minio-go follow the same behavior so the generated presigned urls can work without having signature mismatch error.
This change allows for the GCS server to reject if needed. For size == -1 we just use `Transfer-Encoding: chunked` For size >= 0 we just use regular upload operation. Related to restic/restic#996
Add a stricter check for MakeBucket alone to avoid creating newer buckets with capitalized letters. Fixes minio/mc#2157
Typo in the message for "NoSuchBucket". It is missing "." Fixes #1034
when resp 404 Not Found, BucketExists error should be nil, exists==false Fixes #1036
Since gofmt changed in go1.11 its not possible to run builds for 1.8 and 1.9 anymore, removing them.
This is the same cleanup which is already being performed on the other buckets created during the test run.
The [S3 dual-stack endpoints][1] map against both A and AAAA records, allowing the client to connect using either IPv4 or IPv6, depending on what is locally available. At this point there appear to be no IPv6 support for the China regions. Related to restic/restic#2129. [1]: https://docs.aws.amazon.com/AmazonS3/latest/dev/dual-stack-endpoints.html
Both provider implements a way to retrieve temporary credentials from Minio STS service - using client grants token (Only Minio) - using web identity token (Both Minio and AWS) These temporary credentials will be used to perform API operations, to be used with applications which are never using static credentials.
We have never set values which are empty on the request
because they are perhaps not useful in the List query,
but this assumption is wrong when there are restricted
policies for a given user, because empty is actually
a valid value in IAM or Bucket policy conditions.
For example following condition would never work with our
ListObjects call and AWS cli would work fine.
```json
"Condition": {
"StringEquals": {
"s3:prefix": [
"",
"data/",
"data"
],
"s3:delimiter": [
"/",
""
]
}
}
```
The reason is empty or not `prefix` and `delimiter` should be
added to the query param in List operation, such that server
can use the value to validate the policies for the incoming
request.
Fixes minio/mc#2647
ErrorResponse.Headers is causing a panic inside golang http package.
The following code can be easily crashed if GetObject API returns an error,
such as ErrSlowDown:
```
reader, err := s3Client.GetObject("my-bucketname", "my-source-objectname", minio.GetObjectOptions{})
if err != nil {
log.Fatalln(err)
}
defer reader.Close()
stat, err := reader.Stat()
if err != nil {
log.Fatalln(err)
}
n, err := s3Client.PutObject("my-bucketname", "my-target-objectname", reader, stat.Size, minio.PutObjectOptions{})
if err != nil {
log.Fatalln(err)
}
```
The reason is that `reader` is passed to s3Client.PutObject therefore to golang http
via request.Body. Since, reader.Read() can return an ErrorResponse, that error will
be tested by golang http, but since ErrorResponse is uncomparable, then it will cause
a crash with the following stackstrace:
```
panic: runtime error: comparing uncomparable type minio.ErrorResponse
goroutine 20 [running]:
net/http.(*Request).write(0xc0002c2300, 0x761e60, 0xc000069f80, 0x0, 0x0, 0x0, 0x7628a0, 0xc000518780)
/home/vadmeste/work/go/src/net/http/request.go:647 +0x74c
net/http.(*persistConn).writeLoop(0xc0000a17a0)
/home/vadmeste/work/go/src/net/http/transport.go:1888 +0x1b8
created by net/http.(*Transport).dialConn
/home/vadmeste/work/go/src/net/http/transport.go:1339 +0x966
exit status 2
```
Hence, removing Headers since it is a map and it is uncomparable.
Trim extra spaces in signed header values for creating canonical request string for v4 signature. As per S3 spec: https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.