-
Notifications
You must be signed in to change notification settings - Fork 148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pass size of source while calling put for storage bucket from kando #809
Conversation
We were passing the object size 0 when we called `.put` on the bucket of object storage and because of that `stow` was not able to figure out the source size and eventually multi part upload was not being triggered.
Please ignore the |
pkg/location/location.go
Outdated
@@ -97,7 +98,12 @@ func writeData(ctx context.Context, pType objectstore.ProviderType, profile para | |||
if err != nil { | |||
return err | |||
} | |||
if err := bucket.Put(ctx, path, in, 0, nil); err != nil { | |||
b, err := ioutil.ReadAll(in) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Won't we end up reading the data twice?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah, we discussed this in yesterday's call. We unknowingly introduced another issue here. After reading the in
is going to be nil
. I am working on this.
570b7bb
to
58e4248
Compare
This requires the version of |
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Unit tests run after updating
|
|
pkg/location/location_test.go
Outdated
// Create dump file | ||
err := os.Truncate(s.testMultipartPath, fileSize) | ||
c.Assert(err, IsNil) | ||
testcontent, err := ioutil.ReadFile(s.testMultipartPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
os.Open()
will give you a reader - you don't need to get the content in-memory then create a bytes.Buffer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now, we are re-using file pointer created at https://github.com/kanisterio/kanister/pull/809/files/147b928221ac56008f56f83ce76f2e525c764eee#diff-c1b47cb14ad134412666df8eff6311f52edc385a3e6ffa6a15564f24ec9b297bR150
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Signed-off-by: PrasadG193 <prasad.ghangal@gmail.com>
Change Overview
We were passing the object size
0
when we called.put
onthe bucket of object storage and because of that whenever
stow
tried to check if source size is more than the allowed size we didn't
get expected result.
This PR tries to fix that by passing the correct object size while calling
put
.Depends on: #809
Pull request type
Please check the type of change your PR introduces:
Issues
Test Plan
I tried to upload the file of size
1000M
, if the size is greater that256M
multipart upload for stow would be called.