Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 multipart enchancement #7350

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

S3 multipart enchancement #7350

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
* [ENHANCEMENT] Store-gateway: add `-blocks-storage.bucket-store.max-concurrent-queue-timeout`. When set, queries at the store-gateway's query gate will not wait longer than that to execute. If a query reaches the wait timeout, then the querier will retry the blocks on a different store-gateway. If all store-gateways are unavailable, then the query will fail with `err-mimir-store-consistency-check-failed`. #7777
* [ENHANCEMENT] Ingester: Optimize querying with regexp matchers. #8106
* [ENHANCEMENT] Distributor: Introduce `-distributor.max-request-pool-buffer-size` to allow configuring the maximum size of the request pool buffers. #8082
* [ENHANCEMENT] Storage Provider: add `-<prefix>.s3.disable-multipart` flag to disable multipart uploads. #7350
* [BUGFIX] Rules: improve error handling when querier is local to the ruler. #7567
* [BUGFIX] Querier, store-gateway: Protect against panics raised during snappy encoding. #7520
* [BUGFIX] Ingester: Prevent timely compaction of empty blocks. #7624
Expand Down
4 changes: 4 additions & 0 deletions docs/sources/mimir/configure/configuration-parameters/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4677,6 +4677,10 @@ The s3_backend block configures the connection to Amazon S3 object storage backe
# CLI flag: -<prefix>.s3.native-aws-auth-enabled
[native_aws_auth_enabled: <boolean> | default = false]

# (experimental) Whether or not to disable multipart updloads.
# CLI flag: -<prefix>.s3.disable-multipart
[disable_multipart: <boolean> | default = false]
fatpat marked this conversation as resolved.
Show resolved Hide resolved

# (experimental) The minimum file size in bytes used for multipart uploads. If
# 0, the value is optimally computed for each object.
# CLI flag: -<prefix>.s3.part-size
Expand Down
1 change: 1 addition & 0 deletions pkg/storage/bucket/s3/bucket_client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ func newS3Config(cfg Config) (s3.Config, error) {
ListObjectsVersion: cfg.ListObjectsVersion,
BucketLookupType: cfg.BucketLookupType,
AWSSDKAuth: cfg.NativeAWSAuthEnabled,
DisableMultipart: cfg.DisableMultipart,
PartSize: cfg.PartSize,
HTTPConfig: s3.HTTPConfig{
IdleConnTimeout: model.Duration(cfg.HTTP.IdleConnTimeout),
Expand Down
2 changes: 2 additions & 0 deletions pkg/storage/bucket/s3/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ type Config struct {
BucketLookupType s3.BucketLookupType `yaml:"bucket_lookup_type" category:"advanced"`
StorageClass string `yaml:"storage_class" category:"experimental"`
NativeAWSAuthEnabled bool `yaml:"native_aws_auth_enabled" category:"experimental"`
DisableMultipart bool `yaml:"disable_multipart" category:"experimental"`
PartSize uint64 `yaml:"part_size" category:"experimental"`
SendContentMd5 bool `yaml:"send_content_md5" category:"experimental"`
STSEndpoint string `yaml:"sts_endpoint"`
Expand All @@ -149,6 +150,7 @@ func (cfg *Config) RegisterFlagsWithPrefix(prefix string, f *flag.FlagSet) {
f.StringVar(&cfg.ListObjectsVersion, prefix+"s3.list-objects-version", "", "Use a specific version of the S3 list object API. Supported values are v1 or v2. Default is unset.")
f.StringVar(&cfg.StorageClass, prefix+"s3.storage-class", "", "The S3 storage class to use, not set by default. Details can be found at https://aws.amazon.com/s3/storage-classes/. Supported values are: "+strings.Join(supportedStorageClasses, ", "))
f.BoolVar(&cfg.NativeAWSAuthEnabled, prefix+"s3.native-aws-auth-enabled", false, "If enabled, it will use the default authentication methods of the AWS SDK for go based on known environment variables and known AWS config files.")
f.BoolVar(&cfg.DisableMultipart, prefix+"s3.disable-multipart", false, "Whether or not to disable multipart uploads.")
f.Uint64Var(&cfg.PartSize, prefix+"s3.part-size", 0, "The minimum file size in bytes used for multipart uploads. If 0, the value is optimally computed for each object.")
f.BoolVar(&cfg.SendContentMd5, prefix+"s3.send-content-md5", false, "If enabled, a Content-MD5 header is sent with S3 Put Object requests. Consumes more resources to compute the MD5, but may improve compatibility with object storage services that do not support checksums.")
f.Var(newBucketLookupTypeValue(s3.AutoLookup, &cfg.BucketLookupType), prefix+"s3.bucket-lookup-type", fmt.Sprintf("Bucket lookup style type, used to access bucket in S3-compatible service. Default is auto. Supported values are: %s.", strings.Join(supportedBucketLookupTypes, ", ")))
Expand Down
Loading