Skip to content

Commit

Permalink
feat: Add system metadata for PUT operations
Browse files Browse the repository at this point in the history
  • Loading branch information
@oxyno-zeta
oxyno-zeta committed Feb 9, 2023
1 parent 8a4a336 commit 2d261b3
Show file tree
Hide file tree
Showing 10 changed files with 346 additions and 69 deletions.
14 changes: 14 additions & 0 deletions conf/config-example.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,20 @@ targets:
# # Values can be templated. Empty values will be flushed.
# metadata:
# key: value
# # System Metadata cases.
# # Values can be templated. Empty values will be flushed.
# systemMetadata:
# # Cache-Control value (will be put as header after)
# cacheControl: ""
# # Content-Disposition value (will be put as header after)
# contentDisposition: ""
# # Content-Encoding value (will be put as header after)
# contentEncoding: ""
# # Content-Language value (will be put as header after)
# contentLanguage: ""
# # Expires value (will be put as header after)
# # Side note: This must have the RFC3339 date format at the end.
# expires: ""
# # Storage class that will be used for uploaded objects
# # See storage class here: https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html
# # Values can be templated. Empty values will be flushed.
Expand Down
14 changes: 14 additions & 0 deletions docs/configuration/example.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,20 @@ targets:
# # Values can be templated. Empty values will be flushed.
# metadata:
# key: value
# # System Metadata cases.
# # Values can be templated. Empty values will be flushed.
# systemMetadata:
# # Cache-Control value (will be put as header after)
# cacheControl: ""
# # Content-Disposition value (will be put as header after)
# contentDisposition: ""
# # Content-Encoding value (will be put as header after)
# contentEncoding: ""
# # Content-Language value (will be put as header after)
# contentLanguage: ""
# # Expires value (will be put as header after)
# # Side note: This must have the RFC3339 date format at the end.
# expires: ""
# # Storage class that will be used for uploaded objects
# # See storage class here: https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html
# # Values can be templated. Empty values will be flushed.
Expand Down
23 changes: 17 additions & 6 deletions docs/configuration/structure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -255,12 +255,23 @@ See more information [here](../feature-guide/key-rewrite.md).

## PutActionConfigConfiguration

| Key | Type | Required | Default | Description |
| ------------- | ----------------------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects. Map Values can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| storageClass | String | No | `""` | Storage class that will be used for uploaded objects. See storage class here: [https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html). Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-storage-class) |
| allowOverride | Boolean | No | `false` | Will allow override objects if enabled |
| webhooks | [[WebhookConfiguration](#webhookconfiguration)] | No | `nil` | Webhooks configuration list to call when a PUT request is performed |
| Key | Type | Required | Default | Description |
| -------------- | ----------------------------------------------------------------------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| metadata | Map[String]String | No | None | Metadata key/values that will be put on S3 objects. Map Values can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| systemMetadata | [PutActionConfigSystemMetadataConfiguration](#putactionconfigsystemmetadataconfiguration) | No | `nil` | This allow to put system metadata values to uploaded objects. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| storageClass | String | No | `""` | Storage class that will be used for uploaded objects. See storage class here: [https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html](https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html). Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-storage-class) |
| allowOverride | Boolean | No | `false` | Will allow override objects if enabled |
| webhooks | [[WebhookConfiguration](#webhookconfiguration)] | No | `nil` | Webhooks configuration list to call when a PUT request is performed |

## PutActionConfigSystemMetadataConfiguration

| Key | Type | Required | Default | Description |
| ------------------ | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| cacheControl | String | `""` | Cache-Control value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| contentDisposition | String | `""` | Content-Disposition value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| contentEncoding | String | `""` | Content-Encoding value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| contentLanguage | String | `""` | Content-Language value. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |
| expires | String | `""` | Expires value This must have the RFC3339 date format at the end. Value can be templated. Empty values will be flushed. See [here](../feature-guide/templates.md#put-metadata) |

## DeleteActionConfiguration

Expand Down
143 changes: 122 additions & 21 deletions pkg/s3-proxy/bucket/requestContext.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"io"
"path"
"strings"
"time"

"emperror.dev/errors"

Expand Down Expand Up @@ -344,6 +345,98 @@ func (rctx *requestContext) Put(ctx context.Context, inp *PutInput) {
// Check if post actions configuration exists
if rctx.targetCfg.Actions.PUT != nil &&
rctx.targetCfg.Actions.PUT.Config != nil {
// Check if system metadata is defined
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata != nil {
// Manage cache control
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata.CacheControl != "" {
// Execute template
val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.SystemMetadata.CacheControl)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}
// Check if value is empty or not
if val != "" {
// Store
input.CacheControl = val
}
}
// Manage content disposition
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentDisposition != "" {
// Execute template
val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentDisposition)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}
// Check if value is empty or not
if val != "" {
// Store
input.ContentDisposition = val
}
}
// Manage content encoding
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentEncoding != "" {
// Execute template
val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentEncoding)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}
// Check if value is empty or not
if val != "" {
// Store
input.ContentEncoding = val
}
}
// Manage content language
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentLanguage != "" {
// Execute template
val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.SystemMetadata.ContentLanguage)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}
// Check if value is empty or not
if val != "" {
// Store
input.ContentLanguage = val
}
}
// Manage content language
if rctx.targetCfg.Actions.PUT.Config.SystemMetadata.Expires != "" {
// Execute template
val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.SystemMetadata.Expires)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}
// Check if value is empty or not
if val != "" {
// Parse
d, err3 := time.Parse(time.RFC3339, val)
// Check error
if err3 != nil {
resHan.InternalServerError(rctx.LoadFileContent, errors.WithStack(err3))

return
}
// Store
input.Expires = &d
}
}
}

// Check if metadata is configured in target configuration
if rctx.targetCfg.Actions.PUT.Config.Metadata != nil {
// Store templated data
Expand All @@ -352,22 +445,13 @@ func (rctx *requestContext) Put(ctx context.Context, inp *PutInput) {
// Render templates
for k, v := range rctx.targetCfg.Actions.PUT.Config.Metadata {
// Execute template
buf, err2 := templateutils.ExecuteTemplate(v, &PutData{
User: models.GetAuthenticatedUserFromContext(ctx),
Input: inp,
Key: key,
})
val, err2 := rctx.tplPutData(ctx, inp, key, v)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}

// Store value
val := buf.String()
// Remove all new lines
val = utils.NewLineMatcherRegex.ReplaceAllString(val, "")
// Check if value is empty or not
if val != "" {
// Store
Expand All @@ -382,23 +466,13 @@ func (rctx *requestContext) Put(ctx context.Context, inp *PutInput) {
// Check if storage class is present in target configuration
if rctx.targetCfg.Actions.PUT.Config.StorageClass != "" {
// Execute template
buf, err2 := templateutils.ExecuteTemplate(rctx.targetCfg.Actions.PUT.Config.StorageClass, &PutData{
User: models.GetAuthenticatedUserFromContext(ctx),
Input: inp,
Key: key,
})

val, err2 := rctx.tplPutData(ctx, inp, key, rctx.targetCfg.Actions.PUT.Config.StorageClass)
// Check error
if err2 != nil {
resHan.InternalServerError(rctx.LoadFileContent, err2)

return
}

// Store value
val := buf.String()
// Remove all new lines
val = utils.NewLineMatcherRegex.ReplaceAllString(val, "")
// Check if value is empty or not
if val != "" {
// Store
Expand Down Expand Up @@ -473,6 +547,33 @@ func (rctx *requestContext) Put(ctx context.Context, inp *PutInput) {
)
}

func (rctx *requestContext) tplPutData(ctx context.Context, inp *PutInput, key, tplStr string) (string, error) {
// Execute template
buf, err := templateutils.ExecuteTemplate(tplStr, &PutData{
User: models.GetAuthenticatedUserFromContext(ctx),
Input: inp,
Key: key,
})

// Check error
if err != nil {
return "", errors.WithStack(err)
}

// Store value
val := buf.String()
// Remove all new lines
val = utils.NewLineMatcherRegex.ReplaceAllString(val, "")
// Check if value is empty or not
if val != "" {
// Store
return val, nil
}

// Default
return "", nil
}

// Delete will delete object in S3.
func (rctx *requestContext) Delete(ctx context.Context, requestPath string) {
// Get response handler
Expand Down
18 changes: 14 additions & 4 deletions pkg/s3-proxy/config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -392,10 +392,20 @@ type PutActionConfig struct {

// PutActionConfigConfig Put action configuration object configuration.
type PutActionConfigConfig struct {
Metadata map[string]string `mapstructure:"metadata"`
StorageClass string `mapstructure:"storageClass"`
AllowOverride bool `mapstructure:"allowOverride"`
Webhooks []*WebhookConfig `mapstructure:"webhooks" validate:"dive"`
Metadata map[string]string `mapstructure:"metadata"`
SystemMetadata *PutActionConfigSystemMetadataConfig `mapstructure:"systemMetadata"`
StorageClass string `mapstructure:"storageClass"`
AllowOverride bool `mapstructure:"allowOverride"`
Webhooks []*WebhookConfig `mapstructure:"webhooks" validate:"dive"`
}

// PutActionConfigSystemMetadataConfig Put action configuration system metadata object configuration.
type PutActionConfigSystemMetadataConfig struct {
CacheControl string `mapstructure:"cacheControl"`
ContentDisposition string `mapstructure:"contentDisposition"`
ContentEncoding string `mapstructure:"contentEncoding"`
ContentLanguage string `mapstructure:"contentLanguage"`
Expires string `mapstructure:"expires"`
}

// GetActionConfig Get action configuration.
Expand Down
17 changes: 11 additions & 6 deletions pkg/s3-proxy/s3client/client.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -106,12 +106,17 @@ type GetOutput struct {

// PutInput Put input object for PUT request.
type PutInput struct {
Key string
Body io.ReadSeeker
ContentType string
ContentSize int64
Metadata map[string]string
StorageClass string
Key string
Body io.ReadSeeker
ContentType string
ContentSize int64
Metadata map[string]string
StorageClass string
CacheControl string
ContentDisposition string
ContentEncoding string
ContentLanguage string
Expires *time.Time
}

// NewManager will return a new S3 client manager.
Expand Down
17 changes: 17 additions & 0 deletions pkg/s3-proxy/s3client/s3Context.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -324,8 +324,25 @@ func (s3ctx *s3Context) PutObject(ctx context.Context, input *PutInput) (*Result
ContentLength: aws.Int64(input.ContentSize),
Bucket: aws.String(s3ctx.target.Bucket.Name),
Key: aws.String(input.Key),
Expires: input.Expires,
}

// Manage cache control case
if input.CacheControl != "" {
inp.CacheControl = aws.String(input.CacheControl)
}
// Manage content disposition case
if input.ContentDisposition != "" {
inp.ContentDisposition = aws.String(input.ContentDisposition)
}
// Manage content encoding case
if input.ContentEncoding != "" {
inp.ContentEncoding = aws.String(input.ContentEncoding)
}
// Manage content language case
if input.ContentLanguage != "" {
inp.ContentLanguage = aws.String(input.ContentLanguage)
}
// Manage content type case
if input.ContentType != "" {
inp.ContentType = aws.String(input.ContentType)
Expand Down

0 comments on commit 2d261b3

Please sign in to comment.