use tusd datastorages #7499

butonic · 2023-10-16T09:10:43Z

adds config options to configure a different prefix for the tus s3 store upload bucket

update-docs · 2023-10-16T09:10:48Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

services/storage-users/pkg/config/config.go

mmattel · 2023-10-16T09:28:29Z

We should add some explanation in the storage-users readme about prepended object prefixes to give admins a better view on it.

mmattel · 2023-11-22T15:28:30Z

services/storage-users/pkg/config/config.go

+	UploadMetadataPrefix     string `yaml:"upload_metadata_prefix" env:"STORAGE_USERS_S3NG_UPLOAD_METADATA_PREFIX" desc:"The metadata object prefix is prepended to the name of each .info and .part S3 object that is created. If it is not set, then object prefix is used."`
+	UploadTemporaryDirectory string `yaml:"upload_temporary_directory" env:"STORAGE_USERS_S3NG_UPLOAD_TEMPORARY_DIRECTORY" desc:"Path where temporary files will be stored on disk during the upload."`
+	DisableSSL               bool   `yaml:"disable_ssl" env:"STORAGE_USERS_S3NG_DISABLE_SSL" desc:"Disable SSL when accessing the S3 bucket."`
+	ForcePathStyle           bool   `yaml:"force_path_style" env:"STORAGE_USERS_S3NG_FORCE_PATH_STYLE" desc:"Force path style S3 requests."`


Suggested change

ForcePathStyle bool `yaml:"force_path_style" env:"STORAGE_USERS_S3NG_FORCE_PATH_STYLE" desc:"Force path style S3 requests."`

ForcePathStyle bool `yaml:"force_path_style" env:"STORAGE_USERS_S3NG_FORCE_PATH_STYLE" desc:"Enforce 'path style' URL requests. With 'false' which is the default, 'virtual hosted-style' URLs are used."`

I hope I got it right from the description...
medium.com and amazon.com

mmattel · 2023-11-23T08:10:28Z

services/storage-users/pkg/config/config.go

+	Bucket                   string `yaml:"bucket" env:"STORAGE_USERS_S3NG_BUCKET" desc:"Name of the S3 bucket."`
+	UploadObjectPrefix       string `yaml:"upload_object_prefix" env:"STORAGE_USERS_S3NG_UPLOAD_OBJECT_PREFIX" desc:"This object prefix is prepended to the name of each S3 object that is created to store uploaded files. It can be used to create a pseudo-directory structure in the bucket, like 'path/to/my/uploads'."`
+	UploadMetadataPrefix     string `yaml:"upload_metadata_prefix" env:"STORAGE_USERS_S3NG_UPLOAD_METADATA_PREFIX" desc:"The metadata object prefix is prepended to the name of each .info and .part S3 object that is created. If it is not set, then object prefix is used."`
+	UploadTemporaryDirectory string `yaml:"upload_temporary_directory" env:"STORAGE_USERS_S3NG_UPLOAD_TEMPORARY_DIRECTORY" desc:"Path where temporary files will be stored on disk during the upload."`


There should be a default value and written in the description, as we use on other locations like:
If not defined, the temp directory derives from $OCIS_BASE_DATA_PATH:/storage/s3ng_temp
As an example see the envvar: STORAGE_USERS_S3NG_ROOT

mmattel · 2023-11-23T08:14:33Z

services/storage-users/pkg/config/config.go

+	Endpoint                 string `yaml:"endpoint" env:"STORAGE_USERS_S3NG_ENDPOINT" desc:"Endpoint for the S3 bucket."`
+	Bucket                   string `yaml:"bucket" env:"STORAGE_USERS_S3NG_BUCKET" desc:"Name of the S3 bucket."`
+	UploadObjectPrefix       string `yaml:"upload_object_prefix" env:"STORAGE_USERS_S3NG_UPLOAD_OBJECT_PREFIX" desc:"This object prefix is prepended to the name of each S3 object that is created to store uploaded files. It can be used to create a pseudo-directory structure in the bucket, like 'path/to/my/uploads'."`
+	UploadMetadataPrefix     string `yaml:"upload_metadata_prefix" env:"STORAGE_USERS_S3NG_UPLOAD_METADATA_PREFIX" desc:"The metadata object prefix is prepended to the name of each .info and .part S3 object that is created. If it is not set, then object prefix is used."`


It is not clear what to set here and what the default is.
If it is not set, then object prefix is used. means what?
When does one needs this?

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Co-authored-by: Martin <github@diemattels.at>

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

This reverts commit c6cf6c8.

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

sonarcloud · 2023-12-05T12:27:54Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

73.7% Coverage
0.0% Duplication

butonic · 2023-12-05T16:08:35Z

I notice that we have no logging whatsoever when uploads fail to an s3 (well when handling tus uploads). We need to be able to get at least the logger config in reva ... hm maybe it is easiest to add a loglevel to the decomposedfs config.
we need s3store fix incomplete part keys tus/tusd#1042

butonic · 2024-07-29T14:46:52Z

We evolved our own DataStore into a OcisSession that also covers postprocessing.

mmattel reviewed Oct 16, 2023

View reviewed changes

services/storage-users/pkg/config/config.go Outdated Show resolved Hide resolved

mmattel reviewed Oct 16, 2023

View reviewed changes

services/storage-users/pkg/config/config.go Outdated Show resolved Hide resolved

butonic force-pushed the use-tus-data-storage branch from 7bec02a to 242808b Compare November 1, 2023 13:18

butonic force-pushed the use-tus-data-storage branch 5 times, most recently from 22d70e2 to 111a9d1 Compare November 22, 2023 14:27

mmattel reviewed Nov 22, 2023

View reviewed changes

butonic force-pushed the use-tus-data-storage branch from 111a9d1 to 313a904 Compare November 22, 2023 16:13

butonic requested review from kulmann and lookacat as code owners November 22, 2023 21:31

mmattel reviewed Nov 23, 2023

View reviewed changes

butonic force-pushed the use-tus-data-storage branch from bea34cd to 50e74e6 Compare November 23, 2023 10:58

This was referenced Nov 24, 2023

do not use existing vars cs3org/reva#4361

Closed

Fix concurrent lookup cs3org/reva#4362

Merged

butonic force-pushed the use-tus-data-storage branch 4 times, most recently from 7189083 to 278739b Compare November 24, 2023 14:26

butonic requested review from wkloucek, micbar, phil-davis and individual-it as code owners November 24, 2023 14:26

butonic force-pushed the use-tus-data-storage branch 3 times, most recently from dbac2df to 9551e4b Compare November 24, 2023 21:24

aduffeck force-pushed the use-tus-data-storage branch 3 times, most recently from 9222db5 to c844761 Compare November 29, 2023 10:32

butonic and others added 16 commits December 4, 2023 14:27

wip

96b5254

use correct reva branch

6bd6ce1

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

doc string fixes

439f6f0

Co-authored-by: Martin <github@diemattels.at>

bump reva

a82b8cd

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

log unparseable role content

10dd3d3

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

log content length and traceid in accesslog

b229354

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

increase grpc timeout

7a3c31e

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fmt.Println debugging

73ad14b

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Debug ci

d0f0e5b

Bump reva to pull in the file locking logging changes

2afcf6b

Increase proxy log level

0db95f2

Try to find the deadlock/lockup

a70ecda

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Test if the hang is related to closing the response body

92f85dd

More logging

c62ac24

Revert "Debug ci"

7b29b73

This reverts commit c6cf6c8.

change log defaults

000c944

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic force-pushed the use-tus-data-storage branch from b4674bd to 000c944 Compare December 5, 2023 09:30

butonic added 2 commits December 5, 2023 12:38

revert debugging code

944b843

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

bump reva

2573ec8

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

wkloucek mentioned this pull request Dec 7, 2023

update s3 bucket permissions for oCIS 5 owncloud/ocis-charts#446

Merged

11 tasks

butonic marked this pull request as draft December 14, 2023 09:59

wkloucek removed request for lookacat and wkloucek February 13, 2024 07:55

butonic closed this Jul 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use tusd datastorages #7499

use tusd datastorages #7499

butonic commented Oct 16, 2023

update-docs bot commented Oct 16, 2023

mmattel commented Oct 16, 2023

mmattel Nov 22, 2023 •

edited

Loading

mmattel Nov 23, 2023

mmattel Nov 23, 2023

sonarcloud bot commented Dec 5, 2023

butonic commented Dec 5, 2023 •

edited

Loading

butonic commented Jul 29, 2024

	ForcePathStyle bool `yaml:"force_path_style" env:"STORAGE_USERS_S3NG_FORCE_PATH_STYLE" desc:"Force path style S3 requests."`
	ForcePathStyle bool `yaml:"force_path_style" env:"STORAGE_USERS_S3NG_FORCE_PATH_STYLE" desc:"Enforce 'path style' URL requests. With 'false' which is the default, 'virtual hosted-style' URLs are used."`

use tusd datastorages #7499

use tusd datastorages #7499

Conversation

butonic commented Oct 16, 2023

update-docs bot commented Oct 16, 2023

mmattel commented Oct 16, 2023

mmattel Nov 22, 2023 • edited Loading

Choose a reason for hiding this comment

mmattel Nov 23, 2023

Choose a reason for hiding this comment

mmattel Nov 23, 2023

Choose a reason for hiding this comment

sonarcloud bot commented Dec 5, 2023

butonic commented Dec 5, 2023 • edited Loading

butonic commented Jul 29, 2024

mmattel Nov 22, 2023 •

edited

Loading

butonic commented Dec 5, 2023 •

edited

Loading