Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuring different max part numbers for some S3 providers #61

Closed
Erisa opened this issue Mar 22, 2022 · 2 comments · Fixed by #62 or #64
Closed

Configuring different max part numbers for some S3 providers #61

Erisa opened this issue Mar 22, 2022 · 2 comments · Fixed by #62 or #64

Comments

@Erisa
Copy link
Contributor

Erisa commented Mar 22, 2022

When setting up zfs_uploader against Scaleway Object Storage (Specifically their GLACIER tier), everything worked as expected except with one caveat: The max part number on Scaleway is 1000 rather than the 10000 used by AWS.

This resulted in an error when uploading with the default setup, since it calculated the part sizes based on 10,000 parts and eventually failed due to exceeding Scaleway's limit of 1,000 parts.

I resolved this for my use-case by simply modifying a number in job.py: Erisa@20ed42f however I feel that going forward it would be a good idea to allow configuration of this value in the zfs_uploader configuration file, and document it on the README.

You could also detect and change the values based on predefined provider limits, however it still would be nice to have the value in a user-configurable place.

@ddebeau
Copy link
Owner

ddebeau commented Mar 25, 2022

I also think it would be a good idea to let users set the value. Would you write a PR to add it to the config file and readme?

@Erisa
Copy link
Contributor Author

Erisa commented Mar 25, 2022

Sure, I can give that a go! I was planning to look into PRing this earlier but couldn't find the time, I can give it a closer look soonish.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants