Support concurrent upload and downloads to same object but different versions

[ScionOfDesign]

Expected Behavior

When uploading a new version of a versioned file, I would expect it to be able to be uploaded even if users are downloading the previous version.

Current Behavior

When uploading a new version of a file that is heavily downloaded, I get the following error:
mc: <ERROR> Failed to copy `MyFile.zip`. A timeout occurred while trying to lock a resource, please reduce your request rate

Possible Solution

Place a CDN in front of our MinIO server. Probably a preferable solution, but I don't understand why MinIO doesn't handle this situation as well.

Steps to Reproduce (for bugs)

1. Upload a large file to the server. (Ours is 500mb).
2. Make the bucket public.
3. Have many users attempt to download the file at once.
4. Try and minio cp a new version on top of the only version.
5. Observer the timeout and error.

Context

We make games with releases stored on MinIO, and occasionally release large updates. These updates lead to a spike in traffic to our MinIO server.
However, if we need to upload a hotfix, we need to restart the server or otherwise sever the active connections in order to be able to upload a new neversion. Is it possible to only have old GET requests continue downloading the old file until the new file is uploaded, (in which case the older requests would still download the old version) and have all new requests download the new file?

Regression

No

Your Environment

Minio instance installed with helm and the MInio Operator, running on Kubernetes. Traefik for routing.
Version RELEASE.2022-11-11T03-44-20Z.

[harshavardhana]

@ScionOfDesign is this a distributed setup and how big is the set-up?

From what I can tell you are concurrently modifying the object as well while downloading?

[harshavardhana]

Okay I see what you are doing, this is not allowed you cannot add new versions concurrently on top while another version is being downloaded.

Versions are a stack and are tied to the object - they do not exist independently. Any concurrent modification of a stack requires serializing them so that reads and writes get consistent results.

What you are looking for is not possible and beyond scope of this project at the moment.

[ScionOfDesign]

Oh wow, is this going to be addressed after all? Thank you!

[klauspost]

@ScionOfDesign Yes. But it has to be a safe change, which will require fundamental changes to locking.

[taai]

Okay I see what you are doing, this is not allowed you cannot add new versions concurrently on top while another version is being downloaded.

Versions are a stack and are tied to the object - they do not exist independently. Any concurrent modification of a stack requires serializing them so that reads and writes get consistent results.

What you are looking for is not possible and beyond scope of this project at the moment.

So, while there is someone downloading a file, there is no way deleting it or creating a new version (overriding it)? That's awful! Imagine someone "locking" all your objects by downloading them very slowly all day, every day! 😞

create and upload a file:

dd if=/dev/urandom of=64MB.bin bs=64M count=1 iflag=fullblock
mc cp ./64MB.bin testminio/testbucket/

in a different terminal start downloading the file:

wget --limit-rate=50k -O /dev/null https://testminio/testbucket/64MB.bin

in a different terminal, while the file is being downloaded both of these commands would "hang":

mc cp ./64MB.bin testminio/testbucket/
mc rm testminio/testbucket/64MB.bin

[harshavardhana]

So, while there is someone downloading a file, there is no way deleting it or creating a new version (overriding it)? That's awful! Imagine someone "locking" all your objects by downloading them very slowly all day, every day! 😞

@taai, you know a fix; feel free to send it. Go ahead and write a distributed locking that is strictly consistent and safe.

in a different terminal, while the file is being downloaded, both of these commands would "hang":

They are not hung; they simply wait for the lock to be relinquished to perform the DELETE call.

Just so you know, we have already addressed this for small objects. However, we are working on addressing this comprehensively overall by using a different kind of sharded locking implementation.

One of the reasons why I reopened this GitHub, hold your horses while we get this out.