Upload chunks as separate files #981
Conversation
|
Attached issue: https://pulp.plan.io/issues/4498 |
lubosmj
force-pushed
the
not-onlys3-chunked-uploads-4498-2
branch
from
8d1a128
to
9a199f4
Compare
|
So this would store chunks in /var/lib/pulp/upload/ by default? |
|
@daviddavis, yes, to the place where uploads were stored before. Do you suggest me to create a new setting called CHUNKED_UPLOAD_DIR_NEW which will point to /var/lib/pulp/chunks/? |
|
No I think /var/lib/pulp/upload is good. |
CHANGES/4498.feature
Outdated
| @@ -0,0 +1 @@ | |||
| Made uploaded chunks to be stored as separated files in the default storage. | |||
There was a problem hiding this comment.
@daviddavis Should we mention, that this finally lifts the requirement that the api-nodes have a shared storage?
lubosmj
force-pushed
the
not-onlys3-chunked-uploads-4498-2
branch
3 times, most recently
from
2ac170e
to
2bd206a
Compare
CHANGES/4498.feature
Outdated
| @@ -0,0 +1,3 @@ | |||
| Made uploaded chunks to be stored as separate files in the default storage. This feature allows | |||
| Pulp nodes to have a shared storage when dealing with clustered deployment. Each node can now | |||
| access all uploaded chunks as they were uploaded directly to that node. | |||
There was a problem hiding this comment.
I think it is: This feature removes the need for a share storage of pulp api nodes, as the chunks are now stored individually in the shared storage and are therefore accessible by all nodes.
There was a problem hiding this comment.
Ah, okay, I will reword it. Thank you!
lubosmj
force-pushed
the
not-onlys3-chunked-uploads-4498-2
branch
from
2bd206a
to
b35c4ab
Compare
| @@ -0,0 +1,3 @@ | |||
| Made uploaded chunks to be stored as separate files in the default storage. This feature removes | |||
| the need for a share storage of pulp api nodes, as the chunks are now stored individually in the | |||
| shared storage and are therefore accessible by all nodes. | |||
| @@ -22,7 +25,14 @@ def commit(upload_id, sha256): | |||
| log.info(_("The upload was not found. Nothing to do.")) | |||
| return | |||
|
|
|||
| file = files.PulpTemporaryUploadedFile.from_file(upload.file.file) | |||
| chunks = models.UploadChunk.objects.filter(upload=upload).order_by("offset") | |||
There was a problem hiding this comment.
Probably not in this pr, but can we add a check, that the chunks uploaded neither overlap nor fail to cover the whole file?
The overlap check can be added to where we save the upload chunk, and then a "sum of all chunk sizes == expected file size" should be sufficient.
|
|
||
| def delete(self, *args, **kwargs): | ||
| """ | ||
| Delete UploadChunk model and the file associated with the model |
closes #4498