Implement deletes per the Docker Registry HTTP API V2 #422
Comments
|
+1 I updated to Registry 2.0 to have the delete operation working, only to find out later that it's not yet implemented. At least the docs should get updated on this to save people some time. |
stevvooe
changed the title
|
@bwb @kjozsa Sorry about the confusion in the documentation. cc @moxiegirl While deletes are part of the API, they cannot be safely implemented on top of an eventually consistent backend (read: s3). Please see #210 and #106 (comment) for some detail. As long as the registry supports varied backends, using an eventually consistent VFS model, safe deletions are not really possible without more infrastructure. We are adding the ability to delete blobs, without cleaning up interrelated content. This can be done easily but issuing such deletes may corrupt individual repositories so they must be done with care. @ncdc I understand this is annoying but ensuring data integrity is priority number one. |
|
Can you go into this a little more? S3 operations are atomic and the consistency model is variable. The fact that you can't delete at all is more disturbing than any sort of temporary effects a deletion on eventual consistency. At the very least an interim solution of deletion on write-guaranteed filesystems (local FS) would be tolerable. Some state would likely need to be kept (at least in memory) to count references, but the cost of that is vastly outweighed by the utility of the function being implemented. |
While the individual operations are atomic, visibility of those operations or a series of those operations are not. If you cannot be sure that no references are held to a given data object, you cannot safely delete it since a new reference, not visible to the deleting process, can be created in the interim. For example, let's say a delete is issued to manifest with digest A references a layer with digest B. When you delete A, the reference for B would decremented (or you would delete a file). During that point, the deletion algorithm would check the number of references (perhaps, by listing a directory). If there are no references, the item would be deleted. That decision must hold a mutex starting at the point of decrement to the point of deletion. Without the mutex, other processes may add references in the interim that would end up being invalid if the deletion was issued. The main danger here is that data is deleted that wasn't meant to be deleted. This is always worse than storing extra data. Deletes could still be safely issued through a management tool that "locks the world" (either by turning off the registry or locking out writes to the backend in some way). Deletes could also be directly issued on blobs that a user is certain must be removed. The registry will still handle this correctly albeit invalid manifests may be made available. There is also a scheme that could separate the registry backend into "generations", where one generation is read-only and one allows writes. The read-only generation is swept and garbage collected, while writes are issued to the other generation. Reads are broadcast to both. An optimization to this approach would be to write references at write time and then issue deletes at read time, when it can be certain they are consistent, avoiding the sweep phase.
We have a bit of a lowest common denominator problem. The main mistake here was that the v1 registry's virtual file system model was carried over to the v2 registry. This prevents one from making certain obvious optimizations in data storage that may be different from driver to driver (for example, leverage atomic moves on a unix filesystem). Instead of files, the registry driver should know how to store a blob or how to index a manifest. One should be able to compose these drivers in the manner they see fit. Imagine managing consistency in a postgres instance while storing blobs in s3. There are still issues that can arise in these scenarios, but I hope that clarifies. Please checkout the eventual consistency section on architecture. It enumerates various cases and the approaches to dealing with it. |
|
@stevvooe perhaps we can find an appropriate stopgap, to give users 90% of what they want, and continue working on the (harder) remaining 10%. For example, we could expose 2 distinct endpoints:
This is not a perfect arrangement, but for most users I think it would be a significant improvement. Users want 2 things right now 1) the ability to make an image unavailable for pull on their registry, and 2) the ability to reclaim disk space. WDYT? |
|
@shykes This sounds reasonable. We can divide this into two features:
If these sound good to you, we can makes issues covering each. |
|
Yes those sound good to me! I think they would make a lot of people very happy |
|
Awesome thanks! |
|
+1 I updated to Registry 2.0，and I found that I was not able to delete the image that I pushed. root@ubuntu24:~# curl -v --cacert ~/certs/docker-registry-ca.pem -X DELETE -u ops:wowotuan https://registry.wowotuan.me/v2/busybox/manifests/latest
|
|
@CptJason looking at #422 (comment) this looks to be planned for the 2.1 release / milestone |
|
+1 |
|
No milestone specified in the issue though 😎 |
|
We use TeamCity to build a Docker image for every git feature branch with the following naming pattern Use case 1: Use case 2: Using local FS. Did this help? |
|
@mattes This does. It sounds like you'll be served by delete API + garbage collection (#462). Unfortunately, this is quite a way out unless we can get some help from the community. We are unlikely to bring such deletion policies into the core registry code base. There are several different ways to do this that might be best served by an external notification endpoint. You should be able to create simple callback server that can implement these policies and delete the data directly from the filesystem. |
|
@gesellix I did. I've edited my comment. Thank you. |
|
+1 |
1 similar comment
|
+1 |
|
+1 |
|
+1 |
|
Instead of adding just a |
I'm using registry:2.0 to run a private Docker registry.
Image delete requests fail with an unsupported operation error.
I searched for answers, only to learn that this feature is not implemented:
Please change the Docker Registry 2.0 documentation. It links directly to the Docker Registry HTTP API V2, yet implements only a subset of API V2.
The text was updated successfully, but these errors were encountered: