-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor backend storage layout to meet new requirements (addresses #25, #46) #64
Refactor backend storage layout to meet new requirements (addresses #25, #46) #64
Conversation
Detecting tar files then falling back for calculating digests turned out to be fairly unreliable. Likely, the implementation was broken for content that was not a tarfile. Also, for the use case of the registry, it is really not needed. This functionality has been removed in FromReader and FromBytes. FromTarArchive has been added for convenience. Signed-off-by: Stephen J Day <stephen.day@docker.com>
Several requirements for storing registry data have been compiled and the backend layout has been refactored to comply. Specifically, we now store most data as blobs that are linked from repositories. All data access is traversed through repositories. Manifest updates are no longer destructive and support references by digest or tag. Signatures for manifests are now stored externally to the manifest payload to allow merging of signatures posted at different time. The design is detailed in the documentation for pathMapper. Signed-off-by: Stephen J Day <stephen.day@docker.com>
LGTM |
1 similar comment
LGTM |
return "", err | ||
} | ||
|
||
return path.Join(root, "current/link"), nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit-picky, but this should technically be "current", "link" as two separate arguments.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Combining these saves two string headers in memory and an extra slice header to hold them. :)
This code could use some reorganization to reduce to reflect the actual mapping. Arguably, "link" is a specific object in the mapping but the mapper does not reflect that. We could take the to refactor this into the notion of "resources" that understand where they sit in the filesystem. One of those resources would be a blob "link".
Added some minor comments, but LGTM |
65dcc44
to
a63b4b9
Compare
This change refactors the storage backend to use the new path layout. To facilitate this, manifest storage has been separated into a revision store and tag store, supported by a more general blob store. The blob store is a hybrid object, effectively providing both small object access, keyed by content address, as well as methods that can be used to manage and traverse links to underlying blobs. This covers common operations used in the revision store and tag store, such as linking and traversal. The blob store can also be updated to better support layer reading but this refactoring has been left for another day. The revision store and tag store support the manifest store's compound view of data. These underlying stores provide facilities for richer access models, such as content-addressable access and a richer tagging model. The highlight of this change is the ability to sign a manifest from different hosts and have the registry merge and serve those signatures as part of the manifest package. Various other items, such as the delegate layer handler, were updated to more directly use the blob store or other mechanism to fit with the changes. Signed-off-by: Stephen J Day <stephen.day@docker.com>
a63b4b9
to
83d6262
Compare
Several requirements for storing registry data have been compiled and the backend layout has been refactored to comply. Specifically, we now store most data as blobs that are linked from repositories. All data access is traversed through repositories. Manifest updates are no longer destructive and support references by digest or tag. Signatures for manifests are now stored externally to the manifest payload to allow merging of signatures posted at different time.
The design is detailed in the documentation for pathMapper.
This change refactors the storage backend to use the new path layout. To facilitate this, manifest storage has been separated into a revision store and tag store, supported by a more general blob store. The blob store is a hybrid object, effectively providing both small object access, keyed by content address, as well as methods that can be used to manage and traverse links to underlying blobs. This covers common operations used in the revision store and tag store, such as linking and traversal. The blob store can also be updated to better support layer reading but this refactoring has been left for another day.
The revision store and tag store support the manifest store's compound view of data. These underlying stores provide facilities for richer access models, such as content-addressable access and a richer tagging model. The highlight of this change is the ability to sign a manifest from different hosts and have the registry merge and serve those signatures as part of the manifest package.
Various other items, such as the delegate layer handler, were updated to more directly use the blob store or other mechanism to fit with the changes.
Detecting tar files then falling back for calculating digests turned out to be fairly unreliable. Likely, the implementation was broken for content that was not a tarfile. Also, for the use case of the registry, it is really not needed. This functionality has been removed in FromReader and FromBytes. FromTarArchive has been added for convenience.
This work supports #25 and #46. This PR is dependent on docker/libtrust#49, which may need some discussion before merging.