Refactor backend storage layout to meet new requirements (addresses #25, #46) #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Detecting tar files then falling back for calculating digests turned out to be fairly unreliable. Likely, the implementation was broken for content that was not a tarfile. Also, for the use case of the registry, it is really not needed. This functionality has been removed in FromReader and FromBytes. FromTarArchive has been added for convenience. Signed-off-by: Stephen J Day <stephen.day@docker.com>
Several requirements for storing registry data have been compiled and the backend layout has been refactored to comply. Specifically, we now store most data as blobs that are linked from repositories. All data access is traversed through repositories. Manifest updates are no longer destructive and support references by digest or tag. Signatures for manifests are now stored externally to the manifest payload to allow merging of signatures posted at different time. The design is detailed in the documentation for pathMapper. Signed-off-by: Stephen J Day <stephen.day@docker.com>
|
LGTM |
1 similar comment
|
LGTM |
stevvooe
changed the title
stevvooe
changed the title
| return "", err | ||
| } | ||
|
|
||
| return path.Join(root, "current/link"), nil |
There was a problem hiding this comment.
Nit-picky, but this should technically be "current", "link" as two separate arguments.
There was a problem hiding this comment.
Combining these saves two string headers in memory and an extra slice header to hold them. :)
This code could use some reorganization to reduce to reflect the actual mapping. Arguably, "link" is a specific object in the mapping but the mapper does not reflect that. We could take the to refactor this into the notion of "resources" that understand where they sit in the filesystem. One of those resources would be a blob "link".
|
Added some minor comments, but LGTM |
stevvooe
force-pushed
the
separate-signature-storage
branch
from
65dcc44
to
a63b4b9
Compare
This change refactors the storage backend to use the new path layout. To facilitate this, manifest storage has been separated into a revision store and tag store, supported by a more general blob store. The blob store is a hybrid object, effectively providing both small object access, keyed by content address, as well as methods that can be used to manage and traverse links to underlying blobs. This covers common operations used in the revision store and tag store, such as linking and traversal. The blob store can also be updated to better support layer reading but this refactoring has been left for another day. The revision store and tag store support the manifest store's compound view of data. These underlying stores provide facilities for richer access models, such as content-addressable access and a richer tagging model. The highlight of this change is the ability to sign a manifest from different hosts and have the registry merge and serve those signatures as part of the manifest package. Various other items, such as the delegate layer handler, were updated to more directly use the blob store or other mechanism to fit with the changes. Signed-off-by: Stephen J Day <stephen.day@docker.com>
stevvooe
force-pushed
the
separate-signature-storage
branch
from
a63b4b9
to
83d6262
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Several requirements for storing registry data have been compiled and the backend layout has been refactored to comply. Specifically, we now store most data as blobs that are linked from repositories. All data access is traversed through repositories. Manifest updates are no longer destructive and support references by digest or tag. Signatures for manifests are now stored externally to the manifest payload to allow merging of signatures posted at different time.
The design is detailed in the documentation for pathMapper.
This change refactors the storage backend to use the new path layout. To facilitate this, manifest storage has been separated into a revision store and tag store, supported by a more general blob store. The blob store is a hybrid object, effectively providing both small object access, keyed by content address, as well as methods that can be used to manage and traverse links to underlying blobs. This covers common operations used in the revision store and tag store, such as linking and traversal. The blob store can also be updated to better support layer reading but this refactoring has been left for another day.
The revision store and tag store support the manifest store's compound view of data. These underlying stores provide facilities for richer access models, such as content-addressable access and a richer tagging model. The highlight of this change is the ability to sign a manifest from different hosts and have the registry merge and serve those signatures as part of the manifest package.
Various other items, such as the delegate layer handler, were updated to more directly use the blob store or other mechanism to fit with the changes.
Detecting tar files then falling back for calculating digests turned out to be fairly unreliable. Likely, the implementation was broken for content that was not a tarfile. Also, for the use case of the registry, it is really not needed. This functionality has been removed in FromReader and FromBytes. FromTarArchive has been added for convenience.
This work supports #25 and #46. This PR is dependent on docker/libtrust#49, which may need some discussion before merging.