-
Notifications
You must be signed in to change notification settings - Fork 877
NG: storage driver model #616
Comments
The standard idiom for this seems to be something along using an interface (following your simplification in #612's OP):
Then, in the registry we can just reference that interface and perform operations against it:
This is just off the top of my head, but it seems to be the standard idiom in this case :) As for the case with resumable push, I'm not too sure on how it can be performed in the context of a file store. A filestore is simply a glorified key/value store which reads and writes to an external filesystem. This is probably the registry's territory, as it would know which bits of data still need to be written and which bits have already been pushed. There could be a way to accomplish this. Maybe by doing a server-side check to see what bits have been pushed, and resume pushing the ones that have not already been pushed? |
On Wed, Oct 08, 2014 at 08:51:39PM -0700, Matthew Fisher wrote:
For resumable-push, I think the storage needs to be more than a type Filestore interface { Where GetSize returns the total allocated size and GetCurrentSize I also dropped the Move because I we don't need that for |
On Thu, Oct 09, 2014 at 08:26:14AM -0700, W. Trevor King wrote:
Actually, I'd be happy with Store subclassing (or providing, or |
There certainly are ways to do this: type FileStore interface {
Get
Put
List
...
}
type StreamingFileStore interface {
FileStore
ReadStream
WriteStream
}
type AtomicFileStore interface {
FileStore
...
} At that point you can use the power of duck-typing to figure out what interface the filestore implements :) |
On Thu, Oct 09, 2014 at 08:26:14AM -0700, W. Trevor King wrote:
And if we're serious about the content-addressable part, we should Put(data string) string and have the storage return an arbitrary key for each stored object |
On Thu, Oct 09, 2014 at 08:39:07AM -0700, Matthew Fisher wrote:
Actually, I'm fine with just having my AtomicStore and StreamingStore |
FWIW, S3 does currently support multipart, resumable push. |
Is there still a need for the Move command in this API with all large files being content-addressable? Also, I'm not aware of any distributed storage systems which benefit from a first-class move command. Most seem to approximate this behavior with a full copy followed by a deletion of the old version. |
I'm proposing that we add an offset parameter to the ReadStream method for resumable downloads if we're already planning on supporting resumable uploads. |
On Tue, Oct 14, 2014 at 11:36:28AM -0700, Brian Bland wrote:
Another alternative would be to set these up as seek-able files. |
Youtube handles resumable uploads in an interesting way |
On Wed, Oct 08, 2014 at 08:51:39PM -0700, Matthew Fisher wrote:
The Docker hub had 14000 “applications” ($namespace/$repository?) in Count(prefix string) int to let you see how many objects matched a (possibly empty) prefix and Anyhow, something like this would allow the (streaming) storage driver |
Fwiw, the ability to shard/fan out consistently is a huge +1 for me On Wednesday, October 29, 2014, W. Trevor King notifications@github.com
Raymond "VisualPhoenix" Barbiero |
On Wed, Oct 29, 2014 at 07:20:26AM -0700, W. Trevor King wrote:
Redis lets you do this for lists with 1: LRANGE key start stop (but not for sets, where SMEMBERS returns all the values 2). I |
From #docker-dev today: 11:00 < dmp42> 5. would love some gcs comments regarding the new I'm not sure how solid this freeze is intended to be, but I'd love to version() → [major, minor, patch] so the registry could ask what version of the API the driver was |
I'm pretty sure the freeze is a 0.X version, so still subject to tweaks before final release. I like the idea of checking storage driver compatibility at startup, but I'm not sure that semantic versioning is necessary or real backwards compatibility is worth the complexity. We can use a [Major, Minor] version scheme, as patch version doesn't seem to provide much value when applied to an API (correct me if I'm wrong here). When the driver has a newer minor version than the registry, this should still be usable, but should emit some sort of warning, but if the driver is running an older version than the registry or a newer major version, we should probably exit with a compatibility error for sanity purposes. |
The way registry V1 worked was semver compatible (your driver would depend on |
I think semver is the way to go and we should have backwards compatibility On Monday, November 3, 2014, Olivier Gambier notifications@github.com
Raymond "VisualPhoenix" Barbiero |
On Mon, Nov 03, 2014 at 12:40:50PM -0800, Olivier Gambier wrote:
Yeah, and I think the driver API should be versioned independently of @BrianBland, good point about dropping the patch from the API version.
That works whether the driver or the registry has the newer API, and It doesn't allow for negotiation though, which we'd need if we |
On Mon, Nov 03, 2014 at 01:06:08PM -0800, W. Trevor King wrote:
Oops, this should be:
|
If the registry has a newer minor version than the storage driver, this could mean that the registry added a new method to the driver api that the storage driver does not respect, as per semantic versioning. We would either have to keep a mapping of which versions support which methods and operate differently depending on the driver version, or we could just reject the storage driver for being out of date. |
On Mon, Nov 03, 2014 at 01:15:32PM -0800, Brian Bland wrote:
Ah, right. I think I'd turned this around in my head, and had |
On Mon, Nov 03, 2014 at 01:06:08PM -0800, W. Trevor King wrote:
It also doesn't handle API bifurcation, e.g. if we want to use version() → {api_name: [major, minor], …} with a dictionary of supported APIs and their versions. If we extend version() → {api_name: [[major, minor], …], …} it would also allow us to have drivers that simultaneously supported {'storage': [[1, 8], [2, 3]]} |
I'm not sure how a multi-version storage driver would function. How would we handle changing the behavior, parameters, or return values of an API method without having to rename the method each time or providing a version parameter with each API call? I'm not a fan of either of these approaches... |
On Mon, Nov 03, 2014 at 03:39:10PM -0800, Brian Bland wrote:
I don't see major-version bumps as a frequent feature, and the web |
I believe this, to a large extent, should be closed given #643 was merged. Special thanks to @BrianBland for his continuous work on this, to @wking @bacongobbler @noxiouz @visualphoenix and others who took time for the thorough reviews and discussions. What we have certainly still has some rough edges and can use some tire-kicking and bullet tests, but this is a very good first shot. One single char to sum this first "next-generation" journey: U+2661 ♡ |
Designing the communication API with the drivers is actually the easy part.
The hard part is:
Here are some things that we want to take into account:
@noxiouz @bacongobbler
The text was updated successfully, but these errors were encountered: