Object storage API and sample implementation

[jssmith]

We are moving forward with a prototype object storage API implementation using gRPC as the transport. What's important about this contribution is the interface, not the implementation. What's most important to review is the gRPC file: pkg/objstore/objstore.proto.

Some notes:

• This implementation is all Go and uses a local file system to store the objects. We are using this to get the interface right, then will do a C++ implementation wrapping Anna.
• gRPC has greater latency than Thrift, however it is better suited to streaming large objects. See experience at Alluxio

Some key questions:

• How should object versions work?
• Do we want to be support multi-part object uploads?
• What are we doing for permissions / ACLs?
• What do we want to do for location, e.g., regions?
• What concurrency models / consistency guarantees do we support? Are such guarantees part of the interface / spec or not?
  • How are we encoding error codes? Can we use standard gRPC status or should we encode our own errors on top of gRPC?
• Should we be checking in the Go code generated by protoc? That would make builds simpler by eliminating a dependency?

[vsreekanti]

Include objectName for async request/responses?

[jssmith]

I don't really understand how gRPC asyc requests work but it seems that there should be some id associated with the request, rather than in the data. There appears to be a provision for tagging the responses, e.g., in this example.

[anuragkh]

You could create separate, async versions of each of the requests, where the request returns a handle that one could wait upon.

[vsreekanti]

Nit: This shouldn't need to be redefined. You can use import "google/protobuf/empty.proto"; and then return google.protobuf.Empty. There are a bunch of commonly used protobuf definitions in that package, which you can find here.

[jssmith]

Ok, let's try that. I was going to go that way but then figured this one is actually simpler to define than to import. But maybe some of the other definitions are useful.

[vsreekanti]

Are there standard ways to translate those codes into client languages? If not, you can just use protobuf enums, and those are compiled into enums in the corresponding language, which helps with readability quite a bit (e.g., here).

[jssmith]

The main advantage that I see in using the standard error codes, at least in Go, is that you check the error once.

Here's what it could end up looking like with two levels of errors:

func doPut(objectName string, objectData []byte) {
    // ... get client and context
    r, err := client.Put(ctx, &pb.PutRequest{ObjectName:objectName,ObjectData:data})
    if err != nil {
        return err
    }
    if r.ErrCode != pb.OK {
        return NewObjStoreErr(r.ErrCode)
    }
    return nil
}

vs

func doPut(objectName string, objectData []byte) {
    // ... get client and context
    r, err := client.Put(ctx, &pb.PutRequest{ObjectName:objectName,ObjectData:data})
    if err != nil {
        return err
    }
    return nil
}

[anuragkh]

+1 for standard error codes.

[jssmith]

We are planning to go down the path of standard error codes. It definitely makes the code nicer. One disadvantage we should be mindful of is that the errors become a more explicit part of the interface when we define them as part of the protobuf messages. Fully specifying the behavior requires tests either way, though.

[anuragkh]

Permissions would make sense in a multi-tenant setting. In the worst case, if a store does not support it, it can ignore this value.

[anuragkh]

You could create separate, async versions of each of the requests, where the request returns a handle that one could wait upon.

[anuragkh]

I think deleteBucket(bucketName), delete(objectName) are missing

[anuragkh]

Also, if we do decide to add permissions, some way of looking up permissions..

[anuragkh]

What would location correspond to? Geographic region?

[jssmith]

That's what I had in mind. Going off of the S3 API it's a region. In principle it could mean something else though, e.g., some locality group.

[anuragkh]

+1 for standard error codes.

[vsreekanti]

Would be good to have comments on Protobuf definitions (especially to document decisions to explicitly include/exclude certain things in our existing API) and on the reference implementation to explain what the moving parts in a gRPC object store service are.

[jssmith]

Status update:

• Libo has added some documentation and I've contributed as well. We want the documentation to render properly using the godoc tool, but right now some of it doesn't show up. @Chenlibo, can you please have a look at this?
• The benchmark client currently is a standalone program rather than one that is integrated with the SRK tool and that can run in a cloud function environment. We need to change this.