The transfer service is a simple flexible service which can be used to transfer artifact objects between a source and destination. The flexible API allows each implementation of the transfer interface to determines whether the transfer between the source and destination is possible. This allows new functionality to be added directly by implementations without versioning the API or requiring other implementations to handle an interface change.
The transfer service is built upon the core ideas put forth by the libchan project, that an API with binary streams and data channels as first class objects is more flexible and opens a wider variety of use cases without requiring constant protocol and API updates. To accomplish this, the transfer service makes use of the streaming service to allow binary and object streams to be accessible by transfer objects even when using grpc and ttrpc.
The transfer API consists of a single operation which can be called with various different objects depending on the intended operation.
In Go the API looks like,
type Transferrer interface {
Transfer(ctx context.Context, source interface{}, destination interface{}, opts ...Opt) error
}The proto API looks like,
service Transfer {
rpc Transfer(TransferRequest) returns (google.protobuf.Empty);
}
message TransferRequest {
google.protobuf.Any source = 1;
google.protobuf.Any destination = 2;
TransferOptions options = 3;
}
message TransferOptions {
string progress_stream = 1;
// Progress min interval
}| Source | Destination | Description | Local Implementation Version |
|---|---|---|---|
| Registry | Image Store | "pull" | 1.7 |
| Image Store | Registry | "push" | 1.7 |
| Object stream (Archive) | Image Store | "import" | 1.7 |
| Image Store | Object stream (Archive) | "export" | 1.7 |
| Object stream (Layer) | Mount/Snapshot | "unpack" | Not implemented |
| Mount/Snapshot | Object stream (Layer) | "diff" | Not implemented |
| Image Store | Image Store | "tag" | 1.7 |
| Registry | Registry | mirror registry image | Not implemented |
containerd has a single built in transfer plugin which implements most basic transfer operations. The local plugin can be configured the same way as other containerd plugins
[plugins]
[plugins."io.containerd.transfer.v1"]
Pull Components
flowchart TD
subgraph containerd Client
Client(Client)
end
subgraph containerd
subgraph Service
Streaming(Streaming Service)
Transfer(Transfer Service)
end
subgraph Transfer objects
RS(Registry Source)
ISD(Image Store Destination)
end
subgraph Backend
R(Resolver)
CS(ContentStore)
IS(Image Store)
S(Snapshotter)
end
end
Reg(((Remote Registry)))
Client-- Create Stream --> Streaming
Client-- Pull via Transfer --> Transfer
Transfer-- Get Stream --> Streaming
Transfer-- Progress via Stream--> Client
Transfer-->RS
Transfer-->ISD
Transfer-->CS
RS-->R
ISD-->IS
R-->Reg
ISD-->CS
ISD-->S
Streaming is used by the transfer service to send or receive data streams as part of an operation as well as to handle callbacks (synchronous or asynchronous). The streaming protocol should be invisible to the client Go interface. Object types such as funcs, readers, and writers can be transparently converted to the streaming protocol when going over RPC. The client and server interface can remain unchanged while the proto marshaling and unmarshaling need to be aware of the streaming protocol and have access to the stream manager. Streams are created by clients using the client side stream manager and sent via proto RPC as string stream identifiers. Server implementations of services can lookup the streams by the stream identifier using the server side stream manager.
Progress is an asynchronous callback sent from the server to the client. It is normally representing in the Go interface as a simple callback function, which the the client implements and the server calls.
From Go types progress uses these types
type ProgressFunc func(Progress)
type Progress struct {
Event string
Name string
Parents []string
Progress int64
Total int64
}The proto message type sent over the stream is
message Progress {
string event = 1;
string name = 2;
repeated string parents = 3;
int64 progress = 4;
int64 total = 5;
}Progress can be passed along as a transfer option to get progress on any transfer operation. The progress events may differ based on the transfer operation.
Transfer objects may also use io.Reader and io.WriteCloser directly.
The bytes are transferred over the stream using two simple proto message types
message Data {
bytes data = 1;
}
message WindowUpdate {
int32 update = 1;
}
The sender sends the Data message and the receiver sends the WindowUpdate message. When the client is sending an io.Reader, the client is the sender and server is the receiver. When a client sends an io.WriteCloser, the server is the sender and the client is the receiver.
Binary streams are used for import (sending an io.Reader) and export (sending an io.WriteCloser).
Credentials are handled as a synchronous callback from the server to the client. The callback is made when the server encounters an authorization request from a registry.
The Go interface to use a credential helper in a transfer object looks like
type CredentialHelper interface {
GetCredentials(ctx context.Context, ref, host string) (Credentials, error)
}
type Credentials struct {
Host string
Username string
Secret string
Header string
}It is send over a stream using the proto messages
// AuthRequest is sent as a callback on a stream
message AuthRequest {
// host is the registry host
string host = 1;
// reference is the namespace and repository name requested from the registry
string reference = 2;
// wwwauthenticate is the HTTP WWW-Authenticate header values returned from the registry
repeated string wwwauthenticate = 3;
}
enum AuthType {
NONE = 0;
// CREDENTIALS is used to exchange username/password for access token
// using an oauth or "Docker Registry Token" server
CREDENTIALS = 1;
// REFRESH is used to exchange secret for access token using an oauth
// or "Docker Registry Token" server
REFRESH = 2;
// HEADER is used to set the HTTP Authorization header to secret
// directly for the registry.
// Value should be `<auth-scheme> <authorization-parameters>`
HEADER = 3;
}
message AuthResponse {
AuthType authType = 1;
string secret = 2;
string username = 3;
google.protobuf.Timestamp expire_at = 4;
}