From 0e75c39dce8ebee8b8460f17d09a758115b11c71 Mon Sep 17 00:00:00 2001 From: Jason Young Date: Wed, 20 Feb 2019 12:42:59 -0800 Subject: [PATCH 1/3] document MCP --- mcp/Readme.md | 154 ++++++++++++++++-- .../ResourceSink-connection-setup.svg | 1 + .../ResourceSource-connection-setup.svg | 1 + .../collection-full-state-update-success.svg | 1 + .../collection-incremental-update-success.svg | 1 + .../diagrams/collection-update-error.svg | 1 + 6 files changed, 146 insertions(+), 13 deletions(-) create mode 100644 mcp/v1alpha1/diagrams/ResourceSink-connection-setup.svg create mode 100644 mcp/v1alpha1/diagrams/ResourceSource-connection-setup.svg create mode 100644 mcp/v1alpha1/diagrams/collection-full-state-update-success.svg create mode 100644 mcp/v1alpha1/diagrams/collection-incremental-update-success.svg create mode 100644 mcp/v1alpha1/diagrams/collection-update-error.svg diff --git a/mcp/Readme.md b/mcp/Readme.md index 68e04044171..2183f2863f0 100644 --- a/mcp/Readme.md +++ b/mcp/Readme.md @@ -1,13 +1,141 @@ -# Mesh Configuration Protocol Protos - -This folder contains the proto buffers used by the Mesh Configuration Protocol, -an [XDS](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md#streaming-grpc-subscriptions) -inspired protocol for transferring configuration among Istio components -during runtime. - -The protocol buffers in this folder are not used for configuring Istio. -Instead, they define an internal protocol through which the configuration proto -instances can be delivered to components, such as Mixer and Pilot. - -The server side of the protocol is implemented in Galley, Istio's config -aggregation and distribution component. +# Mesh Configuration Protocol (MCP) + +## Introduction + +This folder contains the proto buffers for the Mesh Configuration +Protocol (MCP). MCP is based on +[XDS](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md#streaming-grpc-subscriptions) +and maintains conceptual alignment with it, despite the specific +service and proto definitions being different. + +## Overview + +MCP is a subscription-based configuration distribution API. The +configuration consumer (i.e. sink) requests updates for collections of +resources from a configuration producer (i.e. source). The source +pushes resource updates to the sink when resources are added, updated, +or deleted. The sink positively ACK's the resource update, if it was +accepted, and NACK's if it was rejected, e.g. because a resource was +invalid. The source may push additional update once the previous +update was ACK/NACK'd. The source should only have one outstanding +update (per-collection) in flight at a time. + +MCP is a pair of bidirectional streaming gRPC API services +(`ResourceSource` and `ResourceSink`). + +* The `ResourceSource` service is used when the resource source is the +server and the sink is a client. By default, Galley implements the +`ResourceSource` service and Pilot/Mixer connect as clients. + +* The `ResourceSink` service is used when the resource source is a +client and the sink is the server. Galley can be configured to +optionally "dial-out" to a remote configuration sink, e.g. Pilot is in +another cluster where it cannot, as a client, initiate +connection to Galley. In this scenario, Pilot would implement the +`ResourceSink` service and Galley would connect as a client. + +ResourceSource and ResourceSink are semantically equivalent with +regards to the message exchange. The only meaningful difference is who +initiates the connection and opens the grpc stream. + +## Data model + +MCP is the mechanism of transport whereby Pilot and Mixer can be +configured by a manager component. MCP defines a common per-resource +metadata format and resource specific contents is defined elsewhere +(e.g. https://github.com/istio/api/tree/master/networking/v1alpha3). + +### Collections + +Resources of the same type are organization into named +collections. Istio API collection names are of the form +`istio///` where ``, ``, and `Created with Raphaël 2.2.0Client/SourceClient/SourceServer/SinkServer/Sinkconn := grpc.Dial(S)svc := mcp.NewResourceSinkService(conn)stream := mcp.EstablishResourceStream()stream establishedstream.Send(RequestResources)Resources := stream.Recv() \ No newline at end of file diff --git a/mcp/v1alpha1/diagrams/ResourceSource-connection-setup.svg b/mcp/v1alpha1/diagrams/ResourceSource-connection-setup.svg new file mode 100644 index 00000000000..71af7d3a034 --- /dev/null +++ b/mcp/v1alpha1/diagrams/ResourceSource-connection-setup.svg @@ -0,0 +1 @@ +Created with Raphaël 2.2.0Client/SinkClient/SinkServer/SourceServer/Sourceconn := grpc.Dial(S)svc := mcp.NewResourceSourceService(conn)stream := svc.EstablishResourceStream()stream establishedstream.Send(RequestResources)Resources := stream.Recv() \ No newline at end of file diff --git a/mcp/v1alpha1/diagrams/collection-full-state-update-success.svg b/mcp/v1alpha1/diagrams/collection-full-state-update-success.svg new file mode 100644 index 00000000000..0be0e75ed0d --- /dev/null +++ b/mcp/v1alpha1/diagrams/collection-full-state-update-success.svg @@ -0,0 +1 @@ +Created with Raphaël 2.2.0SinkSinkSourceSourceestablish streamRequestResources{collection=A,nonce=}Resources{collection=A,nonce=1,resource={(foo,v0),(bar,v0)}RequestResources{collection=A,nonce=1} (ACK)resources={(foo,v0),(bar,v0)}Resources{collection=A,nonce=2,resource={(foo,v0),(bar,v0),(baz,v0)}}RequestResources{collection=A,nonce=2} (ACK)resources={(foo,v0),(bar,v0),(baz,v0}Resources{collection=A,nonce=3,resource={(bar, v1),(baz, v0)}}RequestResources{collection=A,nonce=3} (ACK)resources={(bar,v1),(baz,v0)} \ No newline at end of file diff --git a/mcp/v1alpha1/diagrams/collection-incremental-update-success.svg b/mcp/v1alpha1/diagrams/collection-incremental-update-success.svg new file mode 100644 index 00000000000..7fa6d44ee53 --- /dev/null +++ b/mcp/v1alpha1/diagrams/collection-incremental-update-success.svg @@ -0,0 +1 @@ +Created with Raphaël 2.2.0SinkSinkSourceSourceestablish streamRequestResources{collection=A,nonce=}Resources{collection=A,nonce=1,resource={(foo,v0),(bar,v0)},incremental=true}RequestResources{collection=A,nonce=1,incremental=true} (ACK)resources={(foo,v0),(bar,v0)}Resources{collection=A,nonce=2,resource={(baz,v0)},incremental=true}RequestResources{collection=A,nonce=2,incremental=true} (ACK)resources={(foo,v0),(bar,v0),(baz,v0}Resources{collection=A,nonce=3,resource={(bar, v1)},removed={foo}incremental=true}RequestResources{collection=A,nonce=3,incremental=true} (ACK)resources={(bar,v1),(baz,v0)} \ No newline at end of file diff --git a/mcp/v1alpha1/diagrams/collection-update-error.svg b/mcp/v1alpha1/diagrams/collection-update-error.svg new file mode 100644 index 00000000000..8b4a86fbade --- /dev/null +++ b/mcp/v1alpha1/diagrams/collection-update-error.svg @@ -0,0 +1 @@ +Created with Raphaël 2.2.0SinkSinkSourceSourceestablish streamRequestResources{collection=A,nonce=}Resources{collection=A,nonce=1,resource={(foo,v0),(bar,v0)}RequestResources{collection=A,nonce=1} (ACK)resources={(foo,v0),(bar,v0)}Resources{collection=A,nonce=2,resource={(baz,v0)}RequestResources{collection=A,nonce=2,error="could not apply"} (NACK)resources={(foo,v0),(bar,v0)}Resources{collection=A,nonce=3,resource={(bar, v1)},remove={(foo,v0)}RequestResources{collection=A,nonce=3} (ACK)resources={(bar,v1),(baz,v0)} \ No newline at end of file From 7797a211fb572a7112d3e721e210443491daccae Mon Sep 17 00:00:00 2001 From: Jason Young Date: Thu, 21 Feb 2019 10:42:05 -0800 Subject: [PATCH 2/3] add additional note on NACK behavior --- mcp/Readme.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mcp/Readme.md b/mcp/Readme.md index 2183f2863f0..0e1939bd722 100644 --- a/mcp/Readme.md +++ b/mcp/Readme.md @@ -135,6 +135,13 @@ The following example shows what happens when a change cannot be applied. ![Resource update error](v1alpha1/diagrams/collection-update-error.svg) +The sink should only NACK in _exceptional_ cases. For example, if a set of +resources was invalid, malformed, or could not be decoded. NACK'd updates +should raise an alarm for subsequent investigation by a human. The source +should not resend the same set of resources that were previously NACK'd. +Canary pushes to dedicated sinks may also be used to verify correctness +(non-NACK) before pushing to a larger fleet of resource sinks. + The nonce in MCP is used to match RequestResources and Resources. On reconnect, the sinks may attempt to resume a session with the same source by specifying the known resources version with From ff9416363477d4d765263569195cc7e727059292 Mon Sep 17 00:00:00 2001 From: Jason Young Date: Thu, 21 Feb 2019 11:43:09 -0800 Subject: [PATCH 3/3] add a note about ignoring stale nonces --- mcp/Readme.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mcp/Readme.md b/mcp/Readme.md index 0e1939bd722..d8b6db6fa12 100644 --- a/mcp/Readme.md +++ b/mcp/Readme.md @@ -110,6 +110,9 @@ ACK/NACK. Upon receiving an update, the sink is expected to send an ACK/NACK relatively quickly after decoding, validating, and persisting the update to its internal configuration store. +The source should ignore requests with stale and unknown nonces that +do not match the nonce in the most recently sent `Resource` message. + ### Success examples The following example shows the sink receiving a sequence of changes