Example on setting Any.Value for DiscoveryResponse.Resources

[guanw]

Hi, curious if there's any example on how to create Resources with type any.Any?

In its documentation examples like

//     {
//       "@type": "type.googleapis.com/google.protobuf.Duration",
//       "value": "1.212s"
//     }

were given but in reality the data envoy is looking for seems to be more nested&complicated.
e.g if I want to manually create a response with type: type.googleapis.com/envoy.api.v2.ClusterLoadAssignment

I'm assuming for this you need to

assignment := &v2.ClusterLoadAssignment{...}
message, err := proto.Marshal(assignment)
if err != nil {
	return
}

res.Resources = append(res.Resources, &any.Any{
	TypeUrl: "type.googleapis.com/envoy.api.v2.ClusterLoadAssignment",
	Value:   message,
})

Is this correct? I think it's better put into a test or example so it's more clear.

relevant question:
After some reading, I got the idea that to actually use go-control-plane one should just implement snapshotCache with logic to delete/add/modify cluster. Is that recommended? Or should developer implement the entire server.Server interface? Thanks in advance.

[cstrahan]

Regarding Any, I think you're looking for this:

func MarshalAny(pb proto.Message) (*any.Any, error)

See https://godoc.org/github.com/golang/protobuf/ptypes#MarshalAny

[alecholmez]

@guanw for your second part about the message assignment, that is correct, if you don't in server/server.go there's a little piece of logic:

// Envoy relies on serialized protobuf bytes for detecting changes to the resources.
		// This requires deterministic serialization.
		if resp.ResourceMarshaled {
			resources[i] = &any.Any{
				TypeUrl: typeURL,
				Value:   resp.MarshaledResources[i],
			}
		} else {
			marshaledResource, err := cache.MarshalResource(resp.Resources[i])
			if err != nil {
				return nil, err
			}

			resources[i] = &any.Any{
				TypeUrl: typeURL,
				Value:   marshaledResource,
			}
		}

For your third question, I wouldn't recommend that. snapshotCache is just the underlying resource layer for the Envoy discovery services which live in the Server interface that you mention. So if you actually want Envoy to be able to utilize that cache which just basically a map of listeners, clusters, endpoints, and routes, you do need to fully implement that Server. But go-control-plane does that for you already. All you need to do is something like:

func main() {
	snapshotCache := cache.NewSnapshotCache(false, cache.IDHash{}, nil)
	server := xds.NewServer(context.Background(), snapshotCache, nil)
	grpcServer := grpc.NewServer()
	lis, _ := net.Listen("tcp", ":8080")

	discovery.RegisterAggregatedDiscoveryServiceServer(grpcServer, server)
	api.RegisterEndpointDiscoveryServiceServer(grpcServer, server)
	api.RegisterClusterDiscoveryServiceServer(grpcServer, server)
	api.RegisterRouteDiscoveryServiceServer(grpcServer, server)
	api.RegisterListenerDiscoveryServiceServer(grpcServer, server)
	go func() {
		if err := grpcServer.Serve(lis); err != nil {
			// error handling
		}
	}()
}

Which we note in the README in the root of the project

[guanw]

@alecholmez hmmm. I guess I'm a bit confused at this point as to what role does go-control-plane plays here.( partially because a lot of envoy and go-control-plane doc have invalid link references)

I started experimenting with envoy with this
and long story short the repo above runs a server that implements EDS query API and its logic to add/delete/update hosts so that you can easily register/unregister new hosts to serve requests.

I thought go-control-plane could give you service discovery out of box like that. (At least from #85 it looks like you can achieve service discovery with http gateway as well as the server.Server you implement on your own) Or am I completely wrong? (The example you provide doesn't seem to provide ability to register new host for service discovery purposes) If so, what would be a better option to achieve service discovery with my own logic to register new service hosts?

I hope my question would also help a lot of new comers to understand control-plane better. Thanks!

[alecholmez]

@guanw so at least from my understanding, go-control-plane is simply just a functional implementation of the xDS protocols. I personally think the term "service discovery" is a bit overloaded at times.

Out of the box, go control plane will add instances when it detects them into the envoy configurations that it broadcasts out over it's streams. So if you were to implement something that had the ability to add EDS objects into the snapshot, then yes it will do service discovery. What it does not do is something like watch your underlying infrastructure and add instances when it detects pods and such. That's when things like GreyMatter and Istio come into play with their own custom control planes.

I would recommend checking out the full integration test that this repo has it might make some of this a bit clearer:
https://github.com/envoyproxy/go-control-plane/tree/master/pkg/test/main

What it demonstrates is programmatically adding envoy configuration objects into the snapshot cache, which are then handled by the rest of the go control plane logic to be sent out to each envoy registered in the mesh. So for instance, many people when building their own control plane will use go control plane as a starting point (which is what it's intended for) and build their own host discovery mechanism on top. As an example, if you were to be running this on K8s, and you wanted to add hosts to clusters when pods come alive, you could watch the K8s API and retrieve pod information as they spawn. When that happens you take the host information and then add it to a new snapshot as a cluster or endpoint or whatever.. go-control-plane will then take that new snapshot and send it out to all the Envoys you have registered.

I hope that helps a little bit and clears some stuff up. server.Server is already implemented for you.

One little thing I kind of made up to help me distinguish what go-control-plane does is this:
Service/Host discovery, and configuration are not one in the same. They are related but perform different actions. A control plane is what manages all your configuration across your mesh. Service discovery can be used by a control plane to properly manage configuration aspects such as clusters and endpoints.

[guanw]

@alecholmez Thank you for the detailed explanation. I just checked out the integration test and it looks promising. I will use it as a starting point to build out my eds cluster dynamically.

One other thing I notice is that ./build/run_docker.sh make integration seems to fail with following error locally.

any thought? I can paste the log file here but I believe people should get the same log running in same docker environment.

[alecholmez]

@guanw sure thing glad I could help.

Hmm I'll have to give that a look. A full log file would be helpful I'll see if I can replicate the problem

[alecholmez]

@guanw I was able to recreate the issue, I'll look into it I have a feeling the script is just not up to date.

[alecholmez]

According to the log it seems like it can't find the binary:

build/integration.sh: line 30: /usr/local/bin/envoy: No such file or directory

I'll dig further into this. Should have a PR with a fix soon. Thanks for the catch on that

[kyessenov]

The test relies on a compiled envoy to verify some xDS use cases (for now that's the best approximation an xDS client). You can override the location of the binary with ENVOY variable.

[guanw]

@alecholmez thanks for confirming. Glad I helped.

[alecholmez]

@kyessenov thanks for the info. Will do

[MIBc]

According to the log it seems like it can't find the binary:

build/integration.sh: line 30: /usr/local/bin/envoy: No such file or directory

I'll dig further into this. Should have a PR with a fix soon. Thanks for the catch on that

The envoy is at /usr/bin/envoy if use yum to install it.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.