MAISTRA-2294 create CRD for federation ServiceExport

[rcernich]

Please provide a description for what this PR is for.

And to help us figure out who should review this PR, please
put an X in all the areas that this PR affects.

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Pull Request Attributes

Please check any characteristics that apply to this pull request.

[ ] Does not have any changes that may affect Istio users.

[rcernich]

This iteration of the CRD supports exporting services to specific meshes in the following way:

• There is a single ServiceExports resource that defines all services exported to a named remote mesh.
• The name of the ServiceExports resource must match the name of a MeshFederation resource. This is used to identify the remote mesh to which the services are exported
• There may also be a default named ServiceExports resource, which will be applied to all remote meshes. If a named ServiceExports resource exists, the alias rules in the named resource will take precedence.
• A ServiceExports resource contains a list of export rules. Each rule defines:
  • A selector, which is used to select a service or set of services
  • An option alias, which defines the name of the service to be exposed to the remote mesh
  • The first rule that matches will be the rule used to export the service

There are two types of export rules:

• NameSelector allows for selecting a specific service by namespace and name and applying an optional alias for either/both namespace and name, e.g. foo/service-a might be mapped to bar/service-b or bar/service-a or foo/service-b
  • The selector may use * for either namespace or name to select services from any namespace or any name or all services (/)
  • Aliasing when using * will only apply to name or namespace if * is not used, e.g. if */service-a is the selector, only the name may be mapped, so if the alias name were service-b, you'd end up with foo/service-a and bar/service-a being exported as foo/service-b and bar/service-b
• LabelSelector allows for selecting services from a specific namespace using a label selector. It also specifies a list of name mappings to use for aliasing selected services.
  • If namespace is omitted or specified a *, the label selector will be applied to services in all mesh namespaces.
  • Aliases work in the same manner as NameSelector (i.e. this is really a two tiered selector, with the label selector selecting the service, then matching an alias rule to determine the export name). If no alias rule matches, a selected service is exported as-is.

All services are exported as namespace/name. Any domain suffix will be managed by the importing mesh. The rationale here is that this brings export/import of services more in line with how istio manages services in the service registry, as these map nicely to simple name and namespace fields. (Current thinking is that this would be defined in MeshFederation, or just use some convention like, .svc..local. We probably still need to figure out how the domain suffix bit plays into this, but I think this simplifies export for the user. The issues will show up on the import side anyway.)

Questions:

1. Is it acceptable that we allow configuration of a default set of export rules?
2. Is it acceptable that there is a single resource for defining all exports, vs individual export resources for each service?
3. Is it acceptable that we only export namespace/name, letting the importer deal with domain suffixes, etc.?
4. Should we support globbing of names? Should we expand it to support name prefix/suffix?
5. I'm anticipating only the mesh administrator should be able to configure federation, so only the mesh administrator should be able to configure exports. Does this seem reasonable? Using label selectors allows the mesh administrator to delegate some of that configuration, but they can reduce the scope to select namespaces, or not use it at all.

[dgn]

related to maistra/istio-operator#706

[dgn]

This iteration of the CRD supports exporting services to specific meshes in the following way:

* There is a single ServiceExports resource that defines all services exported to a named remote mesh.

The main problem I see with having everything in one resource is lack of RBAC. A system administrator can't really say "you have the right to export services in your namespaces" because it's all handled in one central resource. Some people might find it to be better than having to create individual resources per Service, but I don't really like it. There's no single resource to describe all your Deployments, either. It's a matter of taste, though.

The real question is going to be: do we want the mesh administrator to handle all of it or can Service administrators expose their services themselves? Having individual resources is better for a sort of self-service model.

* The name of the ServiceExports resource must match the name of a MeshFederation resource.  This is used to identify the remote mesh to which the services are exported

Again, this strongly ties in with the 1-to-1 relationship between MeshFederation and ServiceExport. I'd prefer a 1-to-many with an explicit meshFederationRef, but that's very verbose, so maybe this proposal is going to be more popular.

Maybe we could opt for a mix? Have the meshFederationRef instead of matching on name, so that you can have >1 ServiceExport for a single MeshFederation. That way, you can either do it the way you planned here (admin does everything) or the admin can delegate to some power users. If we scope ServiceExport to its own namespace (or the whole mesh, if in control plane namespace), an admin could delegate export permissions for services in specific namespaces via RBAC. Wdyt?

* There may also be a `default` named ServiceExports resource, which will be applied to all remote meshes.  If a named ServiceExports resource exists, the alias rules in the named resource will take precedence.

If we go for meshFederationRef, not specifying one could have this behaviour. Not sure if that's something we'd want

* A ServiceExports resource contains a list of export rules.  Each rule defines:
  
  * A selector, which is used to select a service or set of services
  * An option alias, which defines the name of the service to be exposed to the remote mesh
  * The first rule that matches will be the rule used to export the service

There are two types of export rules:

* NameSelector allows for selecting a specific service by namespace and name and applying an optional alias for either/both namespace and name, e.g. foo/service-a might be mapped to bar/service-b or bar/service-a or foo/service-b
  
  * The selector may use `*` for either namespace or name to select services from any namespace or any name or all services (_/_)
  * Aliasing when using `*` will only apply to name or namespace if `*` is not used, e.g. if */service-a is the selector, only the name may be mapped, so if the alias name were service-b, you'd end up with foo/service-a and bar/service-a being exported as foo/service-b and bar/service-b

I can see a lot of edge cases with the aliasing. What if services overlap because of the aliasing? Ie we map two services onto the same NamespacedName?

* LabelSelector allows for selecting services from a specific namespace using a label selector.  It also specifies a list of name mappings to use for aliasing selected services.

LabelSelector is kind of delegation through the backdoor, right? If a service admin knows the label, they can export. And we can't use RBAC to control it - because if you own a Service, you can label it. So I'd say we drop LabelSelector

  * If namespace is omitted or specified a `*`, the label selector will be applied to services in all mesh namespaces.
  * Aliases work in the same manner as NameSelector (i.e. this is really a two tiered selector, with the label selector selecting the service, then matching an alias rule to determine the export name).  If no alias rule matches, a selected service is exported as-is.

All services are exported as namespace/name. Any domain suffix will be managed by the importing mesh. The rationale here is that this brings export/import of services more in line with how istio manages services in the service registry, as these map nicely to simple name and namespace fields. (Current thinking is that this would be defined in MeshFederation, or just use some convention like, .svc..local. We probably still need to figure out how the domain suffix bit plays into this, but I think this simplifies export for the user. The issues will show up on the import side anyway.)

+1 on this, I think this makes perfect sense

[rcernich]

@dgn, thanks for the feedback. I went with the 1:1, global model because I thought federation was something that should be restricted to the mesh administrator, as it exposes mesh resources to other parties. Given that, I didn't think it made sense to support delegation.

You're right, that label selector is sort of a back door for self-service exporting, which doesn't provide for any RBAC control, other than namespace scoping.

My thoughts on this were:

• If you're really uptight, only the mesh administrator should be exporting services. They are responsible for the mesh and are most likely the one to get called if services disappear (e.g. somebody unexported them, or a gateway goes down), or if a service that shouldn't have been exported gets exported.
• If you're not so uptight, you can use label selector, appropriated to select namespaces, essentially delegating responsibility to any user who can create/update services in those namespaces.
• If you're primarily concerned about HA/FO, you probably have some very simple glob rules that don't require much maintenance (e.g. export */*, no aliasing, as a default rule).

Regarding conflicts, for now I was just going to log errors if a duplicate occurs (first match wins), eventually providing status in a status resource. I agree there are going to be issues here. Originally, I was thinking about 1:1 with service, but that doesn't help with aliasing. I was anticipating that most cases of aliasing would be namespace-a -> namespace-b, so rules like namespace-a/* aliased to namespace-b/*. The other option is to require an explicit mapping for every export, which I thought was burdensome. It is a fallback solution though, as that is supported.

Using something like meshFederationRef is probably a good idea. One concern I had with supporting multiple export resources was how to manage the rule ordering, which is part of the reason I went with a single export resource (e.g. if there are overlapping rules in two export resources, which one takes precedence).

[rcernich]

I guess one thing I didn't touch on was whether or not globbing should be supported at all. I thought it made sense for things like ha/fo, e.g. export all services, or export all services from a specific namespace. Another side effect with globbing is that the set of services exported changes with the set of services in the mesh. I don't know if this is really an issue though.

[longmuir]

Finally going through this - some good discussion points here...

The real question is going to be: do we want the mesh administrator to handle all of it or can Service administrators expose their services themselves? Having individual resources is better for a sort of self-service model.

This is a great question...and one I had to think about... we often talk about "clusters admins" and "mesh admins" as being the two levels of administration, with the assumption being that the "mesh admin" is also the admin for all of the services. That is not always the case though - sometimes customers do want to divide an individual mesh into different "zones", often based on namespaces. I think the idea of being able to have multiple ServiceExport resources makes a lot of sense, as that would allow each resource to have different RBAC scoping (or other config differences). That said, if this gets complicated, I think it's ok if "service admins" have to ask "mesh admins" for permission to export services.

The label selector option offhand doesn't bother me too much - a lot of things work this way in K8s land. Choosing to use this or not will be in the hands of the mesh admin, so they need to be aware of the risks. A "mesh admin" should be aware that if they use a label selector, services will be able to export themselves. If we support "globbing" (There's strong demand for the HA use case, and I can see there being a desire for "export all"), that would be even higher risk, and they should be aware that they're "leaving the door open" for services to automatically be exported. If we do provide these different options, I think this is something we should definitely document these risks as a call out/warning with ServiceExport (cc @JStickler).

In general, I'll favour the simpler paths while meeting the two use cases of sharing distinct services and HA/FO, with us adding additional options/flexibility as requested by customers later. That is, start with one set of options that can be expanded on later. No doubt there will be a lot of requests that we'll have to tweak over time, and those can be hard to predict.

[dgn]

/retest

[rcernich]

@dgn, @knrc, @jwendell, @brian-avery, this should be ready to merge. just needs some reviews.