Goal - Ability to replace Envoy with Orion in Istio Sidecar Mode

[dawid-nowak]

We would like to Orion to replace Envoy in the Istio deployments. This issue is to describe and track the problems to get Istio Hello World running with Orion.

Running and Compiling Istio

In order to use Orion you will need to make re-create necessary Docker images. Check commit db6a785 and commit f063207 for the list of changes and how to rebuild the proxy image.

Running and Compiling Orion

The branch contains a list of necessary changes to get Orion up and running with Istio. Some of those changes are pretty hacky but they show the are that needs to be addressed in future work.

Adding Support for Unix Pipe Address Type.

Envoy and Istio Pilot agent use Unix Pipe to communicate xDS changes. This work has been started on branch

Ignoring Not Supported Parameters

At the moment, Orion is very strict about rejecting configuration parameters which are not supported currently. For example, if Orion receives a new listener with bypass_overload_manager set then currently, Orion would reject such a change. Unfortunately, Istio is using quite a lot of fields which are not currently supported by Orion and it is tedious to go through them one by one.

Suggestion would be to change unsupported_field to record warning instead of returning an error result.

Changes to the Admin interface and Stats

Generally speaking Admin and Stats is the place where the delta between Orion and Envoy is the biggest.

Liveness Detection

Istio Pilot detects if Envoy started successfully and is healthy by querying certain stats exposed in /stats endpoint.
See commit 114

Metrics

While Orion exposes some metrics via /stats/prometheus endpoint these metrics are not usable from the Istio integration perspective.

1. Orion exposes a lot of listener metrics which Istio doesn't seem to be interested in.
2. Orion doesn't really expose cluster related metrics.
3. Orion doesn't expose any Istio specific metrics which are added through Istio extensions.
4. Orion exposed metrics have different names... don't start with envoy_

The liveness detection metrics are crucial as without those the Istio will never move to the ready phase. The other metrics, are needed from the observability perspective.

SSL Context

Istio relies on CombinedSSL context validation which Orion was ignoring. Also, Istio/Envoy allows to be configured without and cert validation which was prohibited in Orion. See changes

xDS Configuration Changes

Istio Agent and Envoy use Delta ADS for the configuration. Orion's implementation while correct from the Envoy's perspective was not working out of the box when connecting to Istio Agent.

Metadata passing

Istio agent needs that the metadata field is populated with the data from the bootstrap file. Commit

Resource Discovery

By default Orion was using a wildcard subscription for all resources (listeners, cluster, cluster load assignments, routes) . This doesn't work with Istio where the wildcard subscription is only used for listeners and clusters. Changes are required to register for cluster load assignments and routes when clusters and listeners are processed by Orion.

See commit and commit

Orion Cluster Logic

In order to satisfy Istio, Orions there are some relatively minor changes required around how we handle changes to cluster load assignment fields. Previously, in Orion changes to cluster load assignment for Static cluster wouldn't be allowed. Similarly, we wouldn't allow an EDS cluster with statically set cluster load assignment.
Also, some more changes were required to handling OriginalDstCluster where logic needed to be added to handle routing by Authority and to bind the original address to an address specified in the bind device listener configuration. The authority routing and device port binding was particularly needed to ensure that traffic forwarded by Orion is correctly routed by Istio pilot iptables.

Filterchain selection

Certain rules were missing in Orions filter chain selection logic. See commit

[awgn]

Currently, Orion exposes several metrics at /stats/prometheus. In particular, some of them are related to upstream statistics:

• UPSTREAM_RQ_TOTAL
• UPSTREAM_RQ_ACTIVE
• UPSTREAM_RQ_TIMEOUT
• UPSTREAM_RQ_PER_TRY_TIMEOUT
• UPSTREAM_RQ_RETRY
• UPSTREAM_CX_TOTAL
• UPSTREAM_CX_IDLE_TIMEOUT
• UPSTREAM_CX_CONNECT_FAIL
• UPSTREAM_CX_DESTROY
• UPSTREAM_CX_ACTIVE

Regarding cluster-specific stats, we currently do not support them. However, since they are few in number, adding them should not be too difficult.

At the moment, these counters are lazy, meaning they are only exported once they are first populated. If this behavior is problematic, we can run an initialisation function during startup to populate them so that they are exported with a default value of 0.

Regarding naming conventions, I’m not sure what the most appropriate format is, but prefixing them with envoy_ if necessary should not be a major issue.

Other than that, supporting additional Istio metrics requires, instead, a more in-depth analysis.

[dawid-nowak]

We need to identify which metrics are used by Istio for their dashboards in Kiali and implement those. + the metrics needed to report the readiness of Orion to Istio.

[YaoZengzeng]

cc @Eeshu-Yadav you could move forward on the work of replacing waypoint of Kmesh by orion, based on the work that dawid have done.

[dawid-nowak]

This work is for sidecar mode, let me know what specific asks there are in terms of waypoint

[Eeshu-Yadav]

cc @Eeshu-Yadav you could move forward on the work of replacing waypoint of Kmesh by orion, based on the work that dawid have done.

okk will proceed it