http: support route mutability

docs/root/intro/arch_overview/http/http_filters.rst

@@ -48,3 +48,26 @@ decoder/encoder filters):
 The connection manager will invoke decoder filters in the order: ``A``, ``B``, ``C``.
 On the other hand, the connection manager will invoke encoder filters in the **reverse**
 order: ``C``, ``B``, ``A``.
+
+.. _arch_overview_http_filters_route_mutation:
+
+Filter route mutation
+---------------------
+
+During HTTP filter chain processing, when ``decodeHeaders()`` is invoked by a filter, the
+connection manager performs route resolution and sets a *cached route* pointing to an upstream
+cluster.
+
+Filters have the capability to directly mutate this *cached route* after route resolution, via the
+``setRoute`` callback and :repo:`DelegatingRoute <source/common/router/delegating_route_impl.h>`
+mechanism. A filter may create a derived/child class of ``DelegatingRoute`` to override specific
+methods (for example, the route’s timeout value or the route entry’s cluster name) while preserving
+the rest of the properties/behavior of the base route that the ``DelegatingRoute`` wraps around.
+Then, ``setRoute`` can be invoked to manually set the cached route to this ``DelegatingRoute``
+instance. An example of such a derived class can be found in :repo:`ExampleDerivedDelegatingRoute
+<test/test_common/delegating_route_utility.h>`.
+
+If no other filters in the chain modify the cached route selection (for example, a common operation
+that filters do is ``clearRouteCache()``, and ``setRoute`` will not survive that), this route
+selection makes it way to the router filter which finalizes the upstream cluster that the request
+will get forwarded to.

docs/root/intro/life_of_a_request.rst

@@ -434,8 +434,9 @@ When ``decodeHeaders()`` is invoked on the :ref:`router <arch_overview_http_rout
 route selection is finalized and a cluster is picked. The HCM selects a route from its
 ``RouteConfiguration`` at the start of HTTP filter chain execution. This is referred to as the
 *cached route*. Filters may modify headers and cause a new route to be selected, by asking HCM to
-clear the route cache and requesting HCM to reevaluate the route selection. When the router filter
-is invoked, the route is finalized. The selected route’s configuration will point at an upstream
+clear the route cache and requesting HCM to reevaluate the route selection. Filters may also
+directly set this cached route selection via a `setRoute` callback. When the router filter is
+invoked, the route is finalized. The selected route’s configuration will point at an upstream
 cluster name. The router filter then asks the `ClusterManager` for an HTTP :ref:`connection pool
 <arch_overview_conn_pool>` for the cluster. This involves load balancing and the connection pool,
 discussed in the next section.

include/envoy/http/filter.h

@@ -225,6 +225,24 @@ class StreamFilterCallbacks {
    */
   virtual Router::RouteConstSharedPtr route(const Router::RouteCallback& cb) PURE;
 
+  /**
+   * Sets the cached route for the current request to the passed-in RouteConstSharedPtr parameter.
+   *
+   * Similar to route(const Router::RouteCallback& cb), this route that is set will be
+   * overridden by clearRouteCache() in subsequent filters. Usage is intended for filters at the end
+   * of the filter chain.
+   *
+   * NOTE: Passing nullptr in as the route parameter is equivalent to route resolution being
+   * attempted and failing to find a route. An example of when this happens is when
+   * RouteConstSharedPtr route(const RouteCallback& cb, const Http::RequestHeaderMap& headers, const
+   * StreamInfo::StreamInfo& stream_info, uint64_t random_value) returns nullptr during a
+   * refreshCachedRoute. It is important to note that setRoute(nullptr) is different from a
+   * clearRouteCache(), because clearRouteCache() wants route resolution to be attempted again.
+   * clearRouteCache() achieves this by setting cached_route_ and cached_cluster_info_ to
+   * absl::optional ptrs instead of null ptrs.
+   */
+  virtual void setRoute(Router::RouteConstSharedPtr route) PURE;
+
   /**
    * Returns the clusterInfo for the cached route.
    * This method is to avoid multiple look ups in the filter chain, it also provides a consistent

source/common/http/async_client_impl.h

@@ -361,6 +361,7 @@ class AsyncStreamImpl : public AsyncClient::Stream,
   Router::RouteConstSharedPtr route(const Router::RouteCallback&) override {
     NOT_IMPLEMENTED_GCOVR_EXCL_LINE;
   }
+  void setRoute(Router::RouteConstSharedPtr) override { NOT_IMPLEMENTED_GCOVR_EXCL_LINE; }
   Upstream::ClusterInfoConstSharedPtr clusterInfo() override { return parent_.cluster_; }
   void clearRouteCache() override {}
   uint64_t streamId() const override { return stream_id_; }

source/common/http/conn_manager_impl.cc

@@ -1263,20 +1263,8 @@ void ConnectionManagerImpl::ActiveStream::refreshCachedRoute(const Router::Route
                                            stream_id_);
     }
   }
-  filter_manager_.streamInfo().route_entry_ = route ? route->routeEntry() : nullptr;
-  cached_route_ = std::move(route);
-  if (nullptr == filter_manager_.streamInfo().route_entry_) {
-    cached_cluster_info_ = nullptr;
-  } else {
-    Upstream::ThreadLocalCluster* local_cluster =
-        connection_manager_.cluster_manager_.getThreadLocalCluster(
-            filter_manager_.streamInfo().route_entry_->clusterName());
-    cached_cluster_info_ = (nullptr == local_cluster) ? nullptr : local_cluster->info();
-  }
 
-  filter_manager_.streamInfo().setUpstreamClusterInfo(cached_cluster_info_.value());
-  refreshCachedTracingCustomTags();
-  refreshDurationTimeout();
+  setRoute(route);
 }
 
 void ConnectionManagerImpl::ActiveStream::refreshCachedTracingCustomTags() {
@@ -1590,6 +1578,31 @@ ConnectionManagerImpl::ActiveStream::route(const Router::RouteCallback& cb) {
   return cached_route_.value();
 }
 
+/**
+ * Sets the cached route to the RouteConstSharedPtr argument passed in. Handles setting the
+ * cached_route_/cached_cluster_info_ ActiveStream attributes, the FilterManager streamInfo, tracing
+ * tags, and timeouts.
+ *
+ * Declared as a StreamFilterCallbacks member function for filters to call directly, but also
+ * functions as a helper to refreshCachedRoute(const Router::RouteCallback& cb).
+ */
+void ConnectionManagerImpl::ActiveStream::setRoute(Router::RouteConstSharedPtr route) {
+  filter_manager_.streamInfo().route_entry_ = route ? route->routeEntry() : nullptr;
+  cached_route_ = std::move(route);
+  if (nullptr == filter_manager_.streamInfo().route_entry_) {
+    cached_cluster_info_ = nullptr;
+  } else {
+    Upstream::ThreadLocalCluster* local_cluster =
+        connection_manager_.cluster_manager_.getThreadLocalCluster(
+            filter_manager_.streamInfo().route_entry_->clusterName());
+    cached_cluster_info_ = (nullptr == local_cluster) ? nullptr : local_cluster->info();
+  }
+
+  filter_manager_.streamInfo().setUpstreamClusterInfo(cached_cluster_info_.value());
+  refreshCachedTracingCustomTags();
+  refreshDurationTimeout();
+}
+
 void ConnectionManagerImpl::ActiveStream::clearRouteCache() {
   cached_route_ = absl::optional<Router::RouteConstSharedPtr>();
   cached_cluster_info_ = absl::optional<Upstream::ClusterInfoConstSharedPtr>();

source/common/http/conn_manager_impl.h

@@ -270,6 +270,7 @@ class ConnectionManagerImpl : Logger::Loggable<Logger::Id::http>,
     const Router::RouteEntry::UpgradeMap* upgradeMap() override;
     Upstream::ClusterInfoConstSharedPtr clusterInfo() override;
     Router::RouteConstSharedPtr route(const Router::RouteCallback& cb) override;
+    void setRoute(Router::RouteConstSharedPtr route) override;
     void clearRouteCache() override;
     absl::optional<Router::ConfigConstSharedPtr> routeConfig() override;
     Tracing::Span& activeSpan() override;

source/common/http/filter_manager.cc

@@ -242,6 +242,10 @@ Router::RouteConstSharedPtr ActiveStreamFilterBase::route(const Router::RouteCal
   return parent_.filter_manager_callbacks_.route(cb);
 }
 
+void ActiveStreamFilterBase::setRoute(Router::RouteConstSharedPtr route) {
+  parent_.filter_manager_callbacks_.setRoute(std::move(route));
+}
+
 void ActiveStreamFilterBase::clearRouteCache() {
   parent_.filter_manager_callbacks_.clearRouteCache();
 }

source/common/http/filter_manager.h

@@ -133,6 +133,7 @@ struct ActiveStreamFilterBase : public virtual StreamFilterCallbacks,
   void resetStream() override;
   Router::RouteConstSharedPtr route() override;
   Router::RouteConstSharedPtr route(const Router::RouteCallback& cb) override;
+  void setRoute(Router::RouteConstSharedPtr route) override;
   Upstream::ClusterInfoConstSharedPtr clusterInfo() override;
   void clearRouteCache() override;
   uint64_t streamId() const override;
@@ -510,6 +511,11 @@ class FilterManagerCallbacks {
    */
   virtual Router::RouteConstSharedPtr route(const Router::RouteCallback& cb) PURE;
 
+  /**
+   * Sets the current route.
+   */
+  virtual void setRoute(Router::RouteConstSharedPtr route) PURE;
+
   /**
    * Clears the cached route.
    */

source/common/router/BUILD

@@ -417,3 +417,12 @@ envoy_cc_library(
         "//include/envoy/router:string_accessor_interface",
     ],
 )
+
+envoy_cc_library(
+    name = "delegating_route_lib",
+    srcs = ["delegating_route_impl.cc"],
+    hdrs = ["delegating_route_impl.h"],
+    deps = [
+        "//include/envoy/router:router_interface",
+    ],
+)

source/common/router/delegating_route_impl.cc

@@ -0,0 +1,188 @@
+#include "common/router/delegating_route_impl.h"
+
+namespace Envoy {
+namespace Router {
+
+// Router::DelegatingRoute
+const DirectResponseEntry* DelegatingRoute::directResponseEntry() const {
+  return base_route_->directResponseEntry();
+}
+
+const RouteEntry* DelegatingRoute::routeEntry() const { return base_route_->routeEntry(); }
+
+const Decorator* DelegatingRoute::decorator() const { return base_route_->decorator(); }
+
+const RouteTracing* DelegatingRoute::tracingConfig() const { return base_route_->tracingConfig(); }
+
+const RouteSpecificFilterConfig* DelegatingRoute::perFilterConfig(const std::string& name) const {
+  return base_route_->perFilterConfig(name);
+}
+
+// Router:DelegatingRouteEntry
+void DelegatingRouteEntry::finalizeResponseHeaders(
+    Http::ResponseHeaderMap& headers, const StreamInfo::StreamInfo& stream_info) const {
+  return base_route_->routeEntry()->finalizeResponseHeaders(headers, stream_info);
+}
+
+Http::HeaderTransforms
+DelegatingRouteEntry::responseHeaderTransforms(const StreamInfo::StreamInfo& stream_info) const {
+  return base_route_->routeEntry()->responseHeaderTransforms(stream_info);
+}
+
+const std::string& DelegatingRouteEntry::clusterName() const {
+  return base_route_->routeEntry()->clusterName();
+}
+
+Http::Code DelegatingRouteEntry::clusterNotFoundResponseCode() const {
+  return base_route_->routeEntry()->clusterNotFoundResponseCode();
+}
+
+const CorsPolicy* DelegatingRouteEntry::corsPolicy() const {
+  return base_route_->routeEntry()->corsPolicy();
+}
+
+absl::optional<std::string>
+DelegatingRouteEntry::currentUrlPathAfterRewrite(const Http::RequestHeaderMap& headers) const {
+  return base_route_->routeEntry()->currentUrlPathAfterRewrite(headers);
+}
+
+void DelegatingRouteEntry::finalizeRequestHeaders(Http::RequestHeaderMap& headers,
+                                                  const StreamInfo::StreamInfo& stream_info,
+                                                  bool insert_envoy_original_path) const {
+  return base_route_->routeEntry()->finalizeRequestHeaders(headers, stream_info,
+                                                           insert_envoy_original_path);
+}
+
+const Http::HashPolicy* DelegatingRouteEntry::hashPolicy() const {
+  return base_route_->routeEntry()->hashPolicy();
+}
+
+const HedgePolicy& DelegatingRouteEntry::hedgePolicy() const {
+  return base_route_->routeEntry()->hedgePolicy();
+}
+
+Upstream::ResourcePriority DelegatingRouteEntry::priority() const {
+  return base_route_->routeEntry()->priority();
+}
+
+const RateLimitPolicy& DelegatingRouteEntry::rateLimitPolicy() const {
+  return base_route_->routeEntry()->rateLimitPolicy();
+}
+
+const RetryPolicy& DelegatingRouteEntry::retryPolicy() const {
+  return base_route_->routeEntry()->retryPolicy();
+}
+
+const InternalRedirectPolicy& DelegatingRouteEntry::internalRedirectPolicy() const {
+  return base_route_->routeEntry()->internalRedirectPolicy();
+}
+
+uint32_t DelegatingRouteEntry::retryShadowBufferLimit() const {
+  return base_route_->routeEntry()->retryShadowBufferLimit();
+}
+
+const std::vector<Router::ShadowPolicyPtr>& DelegatingRouteEntry::shadowPolicies() const {
+  return base_route_->routeEntry()->shadowPolicies();
+}
+
+std::chrono::milliseconds DelegatingRouteEntry::timeout() const {
+  return base_route_->routeEntry()->timeout();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::idleTimeout() const {
+  return base_route_->routeEntry()->idleTimeout();
+}
+
+bool DelegatingRouteEntry::usingNewTimeouts() const {
+  return base_route_->routeEntry()->usingNewTimeouts();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::maxStreamDuration() const {
+  return base_route_->routeEntry()->maxStreamDuration();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::grpcTimeoutHeaderMax() const {
+  return base_route_->routeEntry()->grpcTimeoutHeaderMax();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::grpcTimeoutHeaderOffset() const {
+  return base_route_->routeEntry()->grpcTimeoutHeaderOffset();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::maxGrpcTimeout() const {
+  return base_route_->routeEntry()->maxGrpcTimeout();
+}
+
+absl::optional<std::chrono::milliseconds> DelegatingRouteEntry::grpcTimeoutOffset() const {
+  return base_route_->routeEntry()->grpcTimeoutOffset();
+}
+
+const VirtualCluster* DelegatingRouteEntry::virtualCluster(const Http::HeaderMap& headers) const {
+  return base_route_->routeEntry()->virtualCluster(headers);
+}
+
+const VirtualHost& DelegatingRouteEntry::virtualHost() const {
+  return base_route_->routeEntry()->virtualHost();
+}
+
+bool DelegatingRouteEntry::autoHostRewrite() const {
+  return base_route_->routeEntry()->autoHostRewrite();
+}
+
+const MetadataMatchCriteria* DelegatingRouteEntry::metadataMatchCriteria() const {
+  return base_route_->routeEntry()->metadataMatchCriteria();
+}
+
+const std::multimap<std::string, std::string>& DelegatingRouteEntry::opaqueConfig() const {
+  return base_route_->routeEntry()->opaqueConfig();
+}
+
+bool DelegatingRouteEntry::includeVirtualHostRateLimits() const {
+  return base_route_->routeEntry()->includeVirtualHostRateLimits();
+}
+
+const Envoy::Config::TypedMetadata& DelegatingRouteEntry::typedMetadata() const {
+  return base_route_->routeEntry()->typedMetadata();
+}
+
+const envoy::config::core::v3::Metadata& DelegatingRouteEntry::metadata() const {
+  return base_route_->routeEntry()->metadata();
+}
+
+const TlsContextMatchCriteria* DelegatingRouteEntry::tlsContextMatchCriteria() const {
+  return base_route_->routeEntry()->tlsContextMatchCriteria();
+}
+
+const PathMatchCriterion& DelegatingRouteEntry::pathMatchCriterion() const {
+  return base_route_->routeEntry()->pathMatchCriterion();
+}
+
+const RouteSpecificFilterConfig*
+DelegatingRouteEntry::perFilterConfig(const std::string& name) const {
+  return base_route_->routeEntry()->perFilterConfig(name);
+}
+
+bool DelegatingRouteEntry::includeAttemptCountInRequest() const {
+  return base_route_->routeEntry()->includeAttemptCountInRequest();
+}
+
+bool DelegatingRouteEntry::includeAttemptCountInResponse() const {
+  return base_route_->routeEntry()->includeAttemptCountInResponse();
+}
+
+using UpgradeMap = std::map<std::string, bool>;
+const UpgradeMap& DelegatingRouteEntry::upgradeMap() const {
+  return base_route_->routeEntry()->upgradeMap();
+}
+
+using ConnectConfig = envoy::config::route::v3::RouteAction::UpgradeConfig::ConnectConfig;
+const absl::optional<ConnectConfig>& DelegatingRouteEntry::connectConfig() const {
+  return base_route_->routeEntry()->connectConfig();
+}
+
+const std::string& DelegatingRouteEntry::routeName() const {
+  return base_route_->routeEntry()->routeName();
+}
+
+} // namespace Router
+} // namespace Envoy