Redirection support, as alternative to request forwarding

CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [5.0.0-dev15]
+
+[5.0.0-dev15]: https://github.com/microsoft/CCF/releases/tag/ccf-5.0.0-dev15
+
+### Added
+
+- CCF now supports a mode where HTTP redirect responses are returned, rather than relying on internal forwarding. This can be used to ensure that write requests are executed on a primary, even if they are initially sent to a backup. This behaviour is enabled by setting the `redirections` field in an `rpc_interface` within cchost's launch config. This can be configured to redirect either directly to a node (if each node has a distinct, accessible name), or to a static load balancer address, depending on the current deployment. See docs for description of [redirection behaviour](https://microsoft.github.io/CCF/main/architecture/request_flow.html#redirection-flow) and [configuration](https://microsoft.github.io/CCF/main/operations/configuration.html#redirections).
+
 ## [5.0.0-dev14]
 
 [5.0.0-dev14]: https://github.com/microsoft/CCF/releases/tag/ccf-5.0.0-dev14

CMakeLists.txt

@@ -1378,12 +1378,17 @@ if(BUILD_TESTS)
                       ${CMAKE_SOURCE_DIR}/samples/apps/logging/js
     )
 
+    # This test uses large requests (so too slow for SAN)
     if(NOT SAN)
       add_e2e_test(
         NAME e2e_limits PYTHON_SCRIPT ${CMAKE_SOURCE_DIR}/tests/limits.py
       )
     endif()
 
+    add_e2e_test(
+      NAME e2e_redirects PYTHON_SCRIPT ${CMAKE_SOURCE_DIR}/tests/redirects.py
+    )
+
     add_e2e_test(
       NAME e2e_logging_http2
       PYTHON_SCRIPT ${CMAKE_SOURCE_DIR}/tests/e2e_logging.py

doc/architecture/request_flow.rst

@@ -130,6 +130,65 @@ When follower A receives the forwarded response, it writes this to the TLS sessi
 
       NetStackA-->>User: 200 OK "Copied {a} from {A} to {B}"
 
+Redirection flow
+----------------
+
+CCF supports HTTP redirections as an alternative to forwarding. When a request arrives that cannot be executed locally, rather than forwarding it to an appropriate node over the node-to-node channels, the node can return a HTTP redirect response advising the caller to resubmit the request directly to that node. This uses standard HTTP semantics, reporting the redirect target in a ``Location`` header. Most HTTP clients will have an option to follow this redirect automatically, and all should have an option to enable this behaviour if desired. Alternatively, client applications may choose to intercept this redirect response and manually interpret it, perhaps to alter the resubmitted request or to update the target node for future requests.
+
+.. warning:: Many HTTP clients will strip out ``Authorization`` headers when following Cross-Origin redirects. This means that if your client is automatically following redirects, and you submit a request with a JWT token as authorization, if you are redirected you may see a surprising authorization failure. In this scenario we recommend intercepting the redirect responses manually, so that the request can be resubmitted without stripping headers.
+
+Similar to forwarding, the redirect behaviour is partly controlled by per-endpoint metadata, so the initially receiving node must parse the request and go through endpoint dispatch before making a forwarding decision.
+
+There are currently 2 supported modes for redirections. In the first, the response sends the user directly to the suggested node. This will only work if that node has an accessible name, which can be included in the ``Location`` header and accessed by the user.
+
+.. mermaid::
+
+  sequenceDiagram
+      autonumber
+      participant U as User
+      participant B as Backup (nodeA.ccf.com)
+      participant P as Primary (nodeB.ccf.com)
+
+      U->>B: POST /copy/A/B
+      B->>B: Lookup endpoint
+      B->>B: Decide request should be redirected
+      B->>B: Build redirect response
+      B-->>U: 307 REDIRECT Location: nodeB.ccf.com/copy/A/B
+
+      U->>P: POST /copy/A/B
+      P->>P: Lookup endpoint
+      P->>P: Decide request can be executed
+      P->>P: Execute request
+      P-->>U: 200 OK "Copied {a} from {A} to {B}"
+
+For deployments where nodes are not directly accessible, redirections can still be supported via multiple load balancers. All that is required is `a` public name for each redirect purpose, with up-to-date balancing to the correct nodes. More simply, that currently means maintaining a `write` load balancer which can direct external traffic to a primary.
+
+.. mermaid::
+
+  sequenceDiagram
+      autonumber
+      participant U as User
+      participant LB as General LB (service.ccf.com)
+      participant B as Backup
+      participant WLB as Write LB (write.service.ccf.com)
+      participant P as Primary
+
+      U->>LB: POST /copy/A/B
+      LB->>B: POST /copy/A/B
+      B->>B: Lookup endpoint
+      B->>B: Decide request should be redirected
+      B->>B: Build redirect response
+      B-->>U: 307 REDIRECT Location: write.service.ccf.com/copy/A/B
+
+      U->>WLB: POST /copy/A/B
+      WLB->>P: POST /copy/A/B
+      P->>P: Lookup endpoint
+      P->>P: Decide request can be executed
+      P->>P: Execute request
+      P-->>U: 200 OK "Copied {a} from {A} to {B}"
+
+To use redirection behaviour, and choose whether to redirect to a node or a load balancer, set the ``redirections`` field in the :doc:`cchost launch configuration </operations/configuration>`.
+
 External executor flow
 ----------------------
 

doc/host_config_schema/cchost_config.json

@@ -161,6 +161,17 @@
                 "type": "integer",
                 "default": 3000,
                 "description": "Timeout for forwarded RPC calls (in milliseconds)"
+              },
+              "redirections": {
+                "type": "object",
+                "description": "Configure how redirect responses should be produced on this interface. If this is omitted, then forwarding will be used instead",
+                "properties": {
+                  "to_primary": {
+                    "$ref": "#/$defs/RedirectionResolver",
+                    "description": "Configures how the Location header should be populated, when requests arrive on this interface that must be served by a primary while the receiving node is not a primary"
+                  }
+                },
+                "additionalProperties": false
               }
             },
             "required": ["bind_address"]
@@ -687,5 +698,54 @@
     }
   },
   "required": ["enclave", "network", "command"],
-  "additionalProperties": false
+  "additionalProperties": false,
+  "$defs": {
+    "RedirectionResolver": {
+      "type": "object",
+      "properties": {
+        "kind": {
+          "enum": ["NodeByRole", "StaticAddress"]
+        },
+        "target": {}
+      },
+      "required": ["kind"],
+      "if": {
+        "properties": {
+          "kind": {
+            "const": "NodeByRole"
+          }
+        }
+      },
+      "then": {
+        "properties": {
+          "target": {
+            "type": "object",
+            "properties": {
+              "role": {
+                "enum": ["primary"],
+                "default": "primary"
+              }
+            },
+            "additionalProperties": false
+          }
+        }
+      },
+      "else": {
+        "properties": {
+          "target": {
+            "type": "object",
+            "properties": {
+              "address": {
+                "type": "string"
+              }
+            },
+            "required": ["address"],
+            "additionalProperties": false
+          }
+        },
+        "required": ["target"]
+      },
+      "additionalProperties": false
+    }
+  }
 }

doc/schemas/gov_openapi.json

@@ -686,6 +686,27 @@
               },
               "published_address": {
                 "type": "string"
+              },
+              "redirections": {
+                "properties": {
+                  "to_primary": {
+                    "properties": {
+                      "kind": {
+                        "enum": [
+                          "NodeByRole",
+                          "StaticAddress"
+                        ],
+                        "type": "string"
+                      },
+                      "target": {}
+                    },
+                    "required": [
+                      "kind"
+                    ],
+                    "type": "object"
+                  }
+                },
+                "type": "object"
               }
             },
             "required": [
@@ -788,6 +809,27 @@
                 },
                 "published_address": {
                   "type": "string"
+                },
+                "redirections": {
+                  "properties": {
+                    "to_primary": {
+                      "properties": {
+                        "kind": {
+                          "enum": [
+                            "NodeByRole",
+                            "StaticAddress"
+                          ],
+                          "type": "string"
+                        },
+                        "target": {}
+                      },
+                      "required": [
+                        "kind"
+                      ],
+                      "type": "object"
+                    }
+                  },
+                  "type": "object"
                 }
               },
               "required": [
@@ -1283,7 +1325,7 @@
   "info": {
     "description": "This API is used to submit and query proposals which affect CCF's public governance tables.",
     "title": "CCF Governance API",
-    "version": "4.1.3"
+    "version": "4.1.4"
   },
   "openapi": "3.0.0",
   "paths": {

doc/schemas/node_openapi.json

@@ -584,13 +584,24 @@
           },
           "published_address": {
             "$ref": "#/components/schemas/string"
+          },
+          "redirections": {
+            "$ref": "#/components/schemas/NodeInfoNetwork_v2__NetInterface__Redirections"
           }
         },
         "required": [
           "bind_address"
         ],
         "type": "object"
       },
+      "NodeInfoNetwork_v2__NetInterface__Redirections": {
+        "properties": {
+          "to_primary": {
+            "$ref": "#/components/schemas/RedirectionResolverConfig"
+          }
+        },
+        "type": "object"
+      },
       "NodeMetrics": {
         "properties": {
           "sessions": {
@@ -686,6 +697,27 @@
         ],
         "type": "string"
       },
+      "RedirectionResolutionKind": {
+        "enum": [
+          "NodeByRole",
+          "StaticAddress"
+        ],
+        "type": "string"
+      },
+      "RedirectionResolverConfig": {
+        "properties": {
+          "kind": {
+            "$ref": "#/components/schemas/RedirectionResolutionKind"
+          },
+          "target": {
+            "$ref": "#/components/schemas/json"
+          }
+        },
+        "required": [
+          "kind"
+        ],
+        "type": "object"
+      },
       "RetirementPhase": {
         "enum": [
           "Ordered",
@@ -905,7 +937,7 @@
   "info": {
     "description": "This API provides public, uncredentialed access to service and node state.",
     "title": "CCF Public Node API",
-    "version": "4.9.0"
+    "version": "4.9.1"
   },
   "openapi": "3.0.0",
   "paths": {

include/ccf/endpoint.h

@@ -58,6 +58,20 @@ namespace ccf::endpoints
     Never
   };
 
+  enum class RedirectionStrategy
+  {
+    /** This operation does not need to be redirected, and can be executed on
+       the receiving node. Most read-only operations can be executed on any
+       node, so should be marked as None.  */
+    None,
+
+    /** This operation must be executed on a primary. If the current node is not
+       a primary, it should attempt to redirect to the primary, or else return
+       an error. Any write operations must be executed on a primary, so should
+       be marked as ToPrimary. */
+    ToPrimary
+  };
+
   enum class Mode
   {
     ReadWrite,
@@ -77,6 +91,11 @@ namespace ccf::endpoints
      {ForwardingRequired::Always, "always"},
      {ForwardingRequired::Never, "never"}});
 
+  DECLARE_JSON_ENUM(
+    RedirectionStrategy,
+    {{RedirectionStrategy::None, "none"},
+     {RedirectionStrategy::ToPrimary, "to_primary"}});
+
   DECLARE_JSON_ENUM(
     Mode,
     {{Mode::ReadWrite, "readwrite"},
@@ -106,6 +125,8 @@ namespace ccf::endpoints
     Mode mode = Mode::ReadWrite;
     /// Endpoint forwarding policy
     ForwardingRequired forwarding_required = ForwardingRequired::Always;
+    /// Endpoint redirection policy
+    RedirectionStrategy redirection_strategy = RedirectionStrategy::None;
     /// Authentication policies
     std::vector<std::string> authn_policies = {};
     /// OpenAPI schema for endpoint
@@ -397,6 +418,8 @@ namespace ccf::endpoints
      */
     Endpoint& set_forwarding_required(ForwardingRequired fr);
 
+    Endpoint& set_redirection_strategy(RedirectionStrategy rs);
+
     void install();
   };