Policy-based routing (PBR) in OVN.

PBR provides a mechanism to configure permit/deny and reroute policies on the
router. Permit/deny policies are similar to OVN ACLs, but exist on the
logical-router. Reroute policies are needed for service-insertion and
service-chaining. Currently, policies are stateless.

To achieve this, a new table is introduced in the ingress pipeline of the
Logical-router. The new table is between the ‘IP Routing’ and the ‘ARP/ND
resolution’ table. This way, PBR can override routing decisions and provide a
different next-hop.

This Patch:
a. Changes in OVN NB Schema to introduce a new table in the Logical
router.
b. Add commands to ovn-nbctl to add/delete/list routing policies.
c. Changes in ovn-northd to process routing-policy configurations.

 A new table 'Logical_Router_Policy' has been added in the northbound schema.
The table has the following columns:
      * priority: Rules with numerically higher priority take precedence over
        those with lower.
      * match: Uses the same expression language as the 'match' column of
       'Logical_Flow' table in the OVN Southbound database.
      * action: allow/drop/reroute nexthop: Nexthop IP address.

Each row in this table represents one routing policy for a logical router. The
'action' column for the highest priority matching row in this table determines a
packet's treatment. If no row matches, packets are allowed by default.

The new ovn-nbctl commands are as follows:
     1. Add a new ovn-nbctl command to add a routing policy.
     lr-policy-add ROUTER PRIORITY MATCH ACTION [NEXTHOP]

        Nexthop is an optional parameter. It needs to be provided only when
'action' is 'reroute'. A policy is uniquely identified by priority and match.
Multiple policies can have the same priority.

     2. Add a new ovn-nbctl command to delete a routing policy.
     lr-policy-del ROUTER [PRIORITY [MATCH]]

        Takes priority and match as optional parameters. If priority and match
are specified, the policy with the given priority and match is deleted. If
priority is specified and match is not specified, all rules with that priority
are deleted.  If priority is not specified, all the rules would be deleted.

     3. Add a new ovn-nbctl command to list routing-policies in the logical
router.
     lr-policy-list ROUTER

        ovn-northd changes are to get routing-policies from northbound database
and populate the same as logical flows in the southbound database. A new table
called 'POLICY' is introduced in the Logical router's ingress pipeline. Each
routing-policy configured in the northbound database translates into a single
logical flow in the new table.

        The columns from the Logical_Router_Policy table are used as follows:
The priority column is used as priority in the logical-flow. The match column
is used as the 'match' string in the logical-flow. The action column is used to
determine the action of the logical-flow.

        When the 'action' is reroute, if the nexthop ip-address is a connected
router port or the IP address of a logical port, the logical-flow is constructed
to route the packet to the nexthop ip-address.

Signed-off-by: Mary Manohar <mary.manohar@nutanix.com>
Signed-off-by: Ben Pfaff <blp@ovn.org>

ovn/northd/ovn-northd.c

@@ -143,9 +143,10 @@ enum ovn_stage {
     PIPELINE_STAGE(ROUTER, IN,  ND_RA_OPTIONS,  5, "lr_in_nd_ra_options") \
     PIPELINE_STAGE(ROUTER, IN,  ND_RA_RESPONSE, 6, "lr_in_nd_ra_response") \
     PIPELINE_STAGE(ROUTER, IN,  IP_ROUTING,     7, "lr_in_ip_routing")   \
-    PIPELINE_STAGE(ROUTER, IN,  ARP_RESOLVE,    8, "lr_in_arp_resolve")  \
-    PIPELINE_STAGE(ROUTER, IN,  GW_REDIRECT,    9, "lr_in_gw_redirect")  \
-    PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,    10, "lr_in_arp_request")  \
+    PIPELINE_STAGE(ROUTER, IN,  POLICY,         8, "lr_in_policy")       \
+    PIPELINE_STAGE(ROUTER, IN,  ARP_RESOLVE,    9, "lr_in_arp_resolve")  \
+    PIPELINE_STAGE(ROUTER, IN,  GW_REDIRECT,    10, "lr_in_gw_redirect")  \
+    PIPELINE_STAGE(ROUTER, IN,  ARP_REQUEST,    11, "lr_in_arp_request")  \
                                                                       \
     /* Logical router egress stages. */                               \
     PIPELINE_STAGE(ROUTER, OUT, UNDNAT,    0, "lr_out_undnat")        \
@@ -5010,6 +5011,80 @@ find_lrp_member_ip(const struct ovn_port *op, const char *ip_s)
     return NULL;
 }
 
+static struct ovn_port*
+get_outport_for_routing_policy_nexthop(struct ovn_datapath *od,
+                                       struct hmap *ports,
+                                       int priority, const char *nexthop)
+{
+    if (nexthop == NULL) {
+        return NULL;
+    }
+
+    /* Find the router port matching the next hop. */
+    for (int i = 0; i < od->nbr->n_ports; i++) {
+       struct nbrec_logical_router_port *lrp = od->nbr->ports[i];
+
+       struct ovn_port *out_port = ovn_port_find(ports, lrp->name);
+       if (out_port && find_lrp_member_ip(out_port, nexthop)) {
+           return out_port;
+       }
+    }
+
+    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
+    VLOG_WARN_RL(&rl, "No path for routing policy priority %d; next hop %s",
+                 priority, nexthop);
+    return NULL;
+}
+
+static void
+build_routing_policy_flow(struct hmap *lflows, struct ovn_datapath *od,
+                          struct hmap *ports,
+                          const struct nbrec_logical_router_policy *rule)
+{
+    struct ds match = DS_EMPTY_INITIALIZER;
+    struct ds actions = DS_EMPTY_INITIALIZER;
+
+    if (!strcmp(rule->action, "reroute")) {
+        struct ovn_port *out_port = get_outport_for_routing_policy_nexthop(
+             od, ports, rule->priority, rule->nexthop);
+        if (!out_port) {
+            return;
+        }
+
+        const char *lrp_addr_s = find_lrp_member_ip(out_port, rule->nexthop);
+        if (!lrp_addr_s) {
+            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
+            VLOG_WARN_RL(&rl, "lrp_addr not found for routing policy "
+                         " priority %"PRId64" nexthop %s",
+                         rule->priority, rule->nexthop);
+            return;
+        }
+        bool is_ipv4 = strchr(rule->nexthop, '.') ? true : false;
+        ds_put_format(&actions, "%sreg0 = %s; "
+                      "%sreg1 = %s; "
+                      "eth.src = %s; "
+                      "outport = %s; "
+                      "flags.loopback = 1; "
+                      "next;",
+                      is_ipv4 ? "" : "xx",
+                      rule->nexthop,
+                      is_ipv4 ? "" : "xx",
+                      lrp_addr_s,
+                      out_port->lrp_networks.ea_s,
+                      out_port->json_key);
+
+    } else if (!strcmp(rule->action, "drop")) {
+        ds_put_cstr(&actions, "drop;");
+    } else if (!strcmp(rule->action, "allow")) {
+        ds_put_cstr(&actions, "next;");
+    }
+    ds_put_format(&match, "%s", rule->match);
+    ovn_lflow_add(lflows, od, S_ROUTER_IN_POLICY, rule->priority,
+                  ds_cstr(&match), ds_cstr(&actions));
+    ds_destroy(&match);
+    ds_destroy(&actions);
+}
+
 static void
 add_distributed_nat_routes(struct hmap *lflows, const struct ovn_port *op)
 {
@@ -6827,9 +6902,35 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         }
     }
 
+    /* Logical router ingress table 8: Policy.
+     *
+     * A packet that arrives at this table is an IP packet that should be
+     * permitted/denied/rerouted to the address in the rule's nexthop.
+     * This table sets outport to the correct out_port,
+     * eth.src to the output port's MAC address,
+     * and '[xx]reg0' to the next-hop IP address (leaving
+     * 'ip[46].dst', the packet’s final destination, unchanged), and
+     * advances to the next table for ARP/ND resolution. */
+    HMAP_FOR_EACH (od, key_node, datapaths) {
+        if (!od->nbr) {
+            continue;
+        }
+        /* This is a catch-all rule. It has the lowest priority (0)
+         * does a match-all("1") and pass-through (next) */
+        ovn_lflow_add(lflows, od, S_ROUTER_IN_POLICY, 0, "1", "next;");
+
+        /* Convert routing policies to flows. */
+        for (int i = 0; i < od->nbr->n_policies; i++) {
+            const struct nbrec_logical_router_policy *rule
+                = od->nbr->policies[i];
+            build_routing_policy_flow(lflows, od, ports, rule);
+        }
+    }
+
+
     /* XXX destination unreachable */
 
-    /* Local router ingress table 8: ARP Resolution.
+    /* Local router ingress table 9: ARP Resolution.
      *
      * Any packet that reaches this table is an IP packet whose next-hop IP
      * address is in reg0. (ip4.dst is the final destination.) This table
@@ -7028,7 +7129,7 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
                       "get_nd(outport, xxreg0); next;");
     }
 
-    /* Logical router ingress table 9: Gateway redirect.
+    /* Logical router ingress table 10: Gateway redirect.
      *
      * For traffic with outport equal to the l3dgw_port
      * on a distributed router, this table redirects a subset
@@ -7071,7 +7172,7 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
         ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1", "next;");
     }
 
-    /* Local router ingress table 10: ARP request.
+    /* Local router ingress table 11: ARP request.
      *
      * In the common case where the Ethernet destination has been resolved,
      * this table outputs the packet (priority 0).  Otherwise, it composes

ovn/ovn-nb.ovsschema

@@ -1,7 +1,7 @@
 {
     "name": "OVN_Northbound",
-    "version": "5.15.1",
-    "cksum": "1272095726 22218",
+    "version": "5.16.0",
+    "cksum": "923459061 23095",
     "tables": {
         "NB_Global": {
             "columns": {
@@ -248,6 +248,12 @@
                                             "refType": "strong"},
                                    "min": 0,
                                    "max": "unlimited"}},
+                "policies": {
+                    "type": {"key": {"type": "uuid",
+                                     "refTable": "Logical_Router_Policy",
+                                     "refType": "strong"},
+                             "min": 0,
+                             "max": "unlimited"}},
                 "enabled": {"type": {"key": "boolean", "min": 0, "max": 1}},
                 "nat": {"type": {"key": {"type": "uuid",
                                          "refTable": "NAT",
@@ -315,6 +321,17 @@
                     "type": {"key": "string", "value": "string",
                              "min": 0, "max": "unlimited"}}},
             "isRoot": false},
+        "Logical_Router_Policy": {
+            "columns": {
+                "priority": {"type": {"key": {"type": "integer",
+                                              "minInteger": 0,
+                                              "maxInteger": 32767}}},
+                "match": {"type": "string"},
+                "action": {"type": {
+                    "key": {"type": "string",
+                            "enum": ["set", ["allow", "drop", "reroute"]]}}},
+                "nexthop": {"type": {"key": "string", "min": 0, "max": 1}}},
+            "isRoot": false},
         "NAT": {
             "columns": {
                 "external_ip": {"type": "string"},

ovn/ovn-nb.xml

@@ -1297,6 +1297,10 @@
       Zero or more static routes for the router.
     </column>
 
+    <column name="policies">
+      Zero or more routing policies for the router.
+    </column>
+
     <column name="enabled">
       This column is used to administratively set router state.  If this column
       is empty or is set to <code>true</code>, the router is enabled.  If this
@@ -1921,6 +1925,66 @@
 
   </table>
 
+  <table name="Logical_Router_Policy" title="Logical router policies">
+    <p>
+      Each row in this table represents one routing policy for a logical router
+      that points to it through its <ref column="policies"/> column.  The <ref
+      column="action"/> column for the highest-<ref column="priority"/>
+      matching row in this table determines a packet's treatment.  If no row
+      matches, packets are allowed by default. (Default-deny treatment is
+      possible: add a rule with <ref column="priority"/> 0, <code>1</code> as
+      <ref column="match"/>, and <code>drop</code> as <ref column="action"/>.)
+    </p>
+
+    <column name="priority">
+      <p>
+        The routing policy's priority.  Rules with numerically higher priority
+        take precedence over those with lower. A rule is uniquely identified
+        by the priority and match string.
+      </p>
+    </column>
+
+    <column name="match">
+      <p>
+        The packets that the routing policy should match,
+        in the same expression language used for the
+        <ref column="match" table="Logical_Flow" db="OVN_Southbound"/>
+        column in the OVN Southbound database's
+        <ref table="Logical_Flow" db="OVN_Southbound"/> table.
+      </p>
+
+      <p>
+        By default all traffic is allowed.  When writing a more
+        restrictive policy, it is important to remember to allow flows
+        such as ARP and IPv6 neighbor discovery packets.
+      </p>
+    </column>
+
+    <column name="action">
+      <p>The action to take when the routing policy matches:</p>
+      <ul>
+        <li>
+          <code>allow</code>: Forward the packet.
+        </li>
+
+        <li>
+          <code>drop</code>: Silently drop the packet.
+        </li>
+
+        <li>
+          <code>reroute</code>: Reroute packet to <ref column="nexthop"/>.
+        </li>
+      </ul>
+    </column>
+
+    <column name="nexthop">
+      <p>
+        Next-hop IP address for this route, which should be the IP
+        address of a connected router port or the IP address of a logical port.
+      </p>
+    </column>
+  </table>
+
   <table name="NAT" title="NAT rules">
     <p>
       Each record represents a NAT rule.