Update firewall rule host filters with type VPC

[plotnick]

Follow-up to #1636: whenever an instance or NIC is added or deleted in some VPC, all firewall rules that have that VPC as a host filter must be updated. This is particularly important since the default firewall rule allow-internal-inbound uses just such a host filter.

[rzezeski]

Reading this issue now puts #1821 into context a bit more, specifically the inbound rule that I found a bit confusing:

1        65534  inner.ip.src=172.30.0.5                          "Stateful: Firewall"

RFD 21 defines the "default allow internal inbound" as so:

Default allow internal inbound: This rule allows inbound traffic to all instances on the VPC as long as it originated from the VPC.

In this case we can use a firewall rule that matches the IP source based on the CIDR of the VPC subnet. For example it ends up looking like this (this output is from the oxide-vpc integration tests).

Layer firewall                                                                                                                                    
======================================================================                                                                            
Inbound Flows
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                 
                                                                         
Outbound Flows 
----------------------------------------------------------------------
PROTO  SRC IP           SPORT  DST IP           DPORT  HITS     ACTION                 
                                    
Inbound Rules [Default: deny]
----------------------------------------------------------------------
ID       PRI    PREDICATES                                       ACTION             
2        65534  inner.ip.proto=ICMP                              "Stateful Allow"
1        65534  inner.ip.proto=TCP                               "Stateful Allow"
                inner.ulp.dst=22                                 
                                    
0        65534  inner.ip.src=172.30.0.0/22                       "Stateful Allow"
                                    
Outbound Rules [Default: stateful allow]
----------------------------------------------------------------------
ID       PRI    PREDICATES                                       ACTION   

So I think the real solution is to make whatever change is necessary to have Nexus default to a VPC subnet filter for this rule. I think that's what "type": "vpc" should mean for a filter, right?

                {
                    "name": "allow-internal-inbound",
                    "status": "enabled",
                    "direction": "inbound",
                    "targets": [ { "type": "vpc", "value": "default" } ],
                    "filters": { "hosts": [ { "type": "vpc", "value": "default" } ] },
                    "action": "allow",
                    "priority": 65534,
                    "description": "allow inbound traffic to all instances within the VPC if originated within the VPC"
                },

[rmustacc]

So I think the real solution is to make whatever change is necessary to have Nexus default to a VPC subnet filter for this rule. I think that's what "type": "vpc" should mean for a filter, right?

Just to clarify if the target or filter is the VPC type that means the entire VPC and all subnets on it, not just a single one per Table 14 in https://rfd.shared.oxide.computer/rfd/0021#api-target-strings.

[plotnick]

The above discussion was extremely helpful for clarifying my understanding of what's going wrong in the current implementation; thank you both. Just to check: currently, I'm conflating the VPC with the IP addresses of all NICs allocated on it; what I should be using instead is the CIDR of all subnets defined on the VPC. This will still require us to recompute & resend rules on each subnet update, but that seems much better than having to do so on each instance/NIC update.

[rzezeski]

Robert is correct. I misunderstood the VPC target, and the correct interpretation is the one he linked. The current default in Nexus is correct. However, how we implement it can probably be more efficient.

By default, we want an inbound rule that allows traffic from any other guest in the same VPC. I can see 3 ways we could do this.

1. The current method: Add every guest's VIP (Virtual/VPC IP) to every other guest's firewall. So if there are 100 guests in a VPC, each guest has 99 inbound rules allowing traffic from those other VIPs. That's rather inefficient and could mean lots of churn in both the control plane and individual guest's firewall rules.

2. Add notion of "VPC" destination to the oxide-vpc firewall layer. We could add first class support for this type of firewall rule in oxide-vpc itself, but it would require a little work. On decap we would extract the VNI from Geneve and store it as action metadata to pass through the port's processing pipeline. The firewall layer could then add support for filtering on this value. This would be the most efficient in terms of control plane churn and oxide-vpc implementation. I kind of like this idea but not sure there are any edge cases I'm not thinking of.

3. Use the ability for oxide-vpc's firewall to filter on VPC subnet. For each subnet in the VPC, add an inbound rule to all guests in the VPC to allow traffic from a guest on that subnet. This should be much less churn that option (1), but a tad less efficient than option (2). This option should require no additional work in oxide-vpc or opte.

For both options (2) & (3) we need to update Nexus to apply the vpc target more efficiently.

[plotnick]

Right. I also like idea 2 (filter VPC by VNI) and had thought about it some, but think that 3 (filter on subnets) is the best/easiest option for now. We could open an issue for 2 and consider it later if you think it's worth it, but I should have a PR for 3 up today.

[rmustacc]

The original intent was that this was (2) and that we were filtering on the VNI.

[rzezeski]

Okay, I personally prefer (2) myself. So let me look into that today. If for some reason it's more work than I thought or there are other issues I'll write up why and we can revisit (3).