Reducing the need for public IP address

[amarpad]

Today our inbound roaming architecture requires a separate IP for each SGW at each AGW to be exposed to the PGW. In cases where this traffic goes through an IPX, e.g. Comfone(Access Parks IPX) requires that the roaming interface to be a public IP which is expensive and doesn't scale.
Starting a discussion thread to solve this.
Goal: introduce an aggregation device for the userplane that hides the distributed nature of the SGW so we can consolidate into one or a few public IPs.

Options considered:
1 - Just a simple NAT aggregation device: Doesn't work since the ports all need to be 2152 a NAT this does not work.
2 - GTP aware aggregation device: Write a small service that rewrites the te_id that sits in the forwarding path located at a suitable location (e.g. Equinix colo). This doesn't work as the te_id sent by the control plane (FeG) is not the same as the one that rewrites the te_id in the forwarding path.
3 - Per federation partner te_id: Have the orc8r/FeG allocate the te_id per roaming session, then the on-path device just can just maintains a map of te_id -> sgw_ip and rewrites the src_ip with that of the public ip address. The current te_id allocation is local to the AGW (thanks @uri200 for confirmation), which makes this hard.
What do folks think about option 3. cc @uri200 @ulaskozat @pshelar @talkhasib @sudhikan

[sudhikan]

just double clicking on [1], is the PGW expecting to hear from port 2152 also (src. port)? If so then I agree with the assessment that NAT won't work but if the PGW doesn't care on the src port, then NAT should work no?

for [3]: We would still need one public IP per SGW no?

[amarpad]

1 - Yes both source and dst: 2152 also (src. port) unfortunately
On 3, was thinking of having some tunnel into the device (ipsecVPN) in the colo from the SGW which can be private, but I am open to better suggestions.

[uri200]

Not sure if I follow but some context I about what we currently have

• On Control plane: 1 feg per roaming operator (feg will be sitting at each roaming operator network)
• On User plane: we need just need direct connectivity to each PGW-U. That is achieved for example with a VPN. VPN is out of our scope. So as long as there are VPN connections to each operator, we can use the same IP for all of those roaming agreements. Note we also recommend to to set the VPN on the same AGW but to have a dedicated vpn server/router

Maybe @pshelar can bring more insight on the user plane

[amarpad]

I think the question here is if we can do a global per FeG allocation of a TE_ID then the on path device to aggregate connections just needs to rewrite IPs

[pshelar]

yes, I think option 3 looks good. we need to change the TEID allocation for the AGWs.

[uri200]

I understand the issue partially. But not sure I understand why multiple IPs are needed (on data plane?). It would be good to create some kind of diagram so we can all talk about the same.

I created the diagram below, you can copy and modify picture here.

[sudhikan]

Let me see if I can summarize what I learned on the discussion yesterday:

• In this diagram we are assuming that the VPN connection goes straight from local to remote operators. Secondly, this diagram assumes that all AGW traffic is first concentrated in a local transport network before sending it off over a VPN connection.

• In practice this not the case:

  • Each AGW is expected to communicate outward independent of other AGWs and create it's own VPN connection.
  • There is an intermediary IPX that is concentrating the traffic and then sending it off to the remote operator.

• The requirement from the IPX is that each SGW (AGW) have a public IP address. Usually this is not an issue as traditional EPCs have one or two SGWs (active/standby), but in Magma we have many (because of decentralized edge).

• Getting public IPv4 addresses is not feasible due to cost and time considerations. One solution is to have a second VPN tunnel that goes from the access gateways to a new appliance box (tunnel concentrator).

  • This tunnel concentrator will have a public IP on its ingress and egress interfaces.
  • This tunnel concentrator will take all incoming GTP packets over the VPN session, replace the source IP from the AGW to that of its egress public IP and then ship the packet off in the second VPN tunnel toward the IPX and the remote PGW.
  • From the remote PGW's perspective, it only has one SGW connection and that is the VPN concentrator. In the future, it becomes considerably easier to scale the # of AGWs without having to modify the PGW and IPX configuration.
  • However, this also introduces a SPOF.

[AGW_A] <-VPN1--> [VPN concentrator] <-VPN2--> [IPX] <--VPN3--> [PGW]
[AGW_B] <-----------Î     Î       
[AGW_C] <-----------------I

• The issue comes when the return traffic is sent from the PGW back to this concentrator.
• The concentrator needs to know how to route the return packets to. Since the only unique identifier is the TEID, the TEID has to be unique across all AGWs so that the concentrator can know with certainty what the next hop is.
• Currently TEIDs are unique to an AGW so chances of collision do exist across gateways.
• One thought that was discussed yesterday was to create TEID pools based on the 32bit space and ensure each gateway gets its own pool.

I wrote this out just to keep things clear in my head :-) hope this is helpful.

[ulaskozat]

The current TEID allocation is random (if the subscriber per network is sufficiently low e.g., <20K), then maybe we do not need to over optimize this and accept the low collision probabilities. Otherwise, we need to unshelf some of the deterministic unique TEID assignment strategies that @uri200 was leading.

[talkhasib]

Agree, another alternative is an IMSI -> TEID hashing of sorts

[ulaskozat]

hashing does not change the collision prob here if it is not deterministically collision free. Also IMSI is at UE level, we need bearer level. I would rather have an orthogonal TEID pool assignment to each AGW instance and have a local allocation based on that.

[uri200]

@talkhasib We discussed this issue in some sessions during inbound roaming Create Bearer Request. The options were:

1. Creation of Orc8r service to provide Teids
2. Reuse of directoryd to provide Teids
3. Random assignment

Option 3 was the one we implemented to make this simpler and faster but with the idea someday we may have to do either 1 or 2