GTP Use-Case #97

infinitydon · 2022-09-14T21:33:32Z

infinitydon
Sep 14, 2022

Hi

Please how is the GTP configuration done? I am interested particularly how your GTP functionality is tied to how the 5G SMF will be configured, does the UPF need to be aware of the loxilb in terms of compatibility?

Answered by infinitydon

Sep 20, 2022

Questions have been answered, proceeding to close this for now.

View full answer

nik-netlox · 2022-09-16T03:34:55Z

nik-netlox
Sep 16, 2022
Maintainer

Hi,
Please refer to this page to know more about GTP configuration.
Currently, GTP functionality is not tied to 5G SMF or UPF. Though, configuration can be done from outside through provided APIs.

5 replies

infinitydon Sep 17, 2022
Author

Thanks for the response @nik-netlox but I am still not clear about what you are trying to achieve within the concept of GTP traffic flow using the 3gpp. Can you explain how loxilb can be used exactly with respect to a UPF (is loxilb doing something similar to an I-UPF) or PGW-U?

From the documentation you shared it seems loxilb is actually de-encapsulating GTP tunnel which should be the function of a UPF/PGW-U which is a bit confusing to me.

Thanks for your time.

nik-netlox Sep 19, 2022
Maintainer

Hi @infinitydon,

Here are the two scenarios which LoxiLB can be useful:

Local Break Out MEC traffic with Load Balancing in MEC Cluster( Bundling UPF's Traffic Offload Policy together with K8s external service Load Balancing.)
LoxiLB does not intend to replace UPF, instead LoxiLB will be a good fit when you want to break-out the traffic and send it to your MEC cluster. It exposes REST APIs for UL-CL and LB configuration. When the traffic comes to LoxiLB, first, as per UL-CL filters, GTP will be decapped and then it will be load balanced and send towards K8s cluster. By default, other traffic will go to CN. So in this case, LoxiLB will sit just after gNB and before I-UPF/UPF.
Load Balancing 5G CP Services like AMF, SMF. LoxiLB can also serve as service type external load balancer for 5G CP services running on top of TCP/UDP/SCTP(e.g N2, N11, N4 interfaces) depending on the deployment scenario.

Let us know if you have other queries, we will be happy to help!

infinitydon Sep 19, 2022
Author

Thanks It is clear now, as per the first scenario (Local Break Out MEC) it means that all the traffic bound to the UPF must pass through LoxiLB so that it can decide if branching is required or not. One thing that might be beneficial is that is it possible to use a subnet CIDR instead of individual mobile user IP for the ul-cl filter like below:

#Creating session for ue1
$dexec llb2 loxicmd create session user1 32.32.32.0/24 --accessNetworkTunnel 1:10.10.10.59 --coreNetworkTunnel=1:10.10.10.56

#Creating ULCL filter with service IP for ue1
$dexec llb2 loxicmd create sessionulcl user1 --ulclArgs=11:88.88.88.88

Reason for asking is that there may be a requirement to group a set of mobile users to branch out locally, so instead of creating individual ul-cl filters, we can just create one that will capture users that belong to a specific CIDR.

nik-netlox Sep 20, 2022
Maintainer

Thanks for proposing this. Yes, It is possible to do what you suggested,
e.g. "$dexec llb2 loxicmd create sessionulcl --sip 32.32.32.0/24 --dip 88.88.88.0/24"

Currently, our implementation is somewhat focusing on UL-CL per user. Since, it will be more convenient, we will plan to simplify this soon.

infinitydon Sep 20, 2022
Author

Thanks for your response and patience in answering, I appreciate it, will be on the lookout for the CIDR support for the ul-cl filter

infinitydon · 2022-09-20T11:22:43Z

infinitydon
Sep 20, 2022
Author

Questions have been answered, proceeding to close this for now.

0 replies

infinitydon · 2022-10-06T20:10:23Z

infinitydon
Oct 6, 2022
Author

@nik-netlox - I just tested the ulcl scenario but the GTP packet is still going directly to the UPF, below is the commands I used:

./loxicmd create session user1 10.45.0.2 --accessNetworkTunnel=1:192.168.1.125 --coreNetworkTunnel=1:192.168.70.2

./loxicmd create sessionulcl user1 --ulclArgs=1:88.88.88.88

./loxicmd create lb 88.88.88.88 --tcp=2020:80 --endpoints=172.17.0.2:1

10.45.0.2 - This is UE IP address that was allocated by the SMF

192.168.1.125 - This is the gNB (am assuming that the 1: signifies the TEID?)

192.168.70.2 - This is the UPF address (am also assuming that the 1: signifies the TEID?)

172.17.0.2 - This is the IP of nginx docker container, it is running on the same VM with loxilb.

when I initiate a curl to 88.88.88.88:2020, the whole GTP packet is been routed directly to the UPF.

root@core5g-upf-deployment-549f9c55c7-vsfkm:/# tcpdump -i any host 10.45.0.2 -s0 -nv
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:47:28.807635 ogstun In  IP (tos 0x0, ttl 64, id 4042, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.36822 > 88.88.88.88.2020: Flags [S], cksum 0x0123 (correct), seq 1684491581, win 65280, options [mss 1360,sackOK,TS val 554640115 ecr 0,nop,wscale 7], length 0
18:48:02.855673 ogstun In  IP (tos 0x0, ttl 64, id 4043, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.36822 > 88.88.88.88.2020: Flags [S], cksum 0x7c22 (correct), seq 1684491581, win 65280, options [mss 1360,sackOK,TS val 554674163 ecr 0,nop,wscale 7], length 0
18:49:00.268594 ogstun In  IP (tos 0x0, ttl 64, id 37730, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x6ab8 (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554731577 ecr 0,nop,wscale 7], length 0
18:49:01.286827 ogstun In  IP (tos 0x0, ttl 64, id 37731, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x66be (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554732595 ecr 0,nop,wscale 7], length 0
18:49:03.303906 ogstun In  IP (tos 0x0, ttl 64, id 37732, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x5ede (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554734611 ecr 0,nop,wscale 7], length 0
18:49:07.367011 ogstun In  IP (tos 0x0, ttl 64, id 37733, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x4efe (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554738675 ecr 0,nop,wscale 7], length 0
18:49:15.559083 ogstun In  IP (tos 0x0, ttl 64, id 37734, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x2efe (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554746867 ecr 0,nop,wscale 7], length 0
18:49:31.686701 ogstun In  IP (tos 0x0, ttl 64, id 37735, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0xeffe (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554762994 ecr 0,nop,wscale 7], length 0
18:50:05.736861 ogstun In  IP (tos 0x0, ttl 64, id 37736, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.41540 > 88.88.88.88.80: Flags [S], cksum 0x6afd (correct), seq 2188858742, win 65280, options [mss 1360,sackOK,TS val 554797043 ecr 0,nop,wscale 7], length 0
18:53:36.381953 ogstun In  IP (tos 0x0, ttl 64, id 29799, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.34228 > 172.217.1.206.80: Flags [S], cksum 0x1fa7 (correct), seq 4006597962, win 65280, options [mss 1360,sackOK,TS val 471468717 ecr 0,nop,wscale 7], length 0
18:53:36.403332 ogstun Out IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto TCP (6), length 60)

Also note that I am using loxilb as the gateway for the gNB so all the traffic is passing through loxilb.

Interface config of loxilb VM:

lo               UNKNOWN        127.0.0.1/8 ::1/128
enp0s3           UP             192.168.1.124/24 2600:4040:2929:ef00:a00:27ff:fee7:8cfe/64 fe80::a00:27ff:fee7:8cfe/64
enp0s8           UP             192.168.56.101/24 fe80::a00:27ff:fec9:9199/64
enp0s9           UP             192.168.70.1/24 fe80::a00:27ff:fe85:d9bb/64
llb0             UP             fe80::806:9bff:fed2:ff1f/64
docker0          UP             172.17.0.1/16 fe80::42:dcff:fe15:2bad/64
vethfb5e074@if7  UP             fe80::447c:b7ff:fea9:e9fe/64

Attached is the console logs when I start loxilb, maybe that can throw more light.

loxilb-ulcl-oct-06.txt

Another question, how can I redirect the logs of loxilb to a log file instead of stdout? I am not using docker.

15 replies

TrekkieCoder Oct 9, 2022
Maintainer

Hi @infinitydon,
We got some further pointers from here. It seems that qfi is indeed used by ueransim in gtp. So, to narrow down the problem, if possible, can you please give us the following additional information -

Capture and send us a pcap file on loxilb VM's enp0s3 interface which is connected to the gNB after initiating curl ?
Information of "ip l show" in loxilb VM

infinitydon Oct 9, 2022
Author

@TrekkieCoder - Please find attached for the pcap. Looks like QFI is already set to 1.

ueransim-loxilb.zip

Interface printout:

chris@ebenezer-loxilb:~/loxicmd$ ip l show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:e7:8c:fe brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:c9:91:99 brd ff:ff:ff:ff:ff:ff
4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:85:d9:bb brd ff:ff:ff:ff:ff:ff
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:42:c0:fe:4f:d0 brd ff:ff:ff:ff:ff:ff
20: llb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether aa:e5:66:03:90:4d brd ff:ff:ff:ff:ff:ff
    prog/xdp id 300

TrekkieCoder Oct 10, 2022
Maintainer

Thanks @infinitydon for providing the information. We were able to pinpoint the issue

tunnel-id of the packets were set to 2
layer2 mac-address of gtp packets did not match interface enp0s3's mac-address due to which further l3 decap did not happen in loxilb
- We checked in a few changes to bypass this in case a user-session is matched. So kindly do a git pull to sync with latest code changes.
- Still, it is little confusing why ueransim is resolving mac-address to some other mac-address
Yes, qfi was set to 1

Finally, this following config worked in our setup (we did tcpreplay of your pcap to verify)

loxicmd create session user1 10.45.0.2 --accessNetworkTunnel=2:192.168.1.125 --coreNetworkTunnel=2:192.168.70.2
loxicmd create sessionulcl user1 --ulclArgs=1:88.88.88.88
loxicmd create lb 88.88.88.88 --tcp=2020:80 --endpoints=172.17.0.2:1

PS - We need to double check what tunnel-id is expected by ueransim for downlink traffic i.e. whether it is the same as uplink tunnel-id(2). As per the pcap dump, dl tunnel-id seems to be 1 (not 2). In such a case, please change "accessNetworkTunnel" tunnel-id config to "1:192.168.1.125 " accordingly.

infinitydon Oct 10, 2022
Author

Thanks @TrekkieCoder - Works now I am able to see the traffic in the Nginx Pod without it going to through the UPF.

root@nginx-deployment-5d4fc6576c-dmdpw:/# tcpdump -i any host 10.45.0.2 -s 0 -nv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes


20:50:44.002227 IP (tos 0x0, ttl 63, id 13299, offset 0, flags [DF], proto TCP (6), length 60)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [S], cksum 0x1aaa (correct), seq 3730065449, win 65280, options [mss 1360,sackOK,TS val 601366506 ecr 0,nop,wscale 7], length 0
20:50:44.003761 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.70.5.80 > 10.45.0.2.45900: Flags [S.], cksum 0x110b (incorrect -> 0xd355), seq 901514344, ack 3730065450, win 65160, options [mss 1460,sackOK,TS val 2869781029 ecr 601366506,nop,wscale 7], length 0
20:50:44.011816 IP (tos 0x0, ttl 63, id 13300, offset 0, flags [DF], proto TCP (6), length 52)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [.], cksum 0xfea2 (correct), ack 1, win 510, options [nop,nop,TS val 601366516 ecr 2869781029], length 0
20:50:44.011831 IP (tos 0x0, ttl 63, id 13301, offset 0, flags [DF], proto TCP (6), length 132)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [P.], cksum 0x4a95 (correct), seq 1:81, ack 1, win 510, options [nop,nop,TS val 601366518 ecr 2869781029], length 80: HTTP, length: 80
        GET / HTTP/1.1
        Host: 88.88.88.88:2020
        User-Agent: curl/7.81.0
        Accept: */*

20:50:44.011948 IP (tos 0x0, ttl 64, id 54607, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.70.5.80 > 10.45.0.2.45900: Flags [.], cksum 0x1103 (incorrect -> 0xfe49), ack 81, win 509, options [nop,nop,TS val 2869781037 ecr 601366518], length 0
20:50:44.034886 IP (tos 0x0, ttl 64, id 54608, offset 0, flags [DF], proto TCP (6), length 290)
    192.168.70.5.80 > 10.45.0.2.45900: Flags [P.], cksum 0x11f1 (incorrect -> 0x6fc5), seq 1:239, ack 81, win 509, options [nop,nop,TS val 2869781060 ecr 601366518], length 238: HTTP, length: 238
        HTTP/1.1 200 OK
        Server: nginx/1.14.2
        Date: Mon, 10 Oct 2022 20:50:44 GMT
        Content-Type: text/html
        Content-Length: 612
        Last-Modified: Tue, 04 Dec 2018 14:44:49 GMT
        Connection: keep-alive
        ETag: "5c0692e1-264"
        Accept-Ranges: bytes

20:50:44.049242 IP (tos 0x0, ttl 63, id 13302, offset 0, flags [DF], proto TCP (6), length 52)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [.], cksum 0xfd25 (correct), ack 239, win 509, options [nop,nop,TS val 601366549 ecr 2869781060], length 0
20:50:44.049501 IP (tos 0x0, ttl 64, id 54609, offset 0, flags [DF], proto TCP (6), length 664)
    192.168.70.5.80 > 10.45.0.2.45900: Flags [P.], cksum 0x1367 (incorrect -> 0x3edf), seq 239:851, ack 81, win 509, options [nop,nop,TS val 2869781075 ecr 601366549], length 612: HTTP
20:50:44.056009 IP (tos 0x0, ttl 63, id 13303, offset 0, flags [DF], proto TCP (6), length 52)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [.], cksum 0xfaa8 (correct), ack 851, win 505, options [nop,nop,TS val 601366563 ecr 2869781075], length 0
20:50:44.059248 IP (tos 0x0, ttl 63, id 13304, offset 0, flags [DF], proto TCP (6), length 52)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [F.], cksum 0xfaa2 (correct), seq 81, ack 851, win 505, options [nop,nop,TS val 601366568 ecr 2869781075], length 0
20:50:44.059753 IP (tos 0x0, ttl 64, id 54610, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.70.5.80 > 10.45.0.2.45900: Flags [F.], cksum 0x1103 (incorrect -> 0xfa93), seq 851, ack 82, win 509, options [nop,nop,TS val 2869781085 ecr 601366568], length 0
20:50:44.067965 IP (tos 0x0, ttl 63, id 13305, offset 0, flags [DF], proto TCP (6), length 52)
    10.45.0.2.45900 > 192.168.70.5.80: Flags [.], cksum 0xfa92 (correct), ack 852, win 505, options [nop,nop,TS val 601366573 ecr 2869781085], length 0

Below is the config that worked:

./loxicmd create session user1 10.45.0.2 --accessNetworkTunnel=1:192.168.1.125 --coreNetworkTunnel=2:192.168.70.2

./loxicmd create sessionulcl user1 --ulclArgs=1:88.88.88.88

./loxicmd create lb 88.88.88.88 --tcp=2020:80 --endpoints=192.168.70.5:1

TEID 1 and 2 is used, 1 is used when GTP is been sent from the gNB to the UPF and vice-versa for 2.

So I think for this to work in an automated fashion there is need to add some functionality to the SMF since that is where the sessions are been created and TEIDs will be dynamic.

Thanks for your support, I will try to write a blog post for this soon.

TrekkieCoder Oct 12, 2022
Maintainer

Great. Thanks a lot !

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GTP Use-Case #97

{{title}}

Replies: 3 comments 20 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

GTP Use-Case #97

infinitydon Sep 14, 2022

Replies: 3 comments · 20 replies

nik-netlox Sep 16, 2022 Maintainer

infinitydon Sep 17, 2022 Author

nik-netlox Sep 19, 2022 Maintainer

infinitydon Sep 19, 2022 Author

nik-netlox Sep 20, 2022 Maintainer

infinitydon Sep 20, 2022 Author

infinitydon Sep 20, 2022 Author

infinitydon Oct 6, 2022 Author

TrekkieCoder Oct 9, 2022 Maintainer

infinitydon Oct 9, 2022 Author

TrekkieCoder Oct 10, 2022 Maintainer

infinitydon Oct 10, 2022 Author

TrekkieCoder Oct 12, 2022 Maintainer

infinitydon
Sep 14, 2022

Replies: 3 comments 20 replies

nik-netlox
Sep 16, 2022
Maintainer

infinitydon Sep 17, 2022
Author

nik-netlox Sep 19, 2022
Maintainer

infinitydon Sep 19, 2022
Author

nik-netlox Sep 20, 2022
Maintainer

infinitydon Sep 20, 2022
Author

infinitydon
Sep 20, 2022
Author

infinitydon
Oct 6, 2022
Author

TrekkieCoder Oct 9, 2022
Maintainer

infinitydon Oct 9, 2022
Author

TrekkieCoder Oct 10, 2022
Maintainer

infinitydon Oct 10, 2022
Author

TrekkieCoder Oct 12, 2022
Maintainer