Association Between MME (Magma) and HSS (Abot) Failing - after integration of FeGW

[Rupsha-rc]

Hello,

I have been trying to have a capabilities exchange communication between the nodes MME (of Magma AGW) and HSS (of ABot). The FeGW has been integrated. My configuration is as below -

1. IP of abot box, where HSS and PCRF is hosted - 192.168.220.222
2. IP of the AGW box where the MME has been hosted - 192.168.220.221
3. IP of the box containing the FeGW - 192.168.102.120
4. Port for S6A for MME - 3868
5. Port for S6A for HSS - 3868

I have chosen the port 3868 with reference to a s6a packet I downloaded from the magma website.
I am attaching my configuration and the reference packet along with the log I am getting.
Any inputs on the above issue would be extremely helpful.

[uri200]

I see the pcap showing the FEG end. Can you make sure the requests gets to the hss? Can you take a pcap on the hss box? Is the HSS really running on the right port? Is the HSS configured to talk to get using the proper ip and port?

Per the log the HSS is not accepting the connection.

I suggest to define the ports too on NMS configuration

• 'addres' as '192.168.220.222:3868'
• 'local_adreess' as ':3868:

[Rupsha-rc]

Hi Oriol,
Thanks for the input, I tried setting the nms configuration with the ports, but it gave the same result. The SCTP connection is getting aborted as we see from the pcap.
Another thing I tried doing is testing the s6a interface for basic sanity as per suggestion from magma documentation. I ran the below command -
/var/opt/magma/bin/s6a_cli air

The output received is -

root@rebaca:~# /var/opt/magma/bin/s6a_cli air 001010000000001
Using IMSI: 1010000000001; MCC: 101; MNC: 000; PLMN ID: [1 1 0]
2021/09/20 10:44:57 Using default value: magma for flag: product
Using local S6a_proxy
Start sending requests 1
Sending AIR to 127.0.0.1:9098:
{
 "userName": "001010000000001",
 "visitedPlmn": "AQEA",
 "numRequestedEutranVectors": 3,
 "immediateResponsePreferred": true,
 "resyncInfo": null,
 "numRequestedUtranGeranVectors": 0,
 "utranGeranResyncInfo": null
}
protos.AuthenticationInformationRequest{UserName:"001010000000001", VisitedPlmn:[]uint8{0x1, 0x1, 0x0}, NumRequestedEutranVectors:0x3, ImmediateResponsePreferred:true, ResyncInfo:[]uint8(nil), NumRequestedUtranGeranVectors:0x0, UtranGeranResyncInfo:[]uint8(nil), XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}

 

E0920 10:45:57.019197      56 client.go:52] S6a Proxy client initialization error: service s6a_proxy connection error: address: 127.0.0.1:9098 gRPC Dial error: context deadline exceeded
2021/09/20 10:45:57 GRPC AIR Error: S6a Proxy client initialization error: service s6a_proxy connection error: address: 127.0.0.1:9098 gRPC Dial error: context deadline exceeded
2021/09/20 10:45:57 Errors found: 1 request failed out of 1

Any inputs would be extremely helpful. Thanks a lot!

[uri200]

per this log

E0920 10:45:57.019197      56 client.go:52] S6a Proxy client initialization error: service s6a_proxy connection error: address: 127.0.0.1:9098 gRPC Dial error: context deadline exceeded
2021/09/20 10:45:57 GRPC AIR Error: S6a Proxy client initialization error: service s6a_proxy connection error: address: 127.0.0.1:9098 gRPC Dial error: context deadline exceeded

It looks like your s6a_cli is not even able to get to the feg. Is the feg really running on that VM?

Note that client can run an s6_proxy itself (without the need of having feg installed) if you use option s6a_cli air -use_mconfig 001010000000001

Here you have an example of s6a_cli runnung from inside the s6a_proxy pod

root@magma-fedgw:/var/opt/magma/bin# ./s6a_cli air 001010000000001
Using IMSI: 1010000000001; MCC: 101; MNC: 000; PLMN ID: [1 1 0]
Using local S6a_proxy
Start sending requests 1
Sending AIR to 127.0.0.1:9098:
{
 "userName": "001010000000001",
 "visitedPlmn": "AQEA",
 "numRequestedEutranVectors": 3,
 "immediateResponsePreferred": true,
 "resyncInfo": null,
 "numRequestedUtranGeranVectors": 0,
 "utranGeranResyncInfo": null,
 "featureListId2": null
}
protos.AuthenticationInformationRequest{UserName:"001010000000001", VisitedPlmn:[]uint8{0x1, 0x1, 0x0}, NumRequestedEutranVectors:0x3, ImmediateResponsePreferred:true, ResyncInfo:[]uint8(nil), NumRequestedUtranGeranVectors:0x0, UtranGeranResyncInfo:[]uint8(nil), FeatureListId_2:(*protos.FeatureListId2)(nil), XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}

Received AIA:
{
 "errorCode": "UNDEFINED",
 "eutranVectors": [
  {
   "rand": "ASNFZ4mrze8BI0VniavN7w==",
   "xres": "Tcoahj0hBmE=",
   "autn": "bQOlOZg8gAAKkPfq3ymb4Q==",
   "kasme": "2XGt4aFgjgXDLWmNHhlWWN9a3f2MbxNLnsla4iKXG8M="
  },
  {
   "rand": "/ty6mHZUMhD+3LqYdlQyEA==",
   "xres": "d28dXRG2jfU=",
   "autn": "GvC4xJkYgADMEXJdUXrEEA==",
   "kasme": "gGMnDO/hOdqmorOCQi/RL5Cvkx+55pF75MaSMzzU5y0="
  },
  {
   "rand": "ASNFZ4mrze8BI0VniavN7w==",
   "xres": "Tcoahj0hBmE=",
   "autn": "bQOlOZh+gABswivjkIhQ9A==",
   "kasme": "5fd67LIV+l37sSdd+/Ywn6nAmYNoHt8EuZ6VE1RStMs="
  }
 ],
 "utranVectors": [
 ],
 "geranVectors": [
 ]
}
{ErrorCode:UNDEFINED EutranVectors:[rand:"\001#Eg\211\253\315\357\001#Eg\211\253\315\357" xres:"M\312\032\206=!\006a" autn:"m\003\2459\230<\200\000\n\220\367\352\337)\233\341" kasme:"\331q\255\341\241`\216\005\303-i\215\036\031VX\337Z\335\375\214o\023K\236\311Z\342\"\227\033\303"  rand:"\376\334\272\230vT2\020\376\334\272\230vT2\020" xres:"wo\035]\021\266\215\365" autn:"\032\360\270\304\231\030\200\000\314\021r]Qz\304\020" kasme:"\200c'\014\357\3419\332\246\242\263\202B/\321/\220\257\223\037\271\346\221{\344\306\2223<\324\347-"  rand:"\001#Eg\211\253\315\357\001#Eg\211\253\315\357" xres:"M\312\032\206=!\006a" autn:"m\003\2459\230~\200\000l\302+\343\220\210P\364" kasme:"\345\367z\354\262\025\372]\373\261']\373\3660\237\251\300\231\203h\036\337\004\271\236\225\023TR\264\313" ] UtranVectors:[] GeranVectors:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
2021/09/24 16:49:05
All request (1) got a response

Btw, let me add the comments we mentioned on the email below

Can you please take a look at our pcaps collection on our documentation here

https://docs.magmacore.org/docs/resources/ref_pcap#pcap-collection

Also at the end of inbound roaming there is a sample configuration. I will put that on the main FEG page. But on the meantime take at the link below. Go to the bottom and in the zip file check for FEG gateway file and FEG netwolr

https://docs.magmacore.org/docs/next/howtos/inbound_roaming#configuration-example

For questions like this, let's use GitHub for higher visibility so we all can go contribute and create issues out of the questions.

https://github.com/magma/magma/discussions

[ShubhamTatvamasi]

Hi @uri200 , Our s6a_proxy container is restarting after connecting to Orc8r. Could you help us debug this?

CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS                        PORTS     NAMES
acd691a64f5e   magmacore/gateway_python:1.6.0   "/bin/bash -c '/usr/…"   50 minutes ago   Up 50 minutes                           redis
d7725eec0626   magmacore/gateway_python:1.6.0   "/bin/bash -c '/usr/…"   50 minutes ago   Up 50 minutes                           control_proxy
43311bcfd599   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           swx_proxy
cccb02b67e24   magmacore/gateway_python:1.6.0   "python3.8 -m magma.…"   51 minutes ago   Up 51 minutes                           eventd
1e9f8e5c076f   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           feg_hello
19e6a8adaeca   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           eap_sim
1e068b116ed2   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           session_proxy
3f58c6c596c4   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           eap_aka
de54d72a28ca   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           health
962a5b852acb   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           s8_proxy
154d4de04fbd   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           csfb
8a8c39f95dd2   magmacore/gateway_python:1.6.0   "python3.8 -m magma.…"   51 minutes ago   Up 51 minutes                           magmad
a4e406282cbf   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           aaa_server
ccace50a17b9   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Restarting (1) 1 second ago             s6a_proxy
756fc7f74b06   magmacore/gateway_go:1.6.0       "envdir /var/opt/mag…"   51 minutes ago   Up 51 minutes                           radiusd
0d1480ed9f78   magmacore/gateway_python:1.6.0   "/bin/bash -c '/usr/…"   51 minutes ago   Up 51 minutes (healthy)                 td-agent-bit

s6a_proxy.logs.zip

[ShubhamTatvamasi]

as in port 80?

[uri200]

Hey @ShubhamTatvamasi you need to configure ports you want your FEG to use. Those ports will need to be configured on the HSS so the HSS can know what port to use to respond the reequest.

If you go to a previous meesage in this thread (here), you will see an example of configuration we have in our labs here. You can use the same ports we use there.

In our case we use :3868

        "s6a_proxy": {
            "@type": "type.googleapis.com/magma.mconfig.S6aConfig",
            "logLevel": "INFO",
            "minimumRequestThreshold": 1,
            "requestFailureThreshold": 0.5,
            "server": {
                "address": "172.16.1.9:3868",
                "destHost": "magma-fedgw.magma.com",
                "destRealm": "magma.com",
                "host": "string",
                "localAddress": ":3868",

[Rupsha-rc]

Hi @uri200

I am observing from our tcp dumps that the SCTP connection is not being set up between the s6a_proxy and 3rd party HSS node (ABot). Kindly please guide me how do I setup the SCTP connection between these two.

[samit-ghosh]

@uri200 Just wanted to add a diagram so that we understand what we're trying to do

When we try to go according to the flow described above, what we're seeing is that we are unable to establish sctp association with the FeGW. Is it possible that the FeGW waits for a sctp INIT. Can it be configured to behave like a sctp server?

[uri200]

Hey @rupa455 I am posting again an example of configuration for magma
https://docs.magmacore.org/docs/howtos/inbound_roaming#configuration-example
You can check feg files and configure accordingly. That configuration is extracted from swagger API, but you can use NMS to configure s6a

@samit-ghosh
I included a PCAP showing how FEG starts SCTP with HSS. You can check frame 8.

gx_gy_combined_09c_211007_204304-1.pcapng.zip

Also, in the link below you can see collections of pcap showing some tests. It shows many of the interfaces (S1ap, S6a, Gx, Gy). There you can see the
https://docs.magmacore.org/docs/next/resources/ref_pcap#pcap-collection

[uri200]

Can you please provide a pcap showing s6a interface. And can you also enable log level debugging on s6a (see below how v=9)

  s6a_proxy:
    <<: *goservice
    container_name: s6a_proxy
    environment:
      MAGMA_PRINT_GRPC_PAYLOAD: 0
    command: envdir /var/opt/magma/envdir /var/opt/magma/bin/s6a_proxy -logtostderr=true -v=9

You can also enable MAGMA_PRINT_GRPC_PAYLOAD: 1 to see the grpc messages received by s6a_proxy

Can you try to execute this command below? That will trigger an AIR through your local s6a_proxy.

sudo docker-compose exec s6a_proxy /var/opt/magma/bin/s6a_cli air 001002000000810

If you have MAGMA_PRINT_GRPC_PAYLOAD: 1 you should see a request coming to your proxy. Below you can see the GRPC request as *protos.AuthenticationInformationRequest: { and you can see the AIR (and AVPs) that this requests triggers at s6a_proxy | Authentication-Information-Request (AIR)

s6a_proxy        | I1011 23:39:24.125249       1 logcodec.go:46] Received: *protos.AuthenticationInformationRequest: {
s6a_proxy        |      "user_name": "001020000000064",
s6a_proxy        |      "visited_plmn": "APEQ",
s6a_proxy        |      "num_requested_eutran_vectors": 1,
s6a_proxy        |      "immediate_response_preferred": false,
s6a_proxy        |      "resync_info": null,
s6a_proxy        |      "num_requested_utran_geran_vectors": 0,
s6a_proxy        |      "utran_geran_resync_info": null,
s6a_proxy        |      "feature_list_id_2": null
s6a_proxy        | }
s6a_proxy        | I1011 23:39:24.125295       1 ai.go:68] Sending S6a AIR message
s6a_proxy        | Authentication-Information-Request (AIR)
s6a_proxy        | {Code:318,Flags:0xc0,Version:0x1,Length:208,ApplicationId:16777251,HopByHopId:0x2d8d0645,EndToEndId:0x88dc5957}
s6a_proxy        |      Session-Id {Code:263,Flags:0x40,Length:60,VendorId:0,Value:UTF8String{string-s6a;18081196142640516697;3571543878;d14217bd},Padding:1}
s6a_proxy        |      Origin-Host {Code:264,Flags:0x40,Length:16,VendorId:0,Value:DiameterIdentity{string},Padding:2}
s6a_proxy        |      Origin-Realm {Code:296,Flags:0x40,Length:16,VendorId:0,Value:DiameterIdentity{string},Padding:2}
s6a_proxy        |      User-Name {Code:1,Flags:0x40,Length:24,VendorId:0,Value:UTF8String{001020000000064},Padding:1}
s6a_proxy        |      Auth-Session-State {Code:277,Flags:0x40,Length:12,VendorId:0,Value:Enumerated{1}}
s6a_proxy        |      Visited-PLMN-Id {Code:1407,Flags:0xc0,Length:16,VendorId:10415,Value:OctetString{0x00f110},Padding:1}
s6a_proxy        |      Requested-EUTRAN-Authentication-Info {Code:1408,Flags:0xc0,Length:44,VendorId:10415,Value:Grouped{
s6a_proxy        |              Number-Of-Requested-Vectors {Code:1410,Flags:0xc0,Length:16,VendorId:10415,Value:Unsigned32{1}},
s6a_proxy        |              Immediate-Response-Preferred {Code:1412,Flags:0xc0,Length:16,VendorId:10415,Value:Unsigned32{0}},
s6a_proxy        |      }}
s6a_proxy        |
s6a_proxy        | I1011 23:39:24.125758       1 ai.go:99] Received S6a AIA message:
s6a_proxy        | Authentication-Information-Answer (AIA)
s6a_proxy        | {Code:318,Flags:0x40,Version:0x1,Length:344,ApplicationId:16777251,HopByHopId:0x2d8d0645,EndToEndId:0x88dc5957}
s6a_proxy        |      Session-Id {Code:263,Flags:0x40,Length:60,VendorId:0,Value:UTF8String{string-s6a;18081196142640516697;3571543878;d14217bd},Padding:1}
s6a_proxy        |      Vendor-Specific-Application-Id {Code:260,Flags:0x40,Length:32,VendorId:0,Value:Grouped{
s6a_proxy        |              Vendor-Id {Code:266,Flags:0x40,Length:12,VendorId:0,Value:Unsigned32{10415}},
s6a_proxy        |              Auth-Application-Id {Code:258,Flags:0x40,Length:12,VendorId:0,Value:Unsigned32{16777251}},
s6a_proxy        |      }}
s6a_proxy        |      Result-Code {Code:268,Flags:0x40,Length:12,VendorId:0,Value:Unsigned32{2001}}
s6a_proxy        |      Auth-Session-State {Code:277,Flags:0x40,Length:12,VendorId:0,Value:Enumerated{1}}
s6a_proxy        |      Origin-Host {Code:264,Flags:0x40,Length:28,VendorId:0,Value:DiameterIdentity{tvm-vhss.magma.com},Padding:2}
s6a_proxy        |      Origin-Realm {Code:296,Flags:0x40,Length:20,VendorId:0,Value:DiameterIdentity{magma.com},Padding:3}
s6a_proxy        |      Authentication-Info {Code:1413,Flags:0xc0,Length:160,VendorId:10415,Value:Grouped{
s6a_proxy        |              E-UTRAN-Vector {Code:1414,Flags:0xc0,Length:148,VendorId:10415,Value:Grouped{
s6a_proxy        |                      Item-Number {Code:1419,Flags:0xc0,Length:16,VendorId:10415,Value:Unsigned32{0}},
s6a_proxy        |                      RAND {Code:1447,Flags:0xc0,Length:28,VendorId:10415,Value:OctetString{0x0123456789abcdef0123456789abcdef},Padding:0},
s6a_proxy        |                      XRES {Code:1448,Flags:0xc0,Length:20,VendorId:10415,Value:OctetString{0x295c00eae388930d},Padding:0},
s6a_proxy        |                      AUTN {Code:1449,Flags:0xc0,Length:28,VendorId:10415,Value:OctetString{0x54ab644af6418000fffb6d2b3f3418bc},Padding:0},
s6a_proxy        |                      KASME {Code:1450,Flags:0xc0,Length:44,VendorId:10415,Value:OctetString{0xce34caed82510aaea0145525412e708b963a0e52c8d6d1c3f735806df39fb8b8},Padding:0},
s6a_proxy        |              }}
s6a_proxy        |      }}
s6a_proxy        |

[kaderwavelabs]

@Rupsha-rc @samit-ghosh can you check the input and revert back?

[uri200]

Note that diameter connection will not be initiated until the first message is being sent. So starting feg will not trigger a diameter message. Please, check the PCAPs, some of them include the initial SCTP connection and Capabilities setup message sent at the very beginning of a diameter connection. Those messages are not sent until the very first AIR message is received by the FEG.

[samit-ghosh]

@uri200 @kaderwavelabs @Rupsha-rc
Thanks Oriol for the response. What you are saying is that there is no mechanism to setup the sctp connection only between the s6a_proxy and the HSS node befoer the AIR. This means that there is no way to control the transport layer setup without the first application layer packet between them.

[uri200]

Hey @samit-ghosh, yes the SCTPD will not be established until the first attempt is made. Why is this a problem?
Note that if this is just for testing you can easily trigger a manual authentication with s6a_cli as mentioned here #9159 (reply in thread)

[samit-ghosh]

@uri200 @kaderwavelabs @Rupsha-rc
Thanks for all the answers @uri200. Actually, there is way in which we setup the sctp transport before we run the call flow. However, we're going to adjust this and make the test tool create the sctp connection at the time of first application layer message, i.e. AIR in this case.

[uri200]

Hey @samit-ghosh after reviewing the code my answer may not be completely right. I may have get confused because our end to end test does not start HSS until we run the first test, so we do not see any connection until then.

But I was testing Sigscale HSS feature and I could see that when magma starts the Diameter client and sends its first message (Capability Exchange) setting up the connection. You can see it here in our code

If the first attempt fails because HSS is not available, then it tries to set the connection up again with the first message received.

So I guess your connection would be set up if you start HSS first and then magma.

Alternatively, you can manually force an AIR attach to a random UE from the FEG using this command

magma@magma-fedgw:/var/opt/magma/docker$ sudo docker-compose exec s6a_proxy /var/opt/magma/bin/s6a_cli air  001002000000810

You could run that before running your test script to make sure the sctp connection is up
(to see if that command works you can print grpc message to see what you are sending to the service and enable logging -v=9 to see AVPs being sent

  s6a_proxy:
    <<: *goservice
    container_name: s6a_proxy
    environment:
      MAGMA_PRINT_GRPC_PAYLOAD: 1
    command: envdir /var/opt/magma/envdir /var/opt/magma/bin/s6a_proxy -logtostderr=true -v=9

to show the log just go to your docker folder and do

magma@magma-fedgw:/var/opt/magma/docker$ 
docker-compose logs -f s6a_proxy

[Rupsha-rc]

@samit-ghosh @kaderwavelabs @parthiban337 @jayanta-sarkar

Hi @uri200

We had made certain changes so as to create the sctp connection at the time of first application layer message, i.e. AIR in this case and as well automated the capabilities exchange procedure. Our test tool now instantly replies to any capex request received from DUT side.

I am sending our packet capture from our test tool over email.
On receiving the Initial UE Request from the ABot test tool, the Magma DUT initiates both Capex Request and AIR message to which the Test tool responds with Capex Answer and AIA. After that the Magma DUT sends a sctp shutdown.

We tried comparing our packet capture with the reference packet captures available in the magma documentation, but did not find any sctp INIT sequence or Capex exchange. We were wondering if the packet dump had been captured after the INIT sequence and Capex exchange had already been exchanged. If so, then can we get a pcap that shows that the sctp association is setup followed by a capex, so as to understand if the Capex answer we are sending is appropriate or not.

[uri200]

Hey @Rupsha-rc this is a pcap which includes the sctp sync and CE.

As mentioned in my previous message [here](https://github.com/magma/magma/discussions/9159#discussioncomment-1586055) , the s6a_proxy log with debugging enabled will provide more information than the pcap if this is a feg issue. If you can restart docker services, run the test and collect the log here that will help tool