4G 1.6 prod | Gy usage reporting update not Initiated and zero octets reported in CCR-T

[arun-magma]

Issues Title: 4G 1.6 prod | Gy usage reporting update not Initiated and zero octets reported in CCR-T

Scenario:
1, UE Attach in federated mode and in Gx Receives the single Dynamic Rule.
2, Gx Dynamic rules consists of Service Identifier and Rating group.
3, on Gy MSCC only Rating Group value is cascaded from Gx missing Service identifier. #9370
4, OCS responds in Gy CCA with Granted service Units (2000 octets)
5, [issue] There is no Usage reporting Gy CCR-U going after continuous Traffic flow being consumed matching to the rule and also in Gy CCR-T there is no used service units (zero values)

Expectation: Gy-CCR-U should trigger used service units after granted usage is consumed, Also CCR-T should hold the cumulative session Usage report with Used Service Units.

Note:
This issue is observed in 1.6 production release, in later 1.7 release Usage Reporting on Gy is good with same set of rules.

Tested Build: 1.6.0-1625592603-1f26ba81
TeraVm test Scenarios:
gx_gy_combined_05_capacity

Enclosed - Pcaps, Syslog and sessiond logs

Gx-Gy 05 Capacity pcap-syslog-sessiond logs.zip

Analysis Snaps:

[Gy Grants]

[Gy no usu in CCR-T]

[Sessiond logs - Session Termination report shows Zero usage]

[uri200]

Hey @arun-magma , On sessiond logs I see this

Sep 30 20:49:09 ubuntuAGW sessiond[1039730]: I0930 20:49:09.639659 1039730 SessionState.cpp:481] 
Reported rule (PCC144-QCI9-DYNAMIC) not found in IMSI999991234567810-670933, ignoring

On Create Session Request I see the request comes with a different rule name PCC103-QCI9-DYNAMIC (see below)

So I am not sure why pipelined would report PCC144-QCI9-DYNAMIC when create session request installed PCC103-QCI9-DYNAMIC.

Note that in sessiond logs, you can see other IMSIS being reported. Maybe pieplined has some rules left over and it is reporting with that inexistent rule?

To try to bypass this, try to restart sctpd (that will restart all the systems and will clear the state). Then repeat the test

Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       credits {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         success: true
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         sid: "IMSI999991234567810"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         charging_key: 1
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         credit {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           validity_time: 86400
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           granted_units {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             total {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               is_valid: true
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               volume: 2000
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             tx {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               is_valid: true
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               volume: 1000
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             rx {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               is_valid: true
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               volume: 1000
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         result_code: 2001
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         tgpp_ctx {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           gy_dest_host: "tvm-vocs.magma.com"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       dynamic_rules {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         policy_rule {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           id: "PCC103-QCI9-DYNAMIC".   <------
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           priority: 1
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           rating_group: 1
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           flow_list {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             match {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               ip_src {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:                 address: "192.168.127.240"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           flow_list {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             match {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               direction: DOWNLINK
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               ip_dst {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:                 address: "192.168.127.240"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:               }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           qos {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             max_req_bw_ul: 100000000
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             max_req_bw_dl: 100000000
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:             qci: QCI_9
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           redirect {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:           }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       session_id: "IMSI999991234567810-670933"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       tgpp_ctx {
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         gx_dest_host: "tvm-vpcrf.magma.com"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:         gy_dest_host: "tvm-vocs.magma.com"
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       }
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:       event_triggers: UNSUPPORTED
Sep 30 20:49:08 ubuntuAGW sessiond[1039730]:   }

[arun-magma]

@uri200 : Thanks! for checking this, After SCTPD service restart we see this issue resolved and Gy usage reporting being Initiated by magma & test case passed, Updating the status in GitHub with logs, do we have plans for a permanent fix for this issue ?, As we know in production if this issues is unnoticed then it could possibly leak the revenue of the Service Provider, especially for those who rely on OCS based subscriber charging.

Enclosed Logs: pcap and sessiond logs

No Gy CCR-U Retest.zip

[ Usage Reporting is good now ]

[uri200]

Hey @arun-magma this is taken care by the expiration time on the MME.
Sessiond doesn't have the concept of time for a session. Either MME or PCRF are the only ones that can terminate sessions.

[stale]

This pull request has been automatically marked as stale because it has not had any recent activity by the author or maintainer in the past 45 days. It will be closed if no further activity occurs in the next 7 days. Thank you for your contributions.

[stale]

This pull request has exceeded its lifecylce and will be closed. Should you wish to continue working on this, please reopen it or start a new pull request. Thank you again for your contributions.