Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(mme): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure #8417

Merged

Conversation

rsarwad
Copy link
Contributor

@rsarwad rsarwad commented Aug 2, 2021

fix(agw): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure

Summary

Inconsistently memory leak was observed during execution of test_attach_detach_dedicated_deactivation_timer_expiry.py
Cause for leak is, on first timer timer expiry, mme shall start the timer with some arguments, for which memory would be allocated. So part of start, if timer has valid timer id then first timer is stopped and started again. As part of stop, it returns the memory allocated for timer arguments.
Since timer has expired, timer id will not be found in the timer queue, so timer argument returned from stop was corrupted/incorrect. When start the timer again; it was not starting. Since it was not started, it was resulting into memory leak.
Memory allocated to timer argument would be freed, either on final timer expiry or while stopping using api, esm_ebr_stop_timer()

Below is the back trace where memory leak was observed,

Jul 27 13:25:32 magma-dev-focal mme[384880]: Direct leak of 32 byte(s) in 1 object(s) allocated from:
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #0 0x7ff2bad9e712 in __interceptor_calloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:76
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #1 0x55f463698ee7 in esm_ebr_start_timer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/esm_ebr.c:297
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #2 0x55f463697b6f in eps_bearer_deactivate /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:569
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #3 0x55f463697be0 in esm_proc_eps_bearer_context_deactivate_request /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:229
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #4 0x55f46369b522 in esm_sap_send_a /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:926
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #5 0x55f46369c22e in esm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:248
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #6 0x55f463690dab in emm_cn_deactivate_dedicated_bearer_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:802
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #7 0x55f463690dab in emm_cn_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:1212
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #8 0x55f46369392e in emm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_sap.c:113
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #9 0x55f46367201f in nas_proc_delete_dedicated_bearer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/nas_proc.c:523
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #10 0x55f463343aa9 in mme_app_handle_nw_init_bearer_deactv_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_bearer.c:3217
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #11 0x55f46333b0ba in handle_message /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_main.c:405
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #12 0x7ff2b9f755c6 in zloop_start (/lib/x86_64-linux-gnu/libczmq.so.4+0x295c6)
Jul 27 13:25:32 magma-dev-focal mme[384880]: Indirect leak of 16 byte(s) in 1 object(s) allocated from:
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #0 0x7ff2bad9e9c1 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:54
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #1 0x55f463272440 in bstrcpy /home/vagrant/magma/lte/gateway/c/core/oai/lib/bstr/bstrlib.c:486
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #2 0x55f463698f14 in esm_ebr_start_timer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/esm_ebr.c:308
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #3 0x55f463697b6f in eps_bearer_deactivate /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:569
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #4 0x55f463697be0 in esm_proc_eps_bearer_context_deactivate_request /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:229
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #5 0x55f46369b522 in esm_sap_send_a /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:926
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #6 0x55f46369c22e in esm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:248
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #7 0x55f463690dab in emm_cn_deactivate_dedicated_bearer_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:802
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #8 0x55f463690dab in emm_cn_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:1212
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #9 0x55f46369392e in emm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_sap.c:113
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #10 0x55f46367201f in nas_proc_delete_dedicated_bearer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/nas_proc.c:523
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #11 0x55f463343aa9 in mme_app_handle_nw_init_bearer_deactv_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_bearer.c:3217
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #12 0x55f46333b0ba in handle_message /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_main.c:405
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #13 0x7ff2b9f755c6 in zloop_start (/lib/x86_64-linux-gnu/libczmq.so.4+0x295c6)
Jul 27 13:25:32 magma-dev-focal mme[384880]: Indirect leak of 8 byte(s) in 1 object(s) allocated from:
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #0 0x7ff2bad9e9c1 in __interceptor_malloc ../../../../src/libsanitizer/lsan/lsan_interceptors.cpp:54
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #1 0x55f463272460 in bstrcpy /home/vagrant/magma/lte/gateway/c/core/oai/lib/bstr/bstrlib.c:495
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #2 0x55f463698f14 in esm_ebr_start_timer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/esm_ebr.c:308
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #3 0x55f463697b6f in eps_bearer_deactivate /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:569
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #4 0x55f463697be0 in esm_proc_eps_bearer_context_deactivate_request /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/EpsBearerContextDeactivation.c:229
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #5 0x55f46369b522 in esm_sap_send_a /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:926
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #6 0x55f46369c22e in esm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/esm/sap/esm_sap.c:248
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #7 0x55f463690dab in emm_cn_deactivate_dedicated_bearer_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:802
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #8 0x55f463690dab in emm_cn_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_cn.c:1212
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #9 0x55f46369392e in emm_sap_send /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/emm/sap/emm_sap.c:113
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #10 0x55f46367201f in nas_proc_delete_dedicated_bearer /home/vagrant/magma/lte/gateway/c/core/oai/tasks/nas/nas_proc.c:523
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #11 0x55f463343aa9 in mme_app_handle_nw_init_bearer_deactv_req /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_bearer.c:3217
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #12 0x55f46333b0ba in handle_message /home/vagrant/magma/lte/gateway/c/core/oai/tasks/mme_app/mme_app_main.c:405
Jul 27 13:25:32 magma-dev-focal mme[384880]:     #13 0x7ff2b9f755c6 in zloop_start (/lib/x86_64-linux-gnu/libczmq.so.4+0x295c6)
Jul 27 13:25:32 magma-dev-focal mme[384880]: SUMMARY: LeakSanitizer: 216 byte(s) leaked in 5 allocation(s).
Jul 27 13:25:32 magma-dev-focal systemd[1]: magma@mme.service: Main process exited, code=exited, status=23/n/a
Jul 27 13:25:32 magma-dev-focal systemd[1]: magma@mme.service: Failed with result 'exit-code'.
Jul 27 13:25:32 magma-dev-focal systemd[1]: Stopped Magma OAI MME service.

Test Plan

Executed s1ap sanity test suite
Executed test_attach_detach_dedicated_deactivation_timer_expiry.py test case and stopped all services to check memory leaks. This is repeated multiple times.

Signed-off-by: Rashmi rashmi.sarwad@radisys.com

@rsarwad rsarwad requested a review from a team August 2, 2021 04:29
@pull-request-size pull-request-size bot added the size/M Denotes a PR that changes 30-99 lines. label Aug 2, 2021
@rsarwad rsarwad requested a review from ulaskozat August 2, 2021 04:29
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Thanks for opening a PR! 💯

A couple initial guidelines

Howto

  • Reviews. The "Reviewers" listed for this PR are the Magma maintainers who will shepherd it.
  • Checks. All required CI checks must pass before merge.
  • Merge. Once approved and passing CI checks, use the ready2merge label to indicate the maintainers can merge your PR.

More info

Please take a moment to read through the Magma project's

If this is your first Magma PR, also consider reading

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

@magmabot magmabot added the component: agw Access gateway-related issue label Aug 2, 2021
@ardzoht ardzoht changed the title fix(agw): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure fix(mme): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure Aug 2, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

@ardzoht
Copy link
Contributor

ardzoht commented Aug 2, 2021

@rsarwad can you please add into the Summary the dump of the memory leak? it would help for tracing back exact leak output on the logs, thanks!

Copy link
Contributor

@ulaskozat ulaskozat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

feg-workflow

    2 files  145 suites   30s ⏱️
338 tests 338 ✔️ 0 💤 0 ❌
352 runs  352 ✔️ 0 💤 0 ❌

Results for commit 89101dc.

♻️ This comment has been updated with latest results.

@ulaskozat ulaskozat changed the title fix(mme): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure fix(mme): Fix memory leaks observed during network initiated dedicated bearer deactivation procedure Aug 2, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

@ulaskozat ulaskozat changed the title fix(mme): Fix memory leaks observed during network initiated dedicated bearer deactivation procedure fix(mme): Fixed memory leaks observed during network initiated dedicated bearer deactivation procedure Aug 2, 2021
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2021

Oops! Looks like you failed the Semantic PR check.

Howto

@ulaskozat
Copy link
Contributor

@rsarwad could you fix the commit title inline with the PR title as it is a single commit PR, commit title is expected to follow semantic rules as well.

@rsarwad rsarwad force-pushed the rsarwad_memory_leak_fix_on_dedicated_bearer branch from 07462e8 to a762755 Compare August 5, 2021 05:18
@github-actions
Copy link
Contributor

github-actions bot commented Aug 5, 2021

…ted bearer deactivation procedure

Signed-off-by: Rashmi <rashmi.sarwad@radisys.com>
@rsarwad rsarwad force-pushed the rsarwad_memory_leak_fix_on_dedicated_bearer branch from a762755 to 89101dc Compare August 5, 2021 05:42
@ulaskozat ulaskozat enabled auto-merge (squash) August 5, 2021 07:22
@ulaskozat ulaskozat merged commit feb97e6 into magma:master Aug 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: agw Access gateway-related issue size/M Denotes a PR that changes 30-99 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants