Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10. #3764

Closed
389-ds-bot opened this issue Sep 13, 2020 · 25 comments
Closed

Several memory leaks reported by Valgrind for 389-ds 1.3.9.1-10. #3764

389-ds-bot opened this issue Sep 13, 2020 · 25 comments
Labels
closed: fixed Migration flag - Issue
Milestone

Comments

@389-ds-bot
Copy link

Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/50709


Ticket was cloned from Red Hat Bugzilla (product Red Hat Enterprise Linux 7): Bug 1769418

Description of problem:

A customer is experiencing a high memory usage with RHDS.
The memory consumption starts from a few GB and can go up to 100 GB after a
week.

Version-Release number of selected component (if applicable):

$ cat etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
$
$ grep 389-ds-base installed-rpms
389-ds-base-1.3.9.1-10.el7.x86_64                           Mon Aug 26 14:13:40
2019
389-ds-base-debuginfo-1.3.9.1-10.el7.x86_64                 Tue Nov  5 10:20:03
2019
389-ds-base-libs-1.3.9.1-10.el7.x86_64                      Mon Aug 26 14:13:26
2019
$

How reproducible:

Always at customer site.

Steps to Reproduce:

1. Restart the RHDS instance.
The sum of the configured caches is about 1.3 GB:
$ grep total errors | tail -1
[05/Nov/2019:11:32:52.600935718 +0000] - NOTICE - ldbm_back_start - total cache
size: 1316466524 B;
$

2. The memory usage will grow significantly:
Eg:
$ head -1 ./ps
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
$
$ grep ns-slapd ./ps
dirsrv    3696 79.9 93.6 64044164 61602916 ?   Ssl  Oct15 18237:22
/usr/sbin/ns-slapd -D /etc/dirsrv/slapd-XXX -i /var/run/dirsrv/slapd-XXX.pid
$


Actual results:

High memory usage indicating memory leaks.

Expected results:

The memory usage should not go that high.
Need to fix the memory leaks.

Additional info:

From the Valgrind output:

* The leak summary:

Direct leak of 112 byte(s) in 1 object(s) allocated from:
    0 0x7f83d893cc58 in __interceptor_malloc (/lib64/libasan.so.5+0x10dc58)
    1 0x7f83d859b028 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb028)
    2 0x7f83d4267069 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:265
    3 0x7f83d40a3ba9 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x46ba9)
    4 0x7f83d40a0e9c in LASIpEval lib/libaccess/lasip.cpp:496
    5 0x7f83d40a5168 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    6 0x7f83d40a6ef2 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    7 0x7f83d40a7f2d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.3.20191112git15789e895.so+0x4af2d)
    8 0x7f83d4242a00 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    9 0x7f83d424be3e in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x22e3e)
    10 0x7f83d427e2a0 in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    11 0x7f83d86803c9 in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c03c9)
    12 0x7f83d86810bd in slapi_access_allowed (/usr/lib64/dirsrv/libslapd.so.0+0x1c10bd)
    13 0x7f83d424ed11 in acl_check_mods (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x25d11)
    14 0x7f83d8680632 in plugin_call_acl_mods_access (/usr/lib64/dirsrv/libslapd.so.0+0x1c0632)
    15 0x7f83d1948699 in ldbm_back_modify ldap/servers/slapd/back-ldbm/ldbm_modify.c:616
    16 0x7f83d863c019 in op_shared_modify ldap/servers/slapd/modify.c:1021
    17 0x7f83d863fa93 in do_modify (/usr/lib64/dirsrv/libslapd.so.0+0x17fa93)
    18 0x561c3cf403f9 in connection_dispatch_operation ldap/servers/slapd/connection.c:636
    19 0x561c3cf403f9 in connection_threadmain ldap/servers/slapd/connection.c:1765
    20 0x7f83d8287868  (/lib64/libnspr4.so+0x2a868)
@389-ds-bot 389-ds-bot added the closed: fixed Migration flag - Issue label Sep 13, 2020
@389-ds-bot 389-ds-bot added this to the 1.3.10 milestone Sep 13, 2020
@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-11-12 14:40:43

Metadata Update from @tbordaz:

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-11-12 14:53:58

Metadata Update from @tbordaz:

  • Issue assigned to tbordaz

@389-ds-bot
Copy link
Author

Comment from lkrispen (@elkris) at 2019-11-12 14:57:50

I'm sorry, but I don't understand how the leak in acl relates to the memory leak reported by the customer. The customer leaks were in ssl_Recv(), so how does the acl leak fit in

@389-ds-bot
Copy link
Author

Comment from lkrispen (@elkris) at 2019-11-12 14:57:50

Metadata Update from @elkris:

  • Assignee reset
  • Custom field origin adjusted to None
  • Custom field reviewstatus adjusted to None

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-11-12 15:32:17

@elkris the valgrind output contains several signatures. One of them is related to DS_LASIpGetter. It is not the only one but account for one third of the overall leaks.

@389-ds-bot
Copy link
Author

Comment from lkrispen (@elkris) at 2019-11-12 15:40:33

thanks, I hadn't looked at the full report, sorry for the noise

@389-ds-bot
Copy link
Author

389-ds-bot commented Sep 13, 2020

Comment from tbordaz (@tbordaz) at 2019-11-13 18:45:45

PR #3765

@389-ds-bot
Copy link
Author

Comment from mreynolds (@mreynolds389) at 2019-11-14 17:47:16

Metadata Update from @mreynolds389:

  • Issue set to the milestone: 1.4.0 (was: 0.0 NEEDS_TRIAGE)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-12-09 15:54:43

8a604aa..463d6b7 master
19b24c8..8fbe976 389-ds-base-1.4.1
881a53f..e141883 389-ds-base-1.4.0

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-12-09 15:54:59

Metadata Update from @tbordaz:

  • Issue assigned to tbordaz

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2019-12-09 15:55:03

Metadata Update from @tbordaz:

  • Issue close_status updated to: fixed
  • Issue status updated to: Closed (was: Open)

@389-ds-bot
Copy link
Author

Comment from mreynolds (@mreynolds389) at 2020-01-15 22:44:35

This fix causes a crash in ASAN:

py.test ./paged_results_test.py::test_search_dns_ip_aci

ASAN output:

==1009==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b000079f60 at pc 0x7fd3d80cdd7d bp 0x7fd39cef5b20 sp 0x7fd39cef5b10
READ of size 4 at 0x60b000079f60 thread T13
    0 0x7fd3d80cdd7c in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1b1d7c)
    1 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    2 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    3 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    4 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    5 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    6 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    7 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    8 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    9 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    10 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    11 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    12 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    13 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    14 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    15 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    16 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    17 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    18 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    19 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    20 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    21 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)
    22 0x7fd3d527f2dd in start_thread (/lib64/libpthread.so.0+0x82dd)
    23 0x7fd3d4a13e82 in __GI___clone (/lib64/libc.so.6+0xfbe82)

0x60b000079f60 is located 0 bytes inside of 112-byte region [0x60b000079f60,0x60b000079fd0)
freed by thread T13 here:
    0 0x7fd3d857b7b0 in __interceptor_free (/lib64/libasan.so.5+0xef7b0)
    1 0x7fd3d7ff7a6c in slapi_ch_free (/usr/lib64/dirsrv/libslapd.so.0+0xdba6c)
    2 0x7fd3d80c5830 in slapi_pblock_set (/usr/lib64/dirsrv/libslapd.so.0+0x1a9830)
    3 0x7fd3c94ed3cd in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:308
    4 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    5 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    6 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    7 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    8 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    9 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    10 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    11 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    12 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    13 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    14 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    15 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    16 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    17 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    18 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    19 0x7fd3d80afc64 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x193c64)
    20 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    21 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    22 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    23 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

previously allocated by thread T20 here:
    0 0x7fd3d857bb78 in __interceptor_malloc (/lib64/libasan.so.5+0xefb78)
    1 0x7fd3d7ff7277 in slapi_ch_malloc (/usr/lib64/dirsrv/libslapd.so.0+0xdb277)
    2 0x7fd3c94ed499 in DS_LASIpGetter ldap/servers/plugins/acl/acllas.c:269
    3 0x7fd3c9254128 in ACL_GetAttribute (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x46128)
    4 0x7fd3c92513b8 in LASIpEval lib/libaccess/lasip.cpp:496
    5 0x7fd3c92556d5 in ACLEvalAce(NSErr_s*, ACLEvalHandle*, ACLExprHandle*, unsigned long*, PListStruct_s**, PListStruct_s*) lib/libaccess/oneeval.cpp:215
    6 0x7fd3c9257400 in ACL_INTEvalTestRights lib/libaccess/oneeval.cpp:752
    7 0x7fd3c925852d in ACL_EvalTestRights (/usr/lib64/dirsrv/libns-dshttpd-1.4.2.4.so+0x4a52d)
    8 0x7fd3c94c7e81 in acl__TestRights ldap/servers/plugins/acl/acl.c:3289
    9 0x7fd3c94d14ce in acl_access_allowed (/usr/lib64/dirsrv/plugins/libacl-plugin.so+0x224ce)
    10 0x7fd3c950498b in acl_access_allowed_main ldap/servers/plugins/acl/aclplugin.c:371
    11 0x7fd3d80dfb0a in plugin_call_acl_plugin (/usr/lib64/dirsrv/libslapd.so.0+0x1c3b0a)
    12 0x7fd3d804908e in test_filter_access ldap/servers/slapd/filterentry.c:956
    13 0x7fd3d804908e in slapi_vattr_filter_test_ext_internal ldap/servers/slapd/filterentry.c:817
    14 0x7fd3d804aeaa in slapi_vattr_filter_test_ext (/usr/lib64/dirsrv/libslapd.so.0+0x12eeaa)
    15 0x7fd3c5b1bcb9 in ldbm_back_next_search_entry_ext ldap/servers/slapd/back-ldbm/ldbm_search.c:1713
    16 0x7fd3d80aa4ff in iterate ldap/servers/slapd/opshared.c:1307
    17 0x7fd3d80aa4ff in send_results_ext ldap/servers/slapd/opshared.c:1660
    18 0x7fd3d80b0191 in op_shared_search (/usr/lib64/dirsrv/libslapd.so.0+0x194191)
    19 0x56159e8f726c in do_search ldap/servers/slapd/search.c:376
    20 0x56159e8c7092 in connection_dispatch_operation ldap/servers/slapd/connection.c:662
    21 0x56159e8c7092 in connection_threadmain ldap/servers/slapd/connection.c:1767
    22 0x7fd3d58e4567  (/lib64/libnspr4.so+0x2b567)

@389-ds-bot
Copy link
Author

Comment from mreynolds (@mreynolds389) at 2020-01-15 22:44:35

Metadata Update from @mreynolds389:

  • Issue status updated to: Open (was: Closed)

@389-ds-bot
Copy link
Author

389-ds-bot commented Sep 13, 2020

Comment from tbordaz (@tbordaz) at 2020-01-20 11:30:27

New PR for the heap-user-after-free regression

#3887

@389-ds-bot
Copy link
Author

389-ds-bot commented Sep 13, 2020

Comment from mreynolds (@mreynolds389) at 2020-01-20 15:41:12

New PR for the heap-user-after-free regression
#3887

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

@389-ds-bot
Copy link
Author

389-ds-bot commented Sep 13, 2020

Comment from mreynolds (@mreynolds389) at 2020-01-20 15:42:16

New PR for the heap-user-after-free regression
#3887

Please cherry-pick to 1.4.2 & 1.4.1, thanks!

And 1.3.10 for customer hot fix...

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-20 15:50:07

77327b009...058f4da2b master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

@389-ds-bot
Copy link
Author

Comment from mreynolds (@mreynolds389) at 2020-01-20 15:57:41

77327b0...058f4da master
b747be0..b05c86d 389-ds-base-1.4.2
44d9b73..1d748c5 389-ds-base-1.4.1
ea0af42..9fd5623 389-ds-base-1.4.0

Don't forget 1.3.10 ;-)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-21 11:08:50

bcb1e24..f1d48ed 389-ds-base-1.3.10 (thanks @vashirov for the asan build !)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-21 11:10:10

Metadata Update from @tbordaz:

  • Issue set to the milestone: 1.3.10 (was: 1.4.0)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-21 11:10:18

Metadata Update from @tbordaz:

  • Issue close_status updated to: fixed
  • Issue status updated to: Closed (was: Open)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-24 11:00:52

Unfortunately the backport in 1.3.10 is invalid :( and triggers a hang
Working on a fix

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-24 11:01:00

Metadata Update from @tbordaz:

  • Issue status updated to: Open (was: Closed)

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-24 17:35:34

fixing 1.3.10 backport: f1d48ed..dab015b 389-ds-base-1.3.10

@389-ds-bot
Copy link
Author

Comment from tbordaz (@tbordaz) at 2020-01-27 10:07:30

Metadata Update from @tbordaz:

  • Issue close_status updated to: fixed
  • Issue status updated to: Closed (was: Open)

tbordaz added a commit to tbordaz/389-ds-base that referenced this issue Jun 10, 2021
…ssion connection flags

Bug description:
	The fix for ticket 389ds#3764 was broken with a missing break in a
	switch. The consequence is that while setting the client IP
	address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
	connection is erroneously set as replication connection.
        This can lead to crash or failure of testcase
        test_access_from_certain_network_only_ip.
        This bug was quite hidden until the fix for 389ds#4764 is
        showing it more frequently

Fix description:
	Add the missing break

relates: 389ds#4797

Reviewed by: Mark Reynolds

Platforms tested: F33
tbordaz added a commit that referenced this issue Jun 10, 2021
…ssion connection flags (#4799)

Bug description:
	The fix for ticket #3764 was broken with a missing break in a
	switch. The consequence is that while setting the client IP
	address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
	connection is erroneously set as replication connection.
        This can lead to crash or failure of testcase
        test_access_from_certain_network_only_ip.
        This bug was quite hidden until the fix for #4764 is
        showing it more frequently

Fix description:
	Add the missing break

relates: #4797

Reviewed by: Mark Reynolds

Platforms tested: F33
tbordaz added a commit that referenced this issue Jun 10, 2021
…ssion connection flags (#4799)

Bug description:
	The fix for ticket #3764 was broken with a missing break in a
	switch. The consequence is that while setting the client IP
	address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
	connection is erroneously set as replication connection.
        This can lead to crash or failure of testcase
        test_access_from_certain_network_only_ip.
        This bug was quite hidden until the fix for #4764 is
        showing it more frequently

Fix description:
	Add the missing break

relates: #4797

Reviewed by: Mark Reynolds

Platforms tested: F33
tbordaz added a commit that referenced this issue Jun 10, 2021
…ssion connection flags (#4799)

Bug description:
	The fix for ticket #3764 was broken with a missing break in a
	switch. The consequence is that while setting the client IP
	address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
	connection is erroneously set as replication connection.
        This can lead to crash or failure of testcase
        test_access_from_certain_network_only_ip.
        This bug was quite hidden until the fix for #4764 is
        showing it more frequently

Fix description:
	Add the missing break

relates: #4797

Reviewed by: Mark Reynolds

Platforms tested: F33
tbordaz added a commit that referenced this issue Jun 10, 2021
…ssion connection flags (#4799)

Bug description:
    The fix for ticket #3764 was broken with a missing break in a
    switch. The consequence is that while setting the client IP
    address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
    connection is erroneously set as replication connection.
    This can lead to crash or failure of testcase
    test_access_from_certain_network_only_ip.
    This bug was quite hidden until the fix for #4764 is
    showing it more frequently

Fix description:
    Add the missing break

relates: #4797

Reviewed by: Mark Reynolds

Platforms tested: F33
mreynolds389 pushed a commit that referenced this issue Aug 12, 2021
…ssion connection flags (#4799)

Bug description:
	The fix for ticket #3764 was broken with a missing break in a
	switch. The consequence is that while setting the client IP
	address in the pblock (SLAPI_CONN_CLIENTNETADDR_ACLIP), the
	connection is erroneously set as replication connection.
        This can lead to crash or failure of testcase
        test_access_from_certain_network_only_ip.
        This bug was quite hidden until the fix for #4764 is
        showing it more frequently

Fix description:
	Add the missing break

relates: #4797

Reviewed by: Mark Reynolds

Platforms tested: F33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed: fixed Migration flag - Issue
Projects
None yet
Development

No branches or pull requests

1 participant