Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance modify rate: reduce lock contention on the object extension factory #4600

Open
tbordaz opened this issue Feb 8, 2021 · 1 comment
Labels
Need BZ The ticket needs to be cloned to a BZ performance Issue impacts performance priority_high need urgent fix / highly valuable / easy to fix
Milestone

Comments

@tbordaz
Copy link
Contributor

tbordaz commented Feb 8, 2021

Issue Description
/opt/ldapbench/bin/modrate --hostname instance --port 389 --bindDN 'cn=Directory Manager' --bindPassword xxx --entryDN id=user.[1-10000],ou=People,dc=example,dc=com" --attribute street --attribute description --valueLength 12 --numThreads 5

During these modifies, target entry is allocated/duplicated/free. entry structure contains a object (entry) extension that is allocated via a factory slapi_register_object_extension.
ATM the purpose of this factory is to store "unhashed#user#password" in case of update of the password (pw_exp_init).

The factory is protected via a NSPR simple mutex and is the second hottest lock in contention

#        ----------- Cacheline ----------    Total      Tot  ----- LLC Load Hitm -----  ---- Store Reference ----  --- Load Dram ----      LLC    Total
# Index             Address  Node  PA cnt  records     Hitm    Total      Lcl      Rmt    Total    L1Hit   L1Miss       Lcl       Rmt  Ld Miss    Loads
# .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  ........  ........  .......  .......
#
      0      0x7f84e9c28500     0     158     1877   16.32%       31       31        0      838      838        0         0         0        0     1039
      1      0x7f84e9c2a980     0      71      109    5.79%       11       11        0       29       29        0         0         0        0       80
      2      0x7f84e9c28340     0      49       77    4.21%        8        8        0       14       14        0         0         0        0       63
      3      0x7f84d74e1d00     0     223      308    3.68%        7        7        0      210      210        0         0         0        0       98
      4      0x7f84e4c18600     0      70       99    3.16%        6        6        0       34       34        0         0         0        0       65

#        ----- HITM -----  -- Store Refs --  --------- Data address ---------                      ---------- cycles ----------    Total
#   Num      Rmt      Lcl   L1 Hit  L1 Miss              Offset  Node  PA cnt        Code address  rmt hitm  lcl hitm      load  records

  -------------------------------------------------------------
      1        0       11       29        0      0x7f84e9c2a980
  -------------------------------------------------------------
           0.00%    0.00%   55.17%    0.00%                 0x0     0       1      0x7f84ee04ea86         0         0       189       28
            |
            ---__GI___pthread_mutex_lock (inlined)
               PR_Lock
               |
               |--30.92%--factory_destroy_extension
...
                --23.85%--factory_create_extension

Moving this mutex to a pthread mutex reduce the contention to be 5th hottest

#        ----------- Cacheline ----------    Total      Tot  ----- LLC Load Hitm -----  ---- Store Reference ----  --- Load Dram ----      LLC    Total
# Index             Address  Node  PA cnt  records     Hitm    Total      Lcl      Rmt    Total    L1Hit   L1Miss       Lcl       Rmt  Ld Miss    Loads
# .....  ..................  ....  ......  .......  .......  .......  .......  .......  .......  .......  .......  ........  ........  .......  .......
#
      0      0x7f0850628600     0     194     2192   18.38%       34       34        0      967      967        0         0         0        0     1225
      1      0x7f0850628440     0      53       83    7.03%       13       13        0       29       29        0         0         0        0       54
      2      0x7f084b618700     0      74      105    5.41%       10       10        0       23       23        0         0         0        0       82
      3      0x7f085062c500     0      21       27    5.41%       10       10        0        0        0        0         0         0        0       27
      4      0x7f0850628380     0      48       92    3.78%        7        7        0       29       29        0         0         0        0       63

  -------------------------------------------------------------
      4        0        7       29        0      0x7f0850628380
  -------------------------------------------------------------
           0.00%    0.00%   31.03%    0.00%                 0x0     0       1      0x7f0854a41963         0         0       112       28         6  [.] __pthread_mutex_lock        libpthread-2.28.so  __pthread_mutex_lock+67          0
            |
            ---__GI___pthread_mutex_lock (inlined)
               |
               |--22.31%--factory_destroy_extension
...
                --8.28%--factory_create_extension

This change gives a 1% increase of the throughput: 3400 vs 3370/s
This change will also impact all others object extension (conn, op, entry,...)

Note: I think another ticket should be created to avoid creating systematically an entry extension. If possible it could rather been created on demand (if unhashed password is managed, if MODs update userpassword)
Package Version and Platform:
all version

Steps to Reproduce
Using modrate/perf_c2c

Expected results
Reduce contention on entry extension factory

@tbordaz tbordaz added the needs triage The issue will be triaged during scrum label Feb 8, 2021
tbordaz added a commit to tbordaz/389-ds-base that referenced this issue Feb 9, 2021
…bject extension factory

Bug description:
	object extension factory uses a simple mutex to protect allocation/destroy object.
	This mutex is a NSPR mutex. Using modrate load (entry object), the mutext is the
	second hottest contention. Moving it to pthread mutex moves it down to the fifth
	hotest.
	giving a small throughput benefit (1%)

Fix description:
	Use pthread normal mutex

relates: 389ds#4600

Reviewed by:  Simon Pichugin, William Brown

Platforms tested: RHEL 8.3 and F31
tbordaz added a commit that referenced this issue Feb 9, 2021
…bject extension factory (#4601)

Bug description:
	object extension factory uses a simple mutex to protect allocation/destroy object.
	This mutex is a NSPR mutex. Using modrate load (entry object), the mutext is the
	second hottest contention. Moving it to pthread mutex moves it down to the fifth
	hotest.
	giving a small throughput benefit (1%)

Fix description:
	Use pthread normal mutex

relates: #4600

Reviewed by:  Simon Pichugin, William Brown

Platforms tested: RHEL 8.3 and F31
@mreynolds389 mreynolds389 removed the needs triage The issue will be triaged during scrum label Feb 11, 2021
@mreynolds389 mreynolds389 added this to the 1.4.3 milestone Feb 11, 2021
@tbordaz tbordaz added the priority_high need urgent fix / highly valuable / easy to fix label Mar 25, 2021
@tbordaz tbordaz added the Need BZ The ticket needs to be cloned to a BZ label Jun 16, 2021
@tbordaz
Copy link
Contributor Author

tbordaz commented Jun 16, 2021

RHEL epic 8.6

@tbordaz tbordaz added the performance Issue impacts performance label Jun 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Need BZ The ticket needs to be cloned to a BZ performance Issue impacts performance priority_high need urgent fix / highly valuable / easy to fix
Projects
None yet
Development

No branches or pull requests

2 participants