Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rec: reduce RPZ memory usage #13830

Merged
merged 3 commits into from Feb 27, 2024
Merged

rec: reduce RPZ memory usage #13830

merged 3 commits into from Feb 27, 2024

Conversation

omoerbeek
Copy link
Member

@omoerbeek omoerbeek commented Feb 26, 2024

Short description

This PR contains two ways to reduce RPZ memory usage:

  1. Use unique_ptrs for optional data in a Policy object. This reduces the memory used per RPZs by roughly 40% in typical cases.
  2. Avoid keeping a reference to the Lua config object in the thread startup code. This avoid keeping the RPZs loaded at startup around after a rec_control reload-lua-config. Tested with rpzFile and rec_control reload-lua-config, but I suspect rpzPrimary has the same issue.

Fixes #13827 and fixes #12870 (duplicate)

Some memory usage data, first a clean startup wit a couple of RPZ loaded via rpzPrimary():

 1043 otto      10    0 2344M 2327M idle      fsleep    1:15  0.00% pdns_recursor <- master
68681 otto      10    0 1404M 1386M idle      fsleep    1:30  0.00% pdns_recursor <- PR code

On another test with a couple of RPZs loaded wit rpzFile:

      initial   1st  2nd 3rd reload
master    355   682  689 689
PR        218   226  232 232

Checklist

I have:

  • read the CONTRIBUTING.md document
  • compiled this code
  • tested this code
  • included documentation (including possible behaviour changes)
  • documented the code
  • added or modified regression test(s)
  • added or modified unit test(s)

@coveralls
Copy link

Pull Request Test Coverage Report for Build 8046905464

Details

  • -46 of 153 (69.93%) changed or added relevant lines in 8 files are covered.
  • 40 unchanged lines in 8 files lost coverage.
  • Overall coverage decreased (-0.01%) to 59.188%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pdns/recursordist/pdns_recursor.cc 0 2 0.0%
pdns/recursordist/rec-lua-conf.cc 0 5 0.0%
pdns/recursordist/filterpo.cc 39 47 82.98%
pdns/recursordist/filterpo.hh 36 45 80.0%
pdns/recursordist/lua-recursor4.cc 0 22 0.0%
Files with Coverage Reduction New Missed Lines %
pdns/backends/gsql/gsqlbackend.hh 2 97.71%
pdns/misc.hh 3 82.17%
pdns/recursordist/test-syncres_cc2.cc 3 88.85%
pdns/misc.cc 5 62.57%
pdns/signingpipe.cc 5 85.75%
pdns/dnsdist-carbon.cc 6 59.62%
pdns/dnsdist-tcp.cc 8 75.83%
pdns/packethandler.cc 8 72.82%
Totals Coverage Status
Change from base Build 8000517227: -0.01%
Covered Lines: 113563
Relevant Lines: 158645

💛 - Coveralls

@omoerbeek omoerbeek merged commit 5ac9f90 into PowerDNS:master Feb 27, 2024
75 checks passed
@omoerbeek omoerbeek deleted the rec-rpz-mem branch February 27, 2024 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rec: RPZs use more memory than strictly needed rec: optimize RPZ memory usage
3 participants