New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tuning unit file for medium sized setups #4958

Merged
merged 2 commits into from Mar 6, 2017

Conversation

Projects
None yet
3 participants
@paddg
Contributor

paddg commented Jan 31, 2017

Short description

We would propose to introduce the TasksMax setting in the unit file. The default value of 512 seems too low, at least for our setup. In addition, if present in the unit file, people will be aware of it.

LimitNOFILE=4096 seems also a little bit low.

Our setup consists of 8 DNS backend servers. Each Server with 5 IPv4 and 5 IPv6 addresses. That means dnsdist has to deal with 80 backends. For this setup TasksMax=512 (default) was too low and dnsdist stopped working correctly. We saw many of these messages:

Error creating a TCP thread: Resource temporarily unavailable
Had an error accepting new webserver connection: Resource temporarily unavailable

Checklist

I have:

  • read the CONTRIBUTING.md document
  • compiled and tested this code
  • included documentation (including possible behaviour changes)
  • documented the code
  • added regression tests
  • added unit tests
  • checked that this code was merged to master
Tuning unit file for medium sized setups
We would propose to introduce the TasksMax setting in the unit file. The default value of 512 seems too low, at least for our setup. In addition, if present in the unit file, people will be aware of it.

LimitNOFILE seems also a little bit low.

Our setup consists of 8 DNS backend servers. Each Server with 5 IPv4 and 5 IPv6 addresses. That means dnsdist has to deal with 80 backends. For this setup TasksMax=512 (default) was too low and dnsdist stopped working correctly. We saw many of these messages:

Error creating a TCP thread: Resource temporarily unavailable
Had an error accepting new webserver connection: Resource temporarily unavailable
@rgacogne

This comment has been minimized.

Member

rgacogne commented Jan 31, 2017

Hi Winfried, thank you for this PR! The TasksMax increase was probably more needed because of the number of TCP threads (setMaxTCPClientThreads()) than because of the number of backends, but it seems like a good idea to increase TasksMax and LimitNOFILE anyway.

@rgacogne

This comment has been minimized.

Member

rgacogne commented Jan 31, 2017

It looks like TasksMax was introduced in systemd 227, this might be an issue with some distributions.

@paddg

This comment has been minimized.

Contributor

paddg commented Jan 31, 2017

Yes indeed, it came up here with SLES12-SP2

@pieterlexis

This comment has been minimized.

Member

pieterlexis commented Jan 31, 2017

We could have m4/systemd.m4 detect the version of systemd through pkg-config (e.g. PKG_CHECK_MODULES that sets an AM_CONDITIONAL) and add that line to the generated service file.

@paddg

This comment has been minimized.

Contributor

paddg commented Feb 3, 2017

From SYSTEMD.UNIT(5)
"If systemd encounters an unknown option, it will write a warning log message but continue loading the unit."

@rgacogne

This comment has been minimized.

Member

rgacogne commented Feb 3, 2017

Indeed:

Unknown lvalue 'TasksMax' in section 'Service'

@rgacogne

This comment has been minimized.

Member

rgacogne commented Feb 6, 2017

https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html#DefaultTasksMax= suggests than on recent distributions the default value is 4915. I don't think we should lower that value, so how about defaulting to 8192?

@paddg

This comment has been minimized.

Contributor

paddg commented Feb 6, 2017

For our setup I can say, we started dnsdist a week ago and the number of tasks was at ~250. Afterwards it increases from time to time.

systemctl status dnsdist.service | grep Tasks
    Tasks: 247 (limit: 2048)

Now after a week it looks like this:

systemctl status dnsdist.service | grep Tasks
    Tasks: 1004 (limit: 2048)

I'm curious if that stops at some point.

@rgacogne

This comment has been minimized.

Member

rgacogne commented Feb 6, 2017

I'm curious if that stops at some point.

We have three kinds of dynamically started threads:

  • console threads, to handle console connections
  • webserver threads, to handle HTTP requests
  • TCP worker threads

The first two are unbounded but the number of TCP worker threads can only grow up to setMaxTCPClientThreads(). I should probably add some options to restrict the maximum number of console and webserver threads too.

@rgacogne

This comment has been minimized.

Member

rgacogne commented Feb 20, 2017

Looks good, but if you don't mind I'd rather have TasksMax set to 8192 to prevent a possible regression on recent distributions :)

@paddg

This comment has been minimized.

Contributor

paddg commented Feb 20, 2017

Yes that's fine, thanks!

@rgacogne rgacogne merged commit 39c8af9 into PowerDNS:master Mar 6, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment