PowerDNS can't work with large initialization of zone

[XooR]

Problem is when you have around 500 entries and need to create new zone from configuration. Then I get response 500 from PowerDNS webserver:

I added some debug info to figure out why this is happening:

2017-10-18T16:33:27  [139776865289984] INFO  PowerDnsProvider[powerdns] _apply:   creating zone=example.lan.
Response:
<Response [500]>
Internal Server Error

Request:
{'Content-Length': '72640', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.18.4', 'Connection': 'keep-alive', 'X-API-Key': 'bla123', 'Content-Type': 'application/json'}
POST
http://10.120.40.58:8081/api/v1/servers/localhost/zones

If I comment out most of entries (leave only 1 entry) then it passes and in second pass with all entries commented out it works OK. I guess this have something with PowerDNS webserver but I didn't found way to increase some limits in API.

You can reproduce problem with generating file:

mkdir config
for f in $(seq -f 'foo-%05g' 1 500); do echo "$f: { 'type': A, 'value': 127.0.0.1}"; done > config/example.lan.yaml

Use configuration:

---
providers:
  config:
    class: octodns.provider.yaml.YamlProvider
    directory: ./config
  powerdns:
    class: octodns.provider.powerdns.PowerDnsProvider
    host: '10.1.1.1'
    port: 8081
    api_key: 'bla123'

zones:
  example.lan.:
    sources:
      - config
    targets:
      - powerdns

I didn't found any error in PowerDNS except:

Oct 18 16:33:28 localhost pdns_server: Oct 18 16:33:28 HTTP ISE for "/api/v1/servers/localhost/zones": STL Exception: std::bad_alloc

[ross]

Hrm. I'm guessing there's a limit to the request/payload size that's being hit here. It'll likely require some sort of batching of creates similar to some of the other providers.

[ross]

Hi @XooR. Finally got a few minutes to sit down and try and reproduce this (took me a bit to get a local pdns server going 😁) I wasn't able to hit this problem: I tried with 500, 1500, and 5000 records all of which worked.

Any chance you have a proxy server in front of PowerDNS? Perhaps it has a max request limit or something, thought not sure how that'd be a 500. You might try turning pdns's loglevel to get more verbosity there to see what it has to say. Otherwise maybe try turning on debug logging for the http backend requests is using on your system, some info on that. There may be useful info in the response content and/or headers.

[XooR]

I don't have any proxy for pdns API.

In PowerDNS log I get (loglevel: 15):

Oct 23 10:53:49 test-dns-01 pdns_server: Oct 23 10:53:49 HTTP ISE for "/api/v1/servers/localhost/zones": STL Exception: Attempt to print an unset dnsname
Oct 23 10:53:49 test-dns-01 pdns_server: Oct 23 10:53:49 HTTP: Error result for "/api/v1/servers/localhost/zones": 500

requests debug log:

2017-10-23T10:58:05  [140568917448448] DEBUG Manager sync:   checking safety
2017-10-23T10:58:05  [140568917448448] DEBUG Manager sync:   applying
2017-10-23T10:58:05  [140568917448448] INFO  PowerDnsProvider[powerdns] apply: making changes
2017-10-23T10:58:05  [140568917448448] DEBUG PowerDnsProvider[powerdns] _apply: zone=foo.lan., len(changes)=471
2017-10-23T10:58:05  [140568917448448] DEBUG PowerDnsProvider[powerdns] _apply:   sending change request
2017-10-23T10:58:05  [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: method=PATCH, path=zones/foo.lan.
2017-10-23T10:58:05  [140568917448448] DEBUG urllib3.connectionpool Resetting dropped connection: 10.10.10.10
2017-10-23T10:58:05  [140568917448448] DEBUG urllib3.connectionpool http://10.10.10.10:8081 "PATCH /api/v1/servers/localhost/zones/foo.lan. HTTP/1.1" 422 52
2017-10-23T10:58:05  [140568917448448] DEBUG PowerDnsProvider[powerdns] _request:   status=422
2017-10-23T10:58:05  [140568917448448] INFO  PowerDnsProvider[powerdns] _apply:   creating zone=foo.lan.
2017-10-23T10:58:05  [140568917448448] DEBUG PowerDnsProvider[powerdns] _request: method=POST, path=zones
2017-10-23T10:58:05  [140568917448448] DEBUG urllib3.connectionpool Resetting dropped connection: 10.10.10.10
2017-10-23T10:58:06  [140568917448448] DEBUG urllib3.connectionpool http://10.10.10.10:8081 "POST /api/v1/servers/localhost/zones HTTP/1.1" 500 21
2017-10-23T10:58:06  [140568917448448] DEBUG PowerDnsProvider[powerdns] _request:   status=500
2017-10-23T10:58:06  [140568917448448] ERROR PowerDnsProvider[powerdns] _apply:   status=500, text=Internal Server Error
Traceback (most recent call last):
  File "./bin/octodns-sync", line 11, in <module>
    sys.exit(main())
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/cmds/sync.py", line 39, in main
    dry_run=not args.doit, force=args.force)
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/manager.py", line 313, in sync
    total_changes += target.apply(plan)
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/base.py", line 143, in apply
    self._apply(plan)
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 376, in _apply
    self._post('zones', data)
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 67, in _post
    return self._request('POST', path, data=data)
  File "/home/foo/env/local/lib/python2.7/site-packages/octodns/provider/powerdns.py", line 47, in _request
    resp.raise_for_status()
  File "/home/foo/env/local/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://10.10.10.10:8081/api/v1/servers/localhost/zones

Also pip list from my virtualenv:

certifi (2017.7.27.1)
chardet (3.0.4)
dnspython (1.15.0)
futures (3.1.1)
idna (2.6)
incf.countryutils (1.0)
ipaddress (1.0.18)
natsort (5.1.0)
octodns (0.8.7)
pip (9.0.1)
pkg-resources (0.0.0)
python-dateutil (2.6.1)
PyYAML (3.12)
requests (2.18.4)
setuptools (36.6.0)
six (1.11.0)
urllib3 (1.22)
wheel (0.30.0)

UPDATE: I also have installed pdns 4.1.0 version on CentOS 7 with Postgres backend. I can make Ansible playbooks for test setup if that would help.

[ross]

Fwiw, the version of powerdns i'm using is 4.0.4 backed by mysql, it was just the version that the docker container I found relied on. This is starting to 👃 more like a pdns bug than hitting some sort of max request size. I'll poke around and see if I can get my container to use 4.1.0 to see if that makes a difference. If the difference is mysql vs postgres then it'll almost certainly be a pdns issue.

[XooR]

I'll try it out and will let you know what happens. I have already automated setup, so I can test easily.

[XooR]

It looks that 4.1.0 version put limit in webapi requests because when I installed PowerDNS Auth with mysql backend it still failed. With version 4.0.4 it doesn't fail.
I'll check with PowerDNS what we should do in this case and will update this Issue.

[ross]

I'll check with PowerDNS what we should do in this case and will update this Issue.

👍. Cool. I kind of expect it to be a bug more than a limit as if it was an intentional limit I'd imagine they'd use a different status code and print a specific error about the problem. Either way pdns is the place to look at this point. Depending on what we find we can look at a setting a batch size-limit if necessary. Feel free to CC me if you end up filling issues there and we can link this up to them if so.

[joewilliams]

FWIW it seems like powerdns updated it's underlying http library. The update included changes like support for multipart which likely has an impact on large requests like this. I don't see any substantive changes in the webserver that might be applicable to this issue.

[dewet22]

Just ran into this myself; trying to repopulate a new pdns installation results in failure. The workaround that seems to work for me is to create the new zone using only one or two records, and then load the rest of the zone in bulk.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.