ImportError: No module named 'dns.rdtypes.IN.NS'

[Arkiver2]

We received this error for a crawl by the NewsGrabber project on a list of URLs:

ERROR Fatal exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/dns/rdata.py", line 389, in get_rdata_class
    rdclass_text, rdtype_text]))
  File "/usr/local/lib/python3.4/dist-packages/dns/rdata.py", line 374, in import_module
    mod = __import__(name)
ImportError: No module named 'dns.rdtypes.IN.NS'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/wpull/app.py", line 128, in run
    yield From(self._builder.factory['Engine']())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/engine.py", line 281, in __call__
    yield From(self._run_workers())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/local/lib/python3.4/dist-packages/wpull/engine.py", line 70, in _run_workers
    task.result()
  File "/usr/local/lib/python3.4/dist-packages/trollius/futures.py", line 287, in result
    raise self._exception
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/engine.py", line 149, in _run_worker
    yield From(self._process_item(item))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/engine.py", line 330, in _process_item
    yield From(self._process_url_item(url_record))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/engine.py", line 387, in _process_url_item
    yield From(self._processor.process(url_item))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/processor/delegate.py", line 27, in process
    raise Return((yield From(self.web_processor.process(url_item))))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/processor/web.py", line 123, in process
    raise Return((yield From(session.process())))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/processor/web.py", line 215, in process
    yield From(self._process_loop())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/processor/web.py", line 274, in _process_loop
    exit_early, wait_time = yield From(self._fetch_one(self._request))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/processor/web.py", line 319, in _fetch_one
    duration_timeout=self._fetch_rule.duration_timeout
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/http/web.py", line 167, in fetch
    response = yield From(session.fetch(request))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/http/client.py", line 70, in fetch
    yield From(self._stream.reconnect())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/http/stream.py", line 445, in reconnect
    yield From(self._connection.connect())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/connection.py", line 824, in connect
    results = yield From(self._resolver.resolve_dual(self._address[0], self._address[1]))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 143, in resolve_dual
    results = list((yield From(self.resolve_all(host, port))))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 87, in resolve_all
    results = yield From(self._resolve_from_network(host, port))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 197, in _resolve_from_network
    results = yield From(trollius.wait_for(future, self._timeout))
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 259, in _step
    result = coro.send(value)
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 443, in wait_for
    raise Return(fut.result())
  File "/usr/local/lib/python3.4/dist-packages/trollius/futures.py", line 287, in result
    raise self._exception
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 322, in _getaddrinfo_implementation
    yield From(query_ipv4())
  File "/usr/local/lib/python3.4/dist-packages/trollius/tasks.py", line 257, in _step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 306, in query_ipv4
    None, self._query, host, 'A'
  File "/usr/lib/python3.4/concurrent/futures/thread.py", line 54, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.4/dist-packages/wpull/dns.py", line 352, in _query
    host, query_type, raise_on_no_answer=False
  File "/usr/local/lib/python3.4/dist-packages/dns/resolver.py", line 834, in query
    source_port=source_port)
  File "/usr/local/lib/python3.4/dist-packages/dns/query.py", line 230, in udp
    one_rr_per_rrset=one_rr_per_rrset)
  File "/usr/local/lib/python3.4/dist-packages/dns/message.py", line 791, in from_wire
    reader.read()
  File "/usr/local/lib/python3.4/dist-packages/dns/message.py", line 731, in read
    self._get_section(self.message.authority, aucount)
  File "/usr/local/lib/python3.4/dist-packages/dns/message.py", line 704, in _get_section
    self.message.origin)
  File "/usr/local/lib/python3.4/dist-packages/dns/rdata.py", line 476, in from_wire
    cls = get_rdata_class(rdclass, rdtype)
  File "/usr/local/lib/python3.4/dist-packages/dns/rdata.py", line 394, in get_rdata_class
    'ANY', rdtype_text]))
  File "/usr/local/lib/python3.4/dist-packages/dns/rdata.py", line 377, in import_module
    mod = getattr(mod, comp)
AttributeError: 'module' object has no attribute 'ANY'
CRITICAL Sorry, Wpull unexpectedly crashed.
CRITICAL Please report this problem to the authors at Wpull's issue tracker so it may be fixed. If you know how to program, maybe help us fix it? Thank you for helping us help you help us all.
Created lmdb db with map_size=1099511627776
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/igsets
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/max_content_length
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/delay
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/concurrency
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/custom_hooks.py
Picked up the changes to /home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f/ignores
Finished grab c180167f161b5b929cb02a78aea23452 file:///home/archiveteam/newsbuddy/old_lists/list_temp1fgnkyvspif with exit code 1
Output is in directory:
/home/archiveteam/newsbuddy/home-archiveteam-newsbuddy-old_lists-list_temp1fgnkyvspif-2016-04-03-c180167f

[JustAnotherArchivist]

This looks like an issue with dnspython – either a buggy version of that package or a botched installation which somehow didn't install the dns.rdtypes.* packages. The relevant code in dnspython hasn't changed in over 13 years, so this could be an incompatibility with current Python versions. Interesting though that there is no report whatsoever about this on the dnspython issue tracker although there have been at least four cases of this on ArchiveTeam...

wpull itself doesn't use dns.rdata or dns.rdtypes at all (and never has, as far as I can see).