Map fetching URL exceptions need better handling

[wb8tyw]

Some Map URL fetches exceptions seen.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 546, in fetch_url
    data = urllib.request.urlopen(req)
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name
resolution>
03/15/2022 22:24:36:INFO:MapTile:fetch: [0] Failed to fetch `
https://tile.openstreetmap.de/14/8624/5841.png'
Traceback (most recent call last):
  File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/usr/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.9/socket.py", line 822, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 546, in fetch_url
    data = urllib.request.urlopen(req)
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name
resolution>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 501, in fetch
    self.fetch_url(url, self._local_path())
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 554, in fetch_url
    raise MapFetchError(err)
d_rats.map.maptile.MapFetchError: <urlopen error [Errno -3] Temporary
failure in name resolution>
03/15/2022 22:24:36:INFO:MapTile:fetch: [0] Failed to fetch `
https://tile.openstreetmap.de/14/8624/5842.png'
Traceback (most recent call last):
  File "/usr/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.9/http/client.py", line 1417, in connect
    super().connect()
  File "/usr/lib/python3.9/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.9/socket.py", line 822, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 546, in fetch_url
    data = urllib.request.urlopen(req)
  File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name
resolution>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 501, in fetch
    self.fetch_url(url, self._local_path())
  File "/home/riggs498/D-Rats/d_rats/map/maptile.py", line 554, in fetch_url
    raise MapFetchError(err)
d_rats.map.maptile.MapFetchError: <urlopen error [Errno -3] Temporary
failure in name resolution>
munmap_chunk(): invalid pointer
Aborted

[wb8tyw]

Similar issues are all through D-rats where it it was handling broad exceptions instead of specific exceptions.

In the jobs I have worked at, it is considered bad practice to have either "except: or except Exception:" as it indicates that a module has not been properly tested, and that practice can make a programming bug harder to find.

As I do not yet have a unit test suite that simulates all code paths, I need these type of reports to to help fix all the exception handling.

[wb8tyw]

This is actually two issues and the PR 114 only addresses one of them.
The PR addresses removing the extra stack dumps for a class of expected exceptions.

The other problem is a bit harder to fix, that is that there were so many map url requests in progress that were failing that the program ran out of resources.

The code in d-rats launches a new thread for each tile fetch request, with no queuing done to limit the number that can be in flight. This would need to be re-written to have a queue of tile requests so that only a limited number of parallel threads would be active at a time.

Then feature creep sets in to optimize the queue to remove duplicate fetch requests or fetch requests that are not needed any more because the zoom level changed.

[wb8tyw]

Opened a new ticket for the potential throttling of map requests.