Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MinerConnectionError when "helium-miner" container is down #399

Open
posterzh opened this issue Jul 5, 2022 · 1 comment
Open

MinerConnectionError when "helium-miner" container is down #399

posterzh opened this issue Jul 5, 2022 · 1 comment
Labels
B2 device stability device sentry

Comments

@posterzh
Copy link
Contributor

posterzh commented Jul 5, 2022

Follow up to:

Thoughts:

  • For any miner, if helium-miner container is down, then one of the following actions makes MinerConnectionError.
    • open the diagnostics page.
    • call diagnostics/json endpoint
    • upload the diagnostics to the Google big query

For example, Balena device https://dashboard.balena-cloud.com/devices/cae984a6b29efe5198bc582744310fe9

  • stop the helium-miner container intentionally
    image

  • open the diagnostics web page
    image

  • MinerConnectionError is observed from the Balena logs for diagnostics container
    image

INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
DEBUG:flask_caching.backends.simplecache:set key 'view//json'
DEBUG:urllib3.connectionpool:https://o571444.ingest.sentry.io:443 "POST /api/5730184/envelope/ HTTP/1.1" 200 2
DEBUG:flask_caching.backends.simplecache:get key 'view//json' -> miss (expired)
INFO:root:Running periodic hardware diagnostics
2022-07-05 05:26:25,786 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(36) - gateway_mfr response stdout: b'{\n  "key": "112Jp9VrwUBQaVd2f2fhKJvxFm2uf2dVQSiQFpaLjrDJtMAqbb7c",\n  "name": "cool-brick-goblin"\n}\n'
INFO:hm_pyhelper.miner_param:gateway_mfr response stdout: b'{\n  "key": "112Jp9VrwUBQaVd2f2fhKJvxFm2uf2dVQSiQFpaLjrDJtMAqbb7c",\n  "name": "cool-brick-goblin"\n}\n'
2022-07-05 05:26:25,787 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(38) - gateway_mfr response stderr: b''
INFO:hm_pyhelper.miner_param:gateway_mfr response stderr: b''
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): helium-miner:4467
2022-07-05 05:26:25,823 - [ERROR] - hm_pyhelper.miner_param - (miner.py).fetch_miner_data -- /opt/python-dependencies/hw_diag/utilities/miner.py:(21) - Unable to connect to miner http://helium-miner:4467
Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/python-dependencies/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python-dependencies/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1276, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1322, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1271, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1031, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 969, in send
    self.connect()
  File "/opt/python-dependencies/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/python-dependencies/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/python-dependencies/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 27, in __fetch_data
    response = requests.post(self.url, json=req_body)
  File "/opt/python-dependencies/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/python-dependencies/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python-dependencies/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/utilities/miner.py", line 10, in fetch_miner_data
    peerbook = client.get_peer_book()[0]
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 62, in get_peer_book
    return self.__fetch_data('peer_book', addr='self')
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 29, in __fetch_data
    raise MinerConnectionError(
hm_pyhelper.miner_json_rpc.exceptions.MinerConnectionError: Unable to connect to miner http://helium-miner:4467
ERROR:hm_pyhelper.miner_param:Unable to connect to miner http://helium-miner:4467
Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/python-dependencies/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python-dependencies/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1276, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1322, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1271, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1031, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 969, in send
    self.connect()
  File "/opt/python-dependencies/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/python-dependencies/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/python-dependencies/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 27, in __fetch_data
    response = requests.post(self.url, json=req_body)
  File "/opt/python-dependencies/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/python-dependencies/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python-dependencies/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/utilities/miner.py", line 10, in fetch_miner_data
    peerbook = client.get_peer_book()[0]
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 62, in get_peer_book
    return self.__fetch_data('peer_book', addr='self')
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 29, in __fetch_data
    raise MinerConnectionError(
hm_pyhelper.miner_json_rpc.exceptions.MinerConnectionError: Unable to connect to miner http://helium-miner:4467
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): fuzzy-marmalade-warlock.skittles.stakejoy.com:443
DEBUG:urllib3.connectionpool:https://fuzzy-marmalade-warlock.skittles.stakejoy.com:443 "GET /v1/blocks/height HTTP/1.1" 200 27
2022-07-05 05:26:26,271 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(79) - Retrieving list of BLE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of BLE device(s)
2022-07-05 05:26:26,286 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(99) - Found the following BLE Devices: [{'Address': '00:E9:3A:F9:D0:C2', 'Name': 'cae984a', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
INFO:hw_diag.utilities.hardware:Found the following BLE Devices: [{'Address': '00:E9:3A:F9:D0:C2', 'Name': 'cae984a', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
2022-07-05 05:26:26,287 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(145) - Retrieving list of LTE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of LTE device(s)
2022-07-05 05:26:26,297 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(172) - Found the following LTE Devices: []
INFO:hw_diag.utilities.hardware:Found the following LTE Devices: []
INFO:root:Frequency: EU868
@posterzh
Copy link
Contributor Author

posterzh commented Jul 5, 2022

The reason of this issue is most likely the corrupted EMMC and the following actions can fix:

  • Purge the device
  • Reflash the EMMC
  • Replace the EMMC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
B2 device stability device sentry
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@posterzh