Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removal In Progress #4568

Closed
BrixSat opened this issue Sep 19, 2023 · 4 comments
Closed

Removal In Progress #4568

BrixSat opened this issue Sep 19, 2023 · 4 comments

Comments

@BrixSat
Copy link

BrixSat commented Sep 19, 2023

Describe the issue you are experiencing

Hi,

I have a popup that asks to update (current version 2023.9.0) to version 2023.9.2. If i press the update i have docker in the next state for hours. "Removal In Progress"

Tried everything and nothing
docker kill containter_id
docker system prune -a
docker rm container_id
only reboot saves me

I dont know that to do.
edd215993794 ghcr.io/home-assistant/generic-x86-64-homeassistant:2023.9.0 "/init" 6 minutes ago Removal In Progress homeassistant

What type of installation are you running?

Home Assistant OS

Which operating system are you running on?

Home Assistant Operating System

Steps to reproduce the issue

click the upgrade version on home assistant frontent

Anything in the Supervisor logs that might be useful for us?

2023-09-19 10:02:51.714 WARNING (SyncWorker_4) [homeassistant.loader] We found a custom integration mikrotik_router which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-09-19 10:02:51.717 WARNING (SyncWorker_4) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-09-19 10:02:51.719 WARNING (SyncWorker_4) [homeassistant.loader] We found a custom integration ham_radio_propagation which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-09-19 10:02:51.722 WARNING (SyncWorker_4) [homeassistant.loader] We found a custom integration midea_ac_lan which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-09-19 10:03:26.043 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry A minha casa for smartthings
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 387, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/smartthings/__init__.py", line 126, in async_setup_entry
    token = await api.generate_tokens(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pysmartthings/smartthings.py", line 209, in generate_tokens
    result = await self._service.generate_tokens(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pysmartthings/api.py", line 424, in generate_tokens
    raise APIInvalidGrant(data.get("error_description"))
pysmartthings.errors.APIInvalidGrant: Invalid refresh token: 16541dcb-30ce-4516-900d-e83d2c67d4c4 requested by clientId: 04166bfe-53f7-4b38-96e9-8bc7be6bf763 with grantType: refresh_token, e.cause: no cause
2023-09-19 10:03:26.675 ERROR (MainThread) [homeassistant.components.media_player] Error while setting up xiaomi_tv platform for media_player
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 415, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.11/http/client.py", line 1286, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1332, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1281, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1041, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 979, in send
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1293722e10>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.10.2', port=6095): Max retries exceeded with url: /request?action=isalive (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1293722e10>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 359, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/xiaomi_tv/media_player.py", line 55, in setup_platform
    add_entities(XiaomiTV(tv, DEFAULT_NAME) for tv in pymitv.Discover().scan())
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymitv/discover.py", line 33, in scan
    if self.check_ip(ip_check):
       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pymitv/discover.py", line 51, in check_ip
    request = requests.get(tv_url, timeout=request_timeout)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.10.2', port=6095): Max retries exceeded with url: /request?action=isalive (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1293722e10>: Failed to establish a new connection: [Errno 111] Connection refused'))
2023-09-19 10:03:27.031 ERROR (MainThread) [homeassistant.components.sensor] Error adding entities for domain sensor with platform speedtestdotnet
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 591, in state
    numerical_value = float(value)  # type:ignore[arg-type]
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: 'unavailable'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 507, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 752, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1002, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 730, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 830, in _async_write_ha_state
    state, attr = self._async_generate_attributes()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 771, in _async_generate_attributes
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 736, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 593, in state
    raise ValueError(
ValueError: Sensor sensor.speedtest_ping has device class 'None', state class 'measurement' unit 'ms' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2023-09-19 10:03:27.061 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up speedtestdotnet platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 591, in state
    numerical_value = float(value)  # type:ignore[arg-type]
                      ^^^^^^^^^^^^
ValueError: could not convert string to float: 'unavailable'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 367, in _async_setup_platform
    await asyncio.gather(*pending)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 507, in async_add_entities
    await asyncio.gather(*tasks)
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 752, in _async_add_entity
    await entity.add_to_platform_finish()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1002, in add_to_platform_finish
    self.async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 730, in async_write_ha_state
    self._async_write_ha_state()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 830, in _async_write_ha_state
    state, attr = self._async_generate_attributes()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 771, in _async_generate_attributes
    state = self._stringify_state(available)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 736, in _stringify_state
    if (state := self.state) is None:
                 ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/sensor/__init__.py", line 593, in state
    raise ValueError(
ValueError: Sensor sensor.speedtest_ping has device class 'None', state class 'measurement' unit 'ms' and suggested precision 'None' thus indicating it has a numeric value; however, it has the non-numeric value: 'unavailable' (<class 'str'>)
2023-09-19 10:03:37.244 ERROR (SyncWorker_1) [custom_components.mikrotik_router.mikrotik_controller] No kid control devices found on your Mikrotik device, make sure kid-control feature is configured
2023-09-19 10:03:37.336 ERROR (MainThread) [homeassistant.components.sensor] mikrotik_router: Not adding entity with invalid device info: Invalid device info {'configuration_url': 'http://192.168.10.1', 'connections': {('mikrotik_router', 'F0380E65C8B9')}, 'default_manufacturer': 'MikroTik', 'default_model': 'RB4011iGS+', 'default_name': 'Mikrotik RB4011iGS+', 'identifiers': {('mikrotik_router', 'F0380E65C8B9')}, 'sw_version': '7.11.2 (stable)', 'via_device': ('mikrotik_router', 'F0380E65C8B9')} for 'mikrotik_router' config entry: device info needs to either describe a device, link to existing device or provide extra information.

System Health information

System Information

version core-2023.9.0
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.11.5
os_name Linux
os_version 6.1.45
arch x86_64
timezone Europe/Lisbon
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
GitHub API Calls Remaining 5000
Installed Version 1.32.1
Stage running
Available Repositories 1285
Downloaded Repositories 12
Home Assistant Cloud
logged_in false
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 10.5
update_channel stable
supervisor_version supervisor-2023.09.2
agent_version 1.5.1
docker_version 23.0.6
disk_total 116.7 GB
disk_used 12.7 GB
healthy true
supported true
board generic-x86-64
supervisor_api ok
version_api ok
installed_addons File editor (5.6.0), MariaDB (2.6.1), Zigbee2MQTT (1.33.0-1), Node-RED (14.5.0), Mosquitto broker (6.3.1), Home Assistant Google Drive Backup (0.111.1), Cloudflared (4.2.6), Run On Startup.d (0.11a), Advanced SSH & Web Terminal (15.0.8)
Instituto Português do Mar e Atmosfera (IPMA)
api_endpoint_reachable ok
Dashboards
dashboards 1
resources 7
views 9
mode storage
Recorder
oldest_recorder_run August 20, 2023 at 5:36 PM
current_recorder_run September 19, 2023 at 10:02 AM
estimated_db_size 658.67 MiB
database_engine mysql
database_version 10.6.12
Spotify
api_endpoint_reachable ok

Supervisor diagnostics

There was an error creating your Issue: body is too long (maximum is 65536 characters).

{
"home_assistant": {
"installation_type": "Home Assistant OS",
"version": "2023.9.0",
"dev": false,
"hassio": true,
"virtualenv": false,
"python_version": "3.11.5",
"docker": true,
"arch": "x86_64",
"timezone": "Europe/Lisbon",
"os_name": "Linux",
"os_version": "6.1.45",
"supervisor": "2023.09.2",
"host_os": "Home Assistant OS 10.5",
"docker_version": "23.0.6",
"chassis": "embedded",
"run_as_root": true
},

Additional information

No response

@BrixSat BrixSat added the bug label Sep 19, 2023
@BrixSat
Copy link
Author

BrixSat commented Sep 20, 2023

How can i get access to docker logs (not the containers but the daemon it self) ?

@BrixSat
Copy link
Author

BrixSat commented Sep 21, 2023

I was able to access the OS and check some docker logs files of the daemon. I now have 8 containers in "Removal In Progress". I even tried rm -rf the merge folder (wich is empty), but got something using it. Tried also the lsof to see who was using it, and bunch of pids tied to that dead zone.

Sep 21 09:47:21 homeassistant dockerd[222952]: time="2023-09-21T09:47:21.037908669Z" level=error msg="Handler for DELETE /v1.42/containers/8633774e12f241dc2caf972676547e12ec7eced697a6c98e2aea00a35f5527bc returned error: container 8633774e12f241dc2caf972676547e12ec7eced697a6c98e2aea00a35f5527bc: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/93fd5bd77c1bbe8537da92979950e331b92c15bbf894840f55ee47bf66a16bde/merged: device or resource busy"
Sep 21 09:47:21 homeassistant dockerd[222952]: time="2023-09-21T09:47:21.182866610Z" level=error msg="Error removing mounted layer b5797f66e328f1637ae4a075b655537cbb056dddfb89c0ec7fd410e4c671ead5: unlinkat /mnt/data/docker/overlay2/1fb259d0606f1d4f487badb5d58d01c2c1fdf352545e31532f19ee0f6eccc5bd/merged: device or resource busy"
Sep 21 09:47:21 homeassistant dockerd[222952]: time="2023-09-21T09:47:21.183207389Z" level=error msg="Handler for DELETE /v1.42/containers/b5797f66e328f1637ae4a075b655537cbb056dddfb89c0ec7fd410e4c671ead5 returned error: container b5797f66e328f1637ae4a075b655537cbb056dddfb89c0ec7fd410e4c671ead5: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/1fb259d0606f1d4f487badb5d58d01c2c1fdf352545e31532f19ee0f6eccc5bd/merged: device or resource busy"
Sep 21 09:47:21 homeassistant dockerd[222952]: time="2023-09-21T09:47:21.220859179Z" level=error msg="Error removing mounted layer b833dbb5461b59b4ddc1ef4d4f75ce06e19a0e373ccb6f593c8138b7e602860e: unlinkat /mnt/data/docker/overlay2/34459057732e1ea3b6f2f9d41854440cba791bf1a53a7fbd5f80e7a58b4dc22e/merged: device or resource busy"
Sep 21 09:47:21 homeassistant dockerd[222952]: time="2023-09-21T09:47:21.221629116Z" level=error msg="Handler for DELETE /v1.42/containers/b833dbb5461b59b4ddc1ef4d4f75ce06e19a0e373ccb6f593c8138b7e602860e returned error: container b833dbb5461b59b4ddc1ef4d4f75ce06e19a0e373ccb6f593c8138b7e602860e: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/34459057732e1ea3b6f2f9d41854440cba791bf1a53a7fbd5f80e7a58b4dc22e/merged: device or resource busy"
Sep 21 09:47:22 homeassistant dockerd[222952]: time="2023-09-21T09:47:22.073926958Z" level=error msg="Error removing mounted layer fa381e6ff20db2bacaf7892e20beaf8234aabd07c10f1af3190de8c9de32e3b3: unlinkat /mnt/data/docker/overlay2/b32ccb5e13b9714af090d673b0dd0e9bc0cfa3116504e7c8b897735feb78f400/merged: device or resource busy"
Sep 21 09:47:22 homeassistant dockerd[222952]: time="2023-09-21T09:47:22.075017790Z" level=error msg="Handler for DELETE /v1.42/containers/fa381e6ff20db2bacaf7892e20beaf8234aabd07c10f1af3190de8c9de32e3b3 returned error: container fa381e6ff20db2bacaf7892e20beaf8234aabd07c10f1af3190de8c9de32e3b3: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/b32ccb5e13b9714af090d673b0dd0e9bc0cfa3116504e7c8b897735feb78f400/merged: device or resource busy"
Sep 21 09:47:25 homeassistant dockerd[222952]: time="2023-09-21T09:47:25.097509529Z" level=error msg="Error removing mounted layer 029c2d93b851d225acd92646f03d43deb7bc22ed98b8138963ab8ac181c54499: unlinkat /mnt/data/docker/overlay2/7f0c3ce546c6454c42751812e36d888dab93d4fc60c62ae6c2043bc8115166a7/merged: device or resource busy"
Sep 21 09:47:25 homeassistant dockerd[222952]: time="2023-09-21T09:47:25.097875664Z" level=error msg="Handler for DELETE /v1.42/containers/029c2d93b851d225acd92646f03d43deb7bc22ed98b8138963ab8ac181c54499 returned error: container 029c2d93b851d225acd92646f03d43deb7bc22ed98b8138963ab8ac181c54499: driver "overlay2" failed to remove root filesystem: unlinkat /mnt/data/docker/overlay2/7f0c3ce546c6454c42751812e36d888dab93d4fc60c62ae6c2043bc8115166a7/merged: device or resource busy"

@BrixSat
Copy link
Author

BrixSat commented Sep 25, 2023

any help?

@github-actions
Copy link

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Oct 25, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 1, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Dec 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant