Trying to install but getting error 'No such image: ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1'

[townsmcp]

I have been using the addin for a long time on HA and it has been working very well. I went to upgrade today to 0.112.1 however it kept erroring on me (same error as below). I then uninstalled the version I was running, removed the repo from the list, restarted HA, added the repo back and went to try installing again. However now I dont have the addin at all and still get the following error:

Can't install ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1/json: Not Found ("No such image: ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1")

I am running:
Core - 2023.12.0b1
Supervisor - 2023.11.6
Operating System - 11.1
Frontend - 20231130.0

[townsmcp]

@sabeechen any suggestions/ideas? I am now running:
Core - 2023.12.1
Supervisor - 2023.11.6
Operating System - 11.1
Frontend - 20231208.2

Still getting the same error message when trying to install from the repo (https://github.com/sabeechen/hassio-google-drive-backup):

Can't install ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1/json: Not Found ("No such image: ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1")

Thanks

[sabeechen]

I've had people periodically report this error for the addon. In the latest verison I switched away from using dockerhub to store the addon image and its now stored on github (ghcr.io). I was hoping it would solve this issue, but apparently it didn't help.

For some context, this error is coming from Home Assistant's Supervisor when it attempts to download the addon from GitHub. Its strange to get a 404 error, which means "Not found" because I'm 100% positive the image is there and accessible. This is where the error is being thrown in the supervisor.

~$ docker pull  ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1
0.112.1: Pulling from sabeechen/hassio-google-drive-backup-amd64
8a49fdb3b6a5: Already exists
e50d3cb6eeb0: Already exists
866b73a2dc62: Already exists
7d318b991f87: Pull complete
d2b09d5194f3: Pull complete
919dfa8aa335: Pull complete
329f930305ca: Pull complete
f02fbc9829ae: Pull complete
f8f41d906cc8: Pull complete
Digest: sha256:9eee185e3e468c2080a79c0ba333a194fb182bf795247e3bca9832c74809439c
Status: Downloaded newer image for ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1
ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1

I've had no success trying to figure out why this happens for some users, and I've never been able to reproduce it on one of my test machines. I've spent hours trying. It seems to just go away sometimes by itself, and sometimes gets resolved by restarting the host machine.

Home Assistant's supervisor uses the docker-py library under the hood to make docker requests, and I just ran into this bug report suggesting that this error can be caused by running out of space. Is it possible your machine is low on disk?

[townsmcp]

Thanks for the response @sabeechen.
I have rebooted the VM and host a fair bit over the past couple of days due to Proxmox upgrades, sadly it hasn’t fixed the issue.
The VM has about 40gb space left (of 100gb) and the host drive has 300gb free so space should be an issue.
I do have an HA OS that I haven’t upgraded to yet that is showing in HA (11.2). I will see if I can take a VM snapshot and then try the new HA OS and report back

[townsmcp]

Sadly pushing the OS from 11.1 to 11.2 hasn’t worked

[sabeechen]

I suspect the issue is actually with docker-py, since at least some of the errors it runs in to come back miscategorized as "404". The problem is I'm not just trying to sort out a problem outside the addon, its on someone elses machine in a dependent library of a project I don't have any control over. I'd make a new issue for it in the supervisor, but I'm at a loss for how to reproduce it.

You open to some debugging? If I were you I'd do the following:

• Uninstall the addon (if it shows up as installed) and remove the addon's repository.
• Reboot HA's machine (or its VM if you're virtualized)
• Get a shell into ha (you can use the SSH addon if you don't have keyboard/monitor access) and run ha su repair. This does some cleanup of Home Assistant's docker environment.
• Re-add the addon repository and try to reinstall the addon.

[townsmcp]

Sure. I will give that a go to see if the repair works from the OS. Thankfully before upgrading/moving to 112 I selected to backup the addon which I restored tonight.

[townsmcp]

:(
I followed those steps to the letter (su repair took about 5 minutes), added the repo back in after VM reboot and sadly still get the same error message:

Can't install ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1: 404 Client Error for http+docker://localhost/v1.43/images/ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1/json: Not Found ("No such image: ghcr.io/sabeechen/hassio-google-drive-backup-amd64:0.112.1")

Just a thought, I know why you have switched the repo to GitHub, but any chance the Docker Hub repo could be updated and made available with 112 for testing? I appreciate you wouldn't want to maintain 2 repos going forward but it might help rule out the repo location?

[sabeechen]

sadly still get the same error message:

Bummer. This problem has been around for years, I've never been able to crack it. This addon has over 100k users, and this has been reported at least 20 times.

Just a thought, I know why you have switched the repo to GitHub, but any chance the Docker Hub repo could be updated and made available with 112 for testing?

We can try it in a bit of a roundabout way, I can have you install the previous version (v0.111.1) as a "Local Addon" to see if the old repo on dockerhub works. Instructions are below, this is the process someone would use if they wanted to develop an addon:

• Install the Samba addon, or any other addon you use to get access to Home Assistant's folders (eg /ssl, /config, /share, /backups, /media, /addons). You'll need to be able to make files in the "addon" folder.
• Inside the "/addon" folder, create a new folder named "test_addon"
• Add the attached file (config.json) to that folder.
• Go to System > Addons and stop the existing "Home Assistant Google Drive Backup" addon
• Go to "Add-on Store" then "check for updates"
• Refresh the page (a browser refresh)
• Now you should see a new addon at the top of the addon store, named "Local Add-ons" > "Old HAGDB". Install it.

If you're able to install it, then we know dockerhub is working for you. I'm not sure what I'd do if I knew it worked, but its another bit of evidence.

[townsmcp]

That worked - installed all ok and I had to tweak the port so the Old HAGDB add-on would start, but I increment the port (1628) and it started up fine - in the UI it shows the 'Get Started' message

[bedo2405]

Any luck fixing this? I have the same issue :(

[townsmcp]

@sabeechen because that worked for dockerhub install of the old version, is there much overhead to maintain both GitHub and DockerHub?

[sabeechen]

Sorry it took me a bit to get back to you, I've had trouble finding time.

I switched from DockerHub to GitHub because this problem was already present when using Dockerhub. I was hoping it was only a problem with Dockerhub, that isn't the root of the issue. Switching back would just cause the problem for another (seemingly) random subset of users.

I could publish this addon in parallel to multiple repos but it would be ... a bit difficult to maintain if this problem is ever resolved. The issue doesn't seem to be related to either repo. With every update, some number of people are unable to download it.

I hadn't realized until now, but Home Assistant gives you access to its docker environment through the ssh addon, and that might give us a clue as to whats going wrong. Could I have you do the following?

1. Make sure you have a good backup. Nothing I'm having you do here should cause problems, but you can never be sure.
2. If you already have the backup addon installed, uninstall it. Don't uninstall the backup addon's repository.
3. Install the SSH addon, and in particular make sure "Protection Mode" is disabled on the SSH addon's installation page.
4. In the SSH Addon, run these commands:

   docker container ls | grep google-drive-backup
   docker image ls | grep google-drive-backup

   Both commands should print out nothing. If they do print something, make sure you uninstalled the backup addon. If they still print something, then please stop and report back because thats unexpected.
   Next, manually download the backup addon's image from the command line:

   docker pull ghcr.io/sabeechen/hassio-google-drive-backup-$(ha su info | grep arch | sed "s/arch: //"):0.112.1

   This is the part that Home Assistant is failing on when you try to install, so if this command also fails please let me know what the error message is. It might pinpoint what the problem is.
5. If it instead succeeds, thats unexpected. Try going to the Add-on Store in Home Assistant and installing the backup addon. It should complete quickly because the image is already downloaded. If it still fails, please also check the Supervisor's logs for any additional error messages.

If someone is able to do this for me, it might provide the information I need to figure out whats wrong. If any part of those instructions are unclear, let me know and I can elaborate. Right now I basically have nothing to work off of, its just been a mysterious problem around for years that I've never been abel to reproduce. Is suspect it affects a very small number of people, since I've had it reported maybe 20 times and have over 100k unique users.

[townsmcp]

@sabeechen no worries, thank you for looking into this.
Wow, 20 times out of 100k+ is pretty low.
I know there used to be a Portainer add-on that could be installed but that no longer works :(

So I ran through the steps, nothing printed out with the first 2 command, however when running the 3rd command to pull the image etc it did fail:

Error message:
failed to register layer: rename /mnt/data/docker/image/overlay2/layerdb/tmp/write-set-2607581605 /mnt/data/docker/image/overlay2/layerdb/sha256/54df8cd599689a13b510c71e917bcabe0cc900e1ec36a56dcb50b372bc6a9987: file exists.
As you can see, the line says to rename a file. I am guessing as it is in a /tmp/ directory that should be ok? I haven't done that but happy to give it a try if you want?

If I do a docker images I get the following:

I am assuming the 'IMAGE ID' column is truncated, and I can't see any image ID starting with 54df.........

[sabeechen]

Well that explains it. Looks like whats happening is:

1. Something happens in the past that corrupts docker's downloaded cache files, it looks like this most commonly happens when a docker operation is interrupted by something it can't handle (power loss, machine crash, no more disk space, etc)
2. The library Home Assistant uses to interact with docker misreports many errors it doesn't understand as "404 Client Error"
3. Confusion has persisted for years because of all the layers of indirection.

For the curious, there is an interesting discussion about why this is such a tricky problem here. Its made more tricky because Home Assistant goes out of its way to keep you from having ssh access to the host (ssh addon runs in a docker container too, for example), and that broken directory is only on the host. To my knowledge, this is the only way to get Host SSH Access, and its honestly probably simpler to just reinstall. You aren't able to directly delete anything in /mnt/data/docker/image/overlay2/layerdb/sha256/54df8cd599689a13b510c71e917bcabe0cc900e1ec36a56dcb50b372bc6a9987 from inside the ssh addon. Even if you could, there is no guarantee it won't make things worse, as another image could be using the image layer stored there.

To fix this, I think the easiest course of action available is to reinstall Home Assistant. If you're feeling adventurous and willing to risk currupting your install further, I'd be curious to know what happens if you enable host SSH and delete that directory.

I'm going to create a writeup for this and make an issue on the supervisor github to see if they have a better recommendation, they're much more familiar with docker's internals than I am.

[townsmcp]

Because I have my HA instance in Proxmox, I think I can access the host easily from the terminal so might give that a go. I have a full backup and a complete rebuild of HA would be a royal pain in the butt

[sabeechen]

It depends on how you installed in proxmox. If you installed a flavor of Home Assistant OS (which is typical) then I believe the proxmox console will just be the reduced privileged 'ha' command line. If you installed the supervisor manually you will get a real shell. I do the latter on my home's instance, because I'm philosophically opposed to having a linux machine on my network without root.

[sabeechen]

It looks like this issue and its root cause are already pretty well reported in the HA supervisor. I could have saved us a lot of debugging if I'd looked at the HA supervisor issues made in recent months. Whats worse, this doesn't just affect addons, it can come up when updating anything with Docker including Home Assistant and the supervisor itself, rendering an install very broken. I don't see any direct indication of a fix in the works, though things may be happening behind the scenes.

Unfortunately I don't find that surprising, I'm not really sure myself how this could be fixed except perhaps to provide a clearer error message. Current workaround seems to be to reinstall from a backup if this happens to you. Its not great.

If you're able to get real ssh access and try deleting that directory, I don't think you have much to lose so long as you have a good backup.

[townsmcp]

Thanks for looking into it.
Restore from backup isn’t an option.
Rebuild HA is an option, kind of. But right at the bottom of the list lol.
Over the next few days I will give it a go trying to delete that directory and report back

[townsmcp]

@sabeechen so bit of an update. I sorted out the SSH keys today (didn't want to risk blowing up HA with the family home from school). Managed to SSH in to HA with root. I have:

• Uninstalled the 111 plugin
• SSH'd back in to Hassio and rerun the commend from above just to make sure the path hadn't changed (see pic below)
• The SSH'd into HA root and rm -r /mnt/data/docker/image/overlay2/layerdb/sha256/54df8cd599689a13b510c71e917bcabe0cc900e1ec36a56dcb50b372bc6a9987.
• SSH'd back in to the HA add-on (so I am back in Hassio rather than root)
• Run the command docker pull ghcr.io/sabeechen/hassio-google-drive-backup-$(ha su info | grep arch | sed "s/arch: //"):0.112.1
• I then went back to Settings -> Addons and didn't see the Google Drive app so I went to the add-on store and installed.
• The only thing I have to do now, is setup the access to Google Drive and add my settings back as effectively it is a clean new install