-
Notifications
You must be signed in to change notification settings - Fork 142
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Preload fails with "error initializing graphdriver: driver not supported" #1099
Comments
This issue is caused by the removal of the AUFS storage driver ( Known Workarounds Last edit: Nov 2021 Use Linux (e.g. Ubuntu) in a virtual machine like Ubuntu Multipass or VirtualBox, and install the balena CLI for Linux in the virtual machine. When doing so, do not place the OS image file (to be preloaded) in "shared folders" (network mounted filesystem) because the Back in 2019, another workaround was to downgrade to Docker Community Edition 18.06.1-ce-mac73 2018-08-29 (which ships with Docker 18.06.1-ce). However, those older versions of Docker are now "really old" and offer poor support for modern features like multiarch images, so many users will now prefer to use a Linux VM. Proper Solution Balena is working towards replacing AUFS with overlay2 in affected balenaOS images. Newer device types like the Raspberry Pi 4 already use overlay2. For other device types, the difficulty currently being addressed is the host OS update of devices in the field running old balenaOS releases that still use AUFS, and need to be migrated to overlay2 as part of the host OS update. |
Thanks @pdcastro! The workaround is sufficient for me. |
This issue has been reported for Docker for Windows as well. In the case of Docker Community Edition for Windows, the latest version to include AUFS support is 18.06.1-ce-win73 2018-08-29. |
Hi, I also faced same issue and it was works that workaround (downgrade docker version). But when I setup preload image to another .img, it was occured this error. |
Are there any other workarounds that don't require a downgrade of Docker? |
Unfortunately the only other option I am aware of is running Linux -- for example using a VirtualBox virtual machine. Docker for Linux still supports the AUFS driver. |
To clarify, this is because we still use AUFS on a bunch of balenaOS devices. Preloading OS images that have the engine with overlay2 used for containers should not result in this problem. And as soon as we migrate from aufs to overlay2 this will be resolved. |
[roman-mazur] This issue has attached support thread https://jel.ly.fish/#/9df1e32a-b9e1-4fee-a48a-fb185ed96279 |
[jimsynz] This issue has attached support thread https://jel.ly.fish/#/5be791bd-dce4-44d0-a69f-5c51b414295f |
Do we have an update on this issue? Is the only way to downgrade docker? |
The only other option I am aware of is running Linux -- for example using a VirtualBox virtual machine, or AWS instance, or local Linux server. Docker for Linux still supports the AUFS driver. |
[pdcastro] This issue has attached support thread https://jel.ly.fish/ce5dab94-a799-4087-b29f-b9cd3171cbae |
@pdcastro I've just encountered this issue here too. On latest macOS and latest Docker Desktop, the CLI preload command just hangs at |
@chrisys, @kenna-smith, @sazerzac, @erlend, @briggySmalls, (cc: @klutchell) thanks for the feedback. FYI, the following CLI releases improved on error reporting such that the error is caught and reported straight away:
The latest CLI release has the best error handling capabilities. @shawaj, thanks for the heads up on the broken links for the old Docker releases. 👍 I'll fix earlier comments of mine. The ultimate solution to this issue is to replace AUFS with overlay2 in affected balenaOS images, and balena is making good progress in this direction. I understand that the difficulty currently being addressed is the host OS update of devices in the field where installed balenaOS images use AUFS and the new balenaOS images use overlay2. The image will be converted as part of the host OS update. |
I've edited this issue's title because it is not specific to macOS: It also affects Windows and, in some cases, Linux. When the host OS (the machine where the CLI is running) is Linux,
|
[klutchell] This issue has attached support thread https://jel.ly.fish/7254451a-33de-4e6f-94fd-abbe944cb6e8 |
[klutchell] This issue has attached support thread https://jel.ly.fish/33963003-cc34-45f6-b8a5-9511b15cd082 |
Just checking in on this issue. Is using docker |
@panbanda, thanks for asking. I have now updated an earlier comment to advise using a Linux VM and the balena CLI for Linux instead of downgrading Docker, because those older versions of Docker are getting just too old. Have a look at the updated comment for some additional details. Balena is still working towards replacing AUFS with overlay2 in affected balenaOS images, and the tricky process of reliably migrating devices in the field from AUFS to overlay2 as part of host OS updates. Progress is a bit slow but it is critical that the implementation is done right, so it is not something being rushed. |
I wonder if something recently changed in Docker. I have come across this issue to day on GitHub workflows, running on a linux ubuntu machine. Downgrading to Docker 18 isn't really an option. I also build images on one workflow that work with overlay and some that don't. If I try a solution like those seen on StackOverflow you shared, does that impact others? Sounds like the proper fix is in the pipeline, but not that close. Would be good to get a clearer idea of a workaround to follow in the meantime. |
Looks like GitHub runner dropped support for aufs. Not sure if we can treat it as a GitHub issue though, is very outdated it seems. https://forums.balena.io/t/error-preloading-balenaos/349969/4 |
[alexgg] This issue has attached support thread https://jel.ly.fish/c3525c4d-652f-46fb-8360-66b394880496 |
Some info here from the forums that is super helpful:
|
That's great info indeed @Maggie0002 @alexgg. 🎉 So I think it can be said that the ultimate "fix" to this |
2.84.0 on the boards in that list it seems. Other boards not on that list are below 2.84.0 but seem to work ok. So if using one of the boards on the list, then will need to wait for 2.84.0. I had a brief play around with the things from the stack overflow posts you shared. Most of the packages it suggested installing are deprecated and no longer available in the Ubuntu repositories so didn’t get far. |
This is great news! Is there a public kanban or something to view the progress / timelines of the different boards? I'm really only needing the rpi0 but I'm sure others would be interested as well. |
Or at least a tracking issue? Is this issue or the forum one (https://forums.balena.io/t/error-preloading-balenaos/349969) going to be updated when each of the new OS versions are released? I'm assuming it will be relatively soon as there isn't really support for those devices and preload in the meantime? |
What I got from the OS team is that: "The latest beaglebone-black release already performs the migration [AUFS → overlay2]. We are waiting on a decent uptake of BBB fleets to perform the migration [i.e. a large-ish number of devices in the field to have migrated] so that we catch any pending problem before updating bigger fleets." To easily find out what the latest balenaOS version for a device type is, the $ DEVICE_TYPES=(beaglebone-black intel-edison intel-nuc odroid-c1 raspberry-pi raspberry-pi2 raspberrypi3 raspberrypi3-64 fincm3 revpi-core-3 npe-x500-m3 up-board)
$ for dt in "${DEVICE_TYPES[@]}"; do echo "$(balena os versions "${dt}" | head -n 1) ${dt}"; done
v2.85.16+rev1.prod (recommended) beaglebone-black
v2.31.5+rev1.prod (recommended) intel-edison
v2.83.18+rev1.prod (recommended) intel-nuc
v2.38.0+rev1.prod (recommended) odroid-c1
v2.83.21+rev1.prod (recommended) raspberry-pi
v2.83.21+rev1.prod (recommended) raspberry-pi2
v2.83.21+rev1.prod (recommended) raspberrypi3
v2.80.3+rev1.prod (recommended) raspberrypi3-64
v2.83.21+rev1.prod (recommended) fincm3
v2.80.3+rev1.prod (recommended) revpi-core-3
v2.58.3+rev1.prod (recommended) npe-x500-m3
v2.68.1+rev1.prod (recommended) up-board @Maggie0002, regarding "there isn't really support for those devices and preload in the meantime", do you mean when using GitHub workflows specifically? The old workaround of using a Linux Virtual Machine (e.g. with Ubuntu) to run |
I was seeing GitHub Workflows as largely the same thing. It’s a Ubuntu virtual machine for running processes. It seemed like the most recent breaking change was to do with Docker dropping support for AUFs. So a virtual machine would have to roll back to an old Docker for it to still work. Somewhat viable, but the workarounds are narrowing quite quickly. There is some more info here: https://forums.balena.io/t/error-preloading-balenaos/349969/4?u=maggie Thanks for the idea on the script to get the latest images. I’m not entirely sure it’s practical for me to run it everyday, a tracking issue would be better if possible. At the moment I have just decided to drop support for those devices and fully appreciate the stability steps being taken that can cause a delay. It would be helpful though to know when I can restore support for those devices without having to check everyday. I understand there is the potential to set up a custom vm with a rolled back Docker and plug that in as a runner for GitHub or use it manually, or maybe explore downgrading the GitHub runner Docker, or maybe the Ubuntu repository default has an old Docker that could work and could setup some sort of manual workflow. But that’s an awful lot of work I’m going to try and avoid. I would rather just drop the support for it and then know as soon as I can offer it again, assuming it isn’t a long way in the future. |
I understand that the latest stable version of Ubuntu (20.04.3) still supports AUFS. The following test has confirmed it, and it is also stated on Docker's Installation Instructions for Ubuntu:
I like the convenience of using Multipass to create text mode (non-graphical) Ubuntu virtual machines. The following commands achieve the preloading of a Raspberry Pi 3 image (using AUFS), all without leaving a command line prompt and therefore potentially scriptable:
I tested the commands above on a Mac laptop. |
Another approach here from a user, adding for reference. I haven't tried it though, personally still assuming the OS updates won't be too far away and better to wait than redo the workflows: https://forums.balena.io/t/balena-aufs-preload-on-github-actions/350526 |
[fisehara] This issue has attached support thread https://jel.ly.fish/c4d4a45d-974c-4226-b607-91bd711b7f0b |
First off, the preloader is an awesome tool and I really appreciate it!
I have used it successfully in a Linux environment, but under macOS it fails (using the Intel NUC base image https://files.resin.io/resinos/intel-nuc/2.29.2%2Brev2.prod/image/balena.img.zip)
Thanks for your help!
Front conversations
The text was updated successfully, but these errors were encountered: