Cannot install latest hassos on ODROID-C2 #1202

elpiaffo · 2021-02-03T12:30:17Z

Hardware Environment

Raspberry Pi [1/2/3/4]
ODROID [C2/C4/N2(+)/XU4]
ASUS Tinker [S]
Intel NUC (or compatible)
OVA (Open Virtualization Applicance, on Intel NUC or any other hardware, please add the Hypervisor you are using)

Home Assistant OS release:

Fresh installation of release 5.08 - 5.11
Updated from version x.y
Additional information (if accessible):

Supervisor logs:

Journal logs:

Kernel logs:

Description of problem:

I've tried to setup latest hassos on forgotten ODROID-C2, but was unable to complete installation. Board is powered, Kernel is up, but Ethernet card cannot be found.
After initial boot messages, it start to spam these two lines in loop:

meson8b-dwmac c9410000.ethernet eth0: no phy at addr -1
meson8b-dwmac c9410000.ethernet eth0: stmmac_open: Cannot attach to PHY (error: -19)

I've tried with sdcard and with emmc, same situation. I found that there was issue similar to this, reported in #1142 apparently fixed on 5.10, but it seems that this is something different. Same behavior with 5.11.

Tried with 4.20 - no issue, installation does not hang and ethernet working as expected.

The text was updated successfully, but these errors were encountered:

agners · 2021-02-03T15:35:46Z

Hm, I haven't seen this issue so far, and C2 seems to work on my end. So this is probably device specific.

There was an issue with Ethernet and PHY timing (see #1042), but the fix is in 4.20 as well as in 5.10/5.11, so it seems that this patch does not cause or fix this problem.

Can you try with 5.5? It does not contain those fixes but is already on Linux 5.9.

I am intending to update to Linux 5.10 on the dev branch, that might be worth a try as well. I'll post once a development build is available.

elpiaffo · 2021-02-03T16:54:48Z

Can you try with 5.5? It does not contain those fixes but is already on Linux 5.9.

I am intending to update to Linux 5.10 on the dev branch, that might be worth a try as well. I'll post once a development build is available.

Tried to run installation with 5.5 but same issue occurred.

elpiaffo · 2021-02-03T18:07:15Z

I've tried upgrading that working 4.20 to latest, and it failed to boot afterwards (expected), with same error.
Board revision of this C2 is 0.2.20160226

oOoHoCo · 2021-02-09T23:16:28Z

I have the same problem since upgrading from 4.2 to anything higher on my c2
At the same time video out via hdmi introduced many artifacts. Mainly red noise in the picture. This too is not present in 4.2
I think i saw a commit entry stating that some frequencies were adjusted. Could maybe both issues be related?

carolinaxxxxx · 2021-02-10T10:13:42Z

I confirm. When upgrading to a version higher than 4.20:

meson8b-dwmac c9410000.ethernet eth0: no phy at addr -1
meson8b-dwmac c9410000.ethernet eth0: stmmac_open: Cannot attach to PHY (error: -19)

fantoine · 2021-02-20T21:27:57Z

Same error for me with a fresh install on version 5.12.

sylbouns · 2021-02-21T22:30:13Z

Hi, I tried 5.11 few days ago with same issue on my Odroid C2 and used 4.20 instead after reading @oOoHoCo comment. When I saw 5.12 coming, I decided to test it by using firmware OTA update from USB. It performs update and restart correctly. I could use HA locally few times on last version (no more update available notification in supervisor).

I stopped my Odroid and restart it next day with issue back...
I can't remember if I restart it between update and issue back, I believe I used Import config from USB to test something but can't be affirmativ.

I assumed that hassos-boot partition files could be in cause and switch them by 4.20 versions (copy/past on the only partition I could access on Windows) with no success.
Then I used a badass technic to capture HDMI stdout boot screen before kernel starts. Ethernet material error comes before boot.scr execution at first line :

Net: Could not get PHY for ethernet @c9410000: addr -1

Here is the capture, I hope it can help !

sylbouns · 2021-02-23T09:35:42Z

By reading these related issues and commit #1038, #1039 and #1042, #1142, #1148 it seems to be important I mention that I'm booting on uSD card.

Blogshot · 2021-03-09T11:30:19Z

+1

This happened after my Odroid C2 (HW rev. 0.2 20160226) lost power in the night. I've unsuccessfully tried every release from 5.4 to 5.12 with a previously working SDCard (SanDisk 32GB) and eMMC (16 GB).

With 4.20, the issue does not exist.

Note: Because I was dumb and didn't configure backups, I needed to recover my configuration by mounting the SDCard in my PC.

etix · 2021-03-09T14:59:37Z

Same issue on a fresh install, reflashed the 4.20 image on the eMMC, and no problem there. Might be kernel-related but the regression is definitely in 5.x.

elpiaffo · 2021-03-09T15:17:10Z

It's definitely kernel related, as I've found some mentions of same issue on odroid c2 boards with newer kernels, unrelated to hassos. 4.20 works without any issues so far, so it's recommended version I suppose until there's some development on this front. Various users happens to have different network issues on r-pi boards on versions 5.x (happened to me as well), so I'm running sticking to 4.20 on other boards as well.

agners · 2021-03-17T10:38:31Z

Can somebody try a 6.0 development release from here:
https://os-builds.home-assistant.io/6.0.dev20210311/

This comes with the Linux kernel 5.10 for all boards.

Blogshot · 2021-03-17T11:24:05Z

Can somebody try a 6.0 development release from here:
https://os-builds.home-assistant.io/6.0.dev20210311/

This comes with the Linux kernel 5.10 for all boards.

Just tested it via SD-Card. Still no activity LED on ethernet port and the same messages being spammed:

meson8b-dwmac c9410000.ethernet etg0: no phy at addr -1
meson8b-dwmac c9410000.ethernet etg0: stmmac_open: Cannot attach to PHY (error: -19)

I found this thread on the C2's ARMBian-Forum, where the same issue is discussed:
https://forum.armbian.com/topic/17112-odroid-c2-no-eth0-with-latest-image/

There seems to be an issue with uboot. One person "fixed" it by copying 4.19's uboot to his device:
https://forum.armbian.com/topic/17112-odroid-c2-no-eth0-with-latest-image/?do=findComment&comment=120312

Another person compiled a specific version, which he claims to be functional:
https://forum.armbian.com/topic/17112-odroid-c2-no-eth0-with-latest-image/?do=findComment&comment=120657

I'm no developer, but these might be worth to look into. I have spare SD-Cards laying around, so I will gladly test any new build. :)

agners · 2021-03-17T13:09:07Z

Thanks for the pointers those are good finds. It is somewhat unclear if Armbian added new patches or just that new U-Boot fixes it. I actually updated to U-Boot 2021.04-rc4 just yesterday, so we can just give that a shot and see if it fixes the problem. I'll create a build sometime today and will post the link once ready.

agners · 2021-03-17T22:14:45Z

New builds with U-Boot 2021.04-rc4 are ready:
https://os-builds.home-assistant.io/6.0.dev20210317/

Blogshot · 2021-03-18T09:36:35Z

Sadly, no luck with that build, either. The systems still spams the usual messages.

agners · 2021-03-19T10:28:48Z

HAOS 4.20 came with U-Boot 2020.01. With HAOS 5.3 we switched to 2020.10.

HAOS 4.20 came with Linux 5.7.19. With HAOS 5.4 we switched to 5.9.1.

So now the interesting question is: Is this U-Boot or a Linux kernel problem? If 5.3 works, then it must be a Linux problem, however, if 5.3 already shows the problem, it is most likely a U-Boot regression which came in sometime between 2020.01 and 2020.10 release.

@Blogshot can you test 5.3 and see if that works?

Blogshot · 2021-03-19T11:57:34Z

I just confirmed the issue on 5.3 (https://github.com/home-assistant/operating-system/releases/download/5.3/hassos_odroid-c2-5.3.img.gz).

This indicates that the issue indeed lies within uboot.

agners · 2021-03-19T15:01:56Z

I created two custom OS 5.12 builds, one with U-Boot 2020.01 and one with 2020.04, can you give those a try?

https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210319-uboot-2020.01.img.xz
https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210319-uboot-2020.04.img.xz

Blogshot · 2021-03-19T16:51:10Z

Good news: The image with uboot 2020.04 worked. I'm currently sitting in front of "Preparing Home Assistant".

This narrows down the issue to uboot newer than 2020.04. Therefore I haven't tested your 2020.01 image.

garethbradley · 2021-03-26T09:32:40Z

I also got to the "Preparing Home Assistant" using the 2020.04 as @Blogshot reported. But the log is full of:

21-03-26 09:29:39 INFO (SyncWorker_2) [supervisor.docker.interface] Updating image homeassistant/odroid-c2-homeassistant:landingpage to homeassistant/odroid-c2-homeassistant:2021.3.4
21-03-26 09:29:39 INFO (SyncWorker_2) [supervisor.docker.interface] Downloading docker image homeassistant/odroid-c2-homeassistant with tag 2021.3.4.
21-03-26 09:30:06 ERROR (SyncWorker_2) [supervisor.docker.interface] Can't install homeassistant/odroid-c2-homeassistant:2021.3.4 -> 404 Client Error for http+docker://localhost/v1.40/images/homeassistant/odroid-c2-homeassistant:2021.3.4/json: Not Found ("no such image: homeassistant/odroid-c2-homeassistant:2021.3.4: No such image: homeassistant/odroid-c2-homeassistant:2021.3.4").
21-03-26 09:30:06 WARNING (MainThread) [supervisor.homeassistant.core] Error on Home Assistant installation. Retry in 30sec

agners · 2021-04-01T12:22:15Z

@Blogshot I made a version with U-Boot 2020.07, can you test if that version still works?
https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210401-uboot-2020.07.img.xz

agners · 2021-04-01T13:00:28Z

Actually I have a theory what could cause the problem. Commit "arm64: dts: meson: sync dt and bindings from v5.6-rc2" changes reset timings for the Ethernet PHY. Can you try this build:
https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210401-u-boot-2020.10-phy-reset-fix.img.xz

Blogshot · 2021-04-01T18:09:01Z

Actually I have a theory what could cause the problem. Commit "arm64: dts: meson: sync dt and bindings from v5.6-rc2" changes reset timings for the Ethernet PHY. Can you try this build:
https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210401-u-boot-2020.10-phy-reset-fix.img.xz

The PHY-reset build is booting and showing the "Preparing Home Assistant". However, because HomeAssistant is also my DNS Server, it can't finish setup because it can't download the docker images.

Since the PHY-fix has the more recent uboot, I haven't tested the 2020.07 build.

sarnold · 2021-04-01T20:03:23Z

@agners can you post your patch somewhere so I can build/test it using meta-meson?

sarnold · 2021-04-02T02:15:42Z

I would not expect 2020.07 to work, since that was the hard cut-off for the DM changes (that are still missing some in-queue patches to make things work again). For now, it's either 2020.04 or the above patch (I'd probably stick with 2020.04 for public releases for now).

agners · 2021-04-06T07:57:49Z

Actually I have a theory what could cause the problem. Commit "arm64: dts: meson: sync dt and bindings from v5.6-rc2" changes reset timings for the Ethernet PHY. Can you try this build:
https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210401-u-boot-2020.10-phy-reset-fix.img.xz

The PHY-reset build is booting and showing the "Preparing Home Assistant".

Cool! So that is it then. I also changed reset timings in update Linux recently. In this second variant I use that same timing. Can you give this a try too? If this works, then the fix is really a trivial change we can also send to upstream U-Boot:

https://os-builds.home-assistant.io/hassos_odroid-c2-5.12.dev20210401-u-boot-2020.10-phy-reset-fix-variant2.img.xz

However, because HomeAssistant is also my DNS Server, it can't finish setup because it can't download the docker images.

It seems you have a chicken-egg problem there :) You probably would have to change the host system DNS Server to something else to get it to download the Docker image. Should be possible with a nmcli command.

agners · 2021-04-06T08:00:20Z

@sarnold

@agners can you post your patch somewhere so I can build/test it using meta-meson?

This is the patch from the first try:

0001-arm64-dts-meson-revert-breaking-change-for-Ethernet.patch.gz

And this is the change for the variant 2 test. If this fixes it, I will send that upstream.

0001-arm64-dts-meson-fix-PHY-deassert-timing-requirements.patch.gz

Blogshot · 2021-04-06T10:52:31Z

@agners
Your second variant of the fix does work as well.

Off-Topic:

Should be possible with a nmcli command.

I've tried dns options --servers dns://8.8.8.8 followed by host reboot. However, supervisor logs showed that this setting does not affect host dns resolution when connecting to dockerhub. There was no mention on how to access the host shell in any help-command.
Finally, I've noticed on the forums that typing login drops the user to the host shell. I was then able to execute nmcli con modify "HassOS default" +ipv4.dns 8.8.8.8 followed by reboot. This seems to have worked; my resolv.conf now has a second dns server and the WebUI now proceeds to the onboarding subpage.

PS: I've restored my latest backup of 4.20 to 5.12, which has worked flawlessly. :)

agners · 2021-04-06T11:58:44Z

@agners
Your second variant of the fix does work as well.

Cool! Thanks a lot for testing!

Wrt DNS: The dns options command only affects the DNS plug-in, which is used by everything running in containers, but not by the host OS itself. We cannot rely on the DNS plug-in on host side as this would be another chicken-egg problem.

There is some documentation here in the OS repository in the Documentation folder. But its less than optimal and rework is on the todo list.

agners · 2021-04-08T08:20:22Z

@Blogshot

Your second variant of the fix does work as well.

It turns out that this actually should not be working, since the change doens't really do anything 🙈 . Is the problem maybe not appearing always and it was just "bad" luck that it worked in your test?

I committed now the first fix, and the latest development build should definitely fix it:
https://os-builds.home-assistant.io/6.0.dev20210407/

agners · 2021-04-08T12:06:47Z

For reference, upstream patch: https://lists.denx.de/pipermail/u-boot/2021-April/446658.html

agners · 2021-04-08T12:06:58Z

This will be fixed from OS 5.14 onwards.

etix · 2021-06-12T11:21:34Z

For anyone wondering, the upgrade from 4.20 to 6.0 went smoothly on the Odroid-C2.

agners added the board/odroid Hardkernel's ODROID Boards label Feb 3, 2021

Blogshot mentioned this issue Mar 10, 2021

Odroid C2 not booting from eMMC after update to Version 5 and above home-assistant/core#46213

Closed

sarnold mentioned this issue Mar 29, 2021

u-boot ethernet phy regression on gxbb (nanopi-k2, odroid-c2) superna9999/meta-meson#106

Closed

agners added a commit to agners/operating-system that referenced this issue Apr 7, 2021

Fix ODROID-C2 Ethernet on some boards (home-assistant#1202)

7860c1c

agners added a commit that referenced this issue Apr 7, 2021

Fix ODROID-C2 Ethernet on some boards (#1202) (#1305)

bb7964f

agners added a commit that referenced this issue Apr 8, 2021

Fix ODROID-C2 Ethernet on some boards (#1202) (#1305)

342b8ef

agners closed this as completed Apr 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot install latest hassos on ODROID-C2 #1202

Cannot install latest hassos on ODROID-C2 #1202

elpiaffo commented Feb 3, 2021 •

edited

Loading

agners commented Feb 3, 2021

elpiaffo commented Feb 3, 2021

elpiaffo commented Feb 3, 2021

oOoHoCo commented Feb 9, 2021

carolinaxxxxx commented Feb 10, 2021

fantoine commented Feb 20, 2021

sylbouns commented Feb 21, 2021

sylbouns commented Feb 23, 2021

Blogshot commented Mar 9, 2021 •

edited

Loading

etix commented Mar 9, 2021

elpiaffo commented Mar 9, 2021

agners commented Mar 17, 2021

Blogshot commented Mar 17, 2021 •

edited

Loading

agners commented Mar 17, 2021

agners commented Mar 17, 2021

Blogshot commented Mar 18, 2021

agners commented Mar 19, 2021

Blogshot commented Mar 19, 2021

agners commented Mar 19, 2021 •

edited

Loading

Blogshot commented Mar 19, 2021 •

edited

Loading

garethbradley commented Mar 26, 2021 •

edited

Loading

agners commented Apr 1, 2021

agners commented Apr 1, 2021

Blogshot commented Apr 1, 2021

sarnold commented Apr 1, 2021 •

edited

Loading

sarnold commented Apr 2, 2021

agners commented Apr 6, 2021

agners commented Apr 6, 2021

Blogshot commented Apr 6, 2021 •

edited

Loading

agners commented Apr 6, 2021

agners commented Apr 8, 2021

agners commented Apr 8, 2021

agners commented Apr 8, 2021

etix commented Jun 12, 2021 •

edited

Loading

Cannot install latest hassos on ODROID-C2 #1202

Cannot install latest hassos on ODROID-C2 #1202

Comments

elpiaffo commented Feb 3, 2021 • edited Loading

agners commented Feb 3, 2021

elpiaffo commented Feb 3, 2021

elpiaffo commented Feb 3, 2021

oOoHoCo commented Feb 9, 2021

carolinaxxxxx commented Feb 10, 2021

fantoine commented Feb 20, 2021

sylbouns commented Feb 21, 2021

sylbouns commented Feb 23, 2021

Blogshot commented Mar 9, 2021 • edited Loading

etix commented Mar 9, 2021

elpiaffo commented Mar 9, 2021

agners commented Mar 17, 2021

Blogshot commented Mar 17, 2021 • edited Loading

agners commented Mar 17, 2021

agners commented Mar 17, 2021

Blogshot commented Mar 18, 2021

agners commented Mar 19, 2021

Blogshot commented Mar 19, 2021

agners commented Mar 19, 2021 • edited Loading

Blogshot commented Mar 19, 2021 • edited Loading

garethbradley commented Mar 26, 2021 • edited Loading

agners commented Apr 1, 2021

agners commented Apr 1, 2021

Blogshot commented Apr 1, 2021

sarnold commented Apr 1, 2021 • edited Loading

sarnold commented Apr 2, 2021

agners commented Apr 6, 2021

agners commented Apr 6, 2021

Blogshot commented Apr 6, 2021 • edited Loading

agners commented Apr 6, 2021

agners commented Apr 8, 2021

agners commented Apr 8, 2021

agners commented Apr 8, 2021

etix commented Jun 12, 2021 • edited Loading

elpiaffo commented Feb 3, 2021 •

edited

Loading

Blogshot commented Mar 9, 2021 •

edited

Loading

Blogshot commented Mar 17, 2021 •

edited

Loading

agners commented Mar 19, 2021 •

edited

Loading

Blogshot commented Mar 19, 2021 •

edited

Loading

garethbradley commented Mar 26, 2021 •

edited

Loading

sarnold commented Apr 1, 2021 •

edited

Loading

Blogshot commented Apr 6, 2021 •

edited

Loading

etix commented Jun 12, 2021 •

edited

Loading