Fixing some LP bridge flash corruption

[joerick]

I just managed to fix a BERG Cloud bridge and thought it was interesting so wanted to document it.

My bridge hasn't been working for a while now. The device would power on, but periodically reboot, and the LP couldn't connect.

I could SSH in, but since the filesystem is mounted RO, it was hard to see how software was the culprit. I eventually managed to get an error message by commenting out the reboot in /usr/bin/oneshot_bergcloud_bridge.sh, waiting for the system-run invocation of that to finish, and then running it myself over an SSH channel. The output looked like this:

Remounting RW
Traceback (most recent call last):
  File "PKGTMP.35950/usr/local/bergcloud-bridge/updater.py", line 5, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 94, in <module>
  File "/usr/lib/python2.7/httplib.py", line 79, in <module>
  File "/usr/lib/python2.7/mimetools.py", line 11, in <module>
  File "/usr/lib/python2.7/rfc822.py", line 74, in <module>
ImportError: dynamic module does not define init function (inittime)
Remounting RO
Finished updates
Loading override URL
Running compiled BERG Cloud daemon
Traceback (most recent call last):
  File "PKGTMP.35950/usr/local/bergcloud-bridge/weminuche_bridge.py", line 10, in <module>
  File "PKGTMP.35950/usr/local/bergcloud-bridge/api.py", line 4, in <module>
ImportError: dynamic module does not define init function (inittime)
Unclean exit. Remaining daemon restart count is now 4
Remounting RW
Traceback (most recent call last):
  File "PKGTMP.35950/usr/local/bergcloud-bridge/updater.py", line 5, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 94, in <module>
  File "/usr/lib/python2.7/httplib.py", line 79, in <module>
  File "/usr/lib/python2.7/mimetools.py", line 11, in <module>
  File "/usr/lib/python2.7/rfc822.py", line 74, in <module>
ImportError: dynamic module does not define init function (inittime)
Remounting RO
Finished updates
Loading override URL
Running compiled BERG Cloud daemon
Traceback (most recent call last):
  File "PKGTMP.35950/usr/local/bergcloud-bridge/weminuche_bridge.py", line 10, in <module>
  File "PKGTMP.35950/usr/local/bergcloud-bridge/api.py", line 4, in <module>
ImportError: dynamic module does not define init function (inittime)
Unclean exit. Remaining daemon restart count is now 3
Remounting RW
...

It continued like that. (I don't like that it's constantly remounting RW to try to update itself, I'm gonna remove that.) But also, we see this strange error from Python:

Traceback (most recent call last):
  File "PKGTMP.35950/usr/local/bergcloud-bridge/updater.py", line 5, in <module>
  File "/usr/lib/python2.7/urllib2.py", line 94, in <module>
  File "/usr/lib/python2.7/httplib.py", line 79, in <module>
  File "/usr/lib/python2.7/mimetools.py", line 11, in <module>
  File "/usr/lib/python2.7/rfc822.py", line 74, in <module>
ImportError: dynamic module does not define init function (inittime)

Some built-in Python modules are defined as native code, in library files. This one was in /usr/lib/python2.7/lib-dynload/time.so. I pulled it to my laptop, and compared it with the same file from berg-bridge-dump.

Somehow that file has had some bytes zero'd out. I have to think that this is hardware failure, that byte offset aligns with a 4kB block size, so some portion of the flash chip has lost its data.

I copied back over the berg-bridge-dump version, (thanks again @kpeeem) and the bridge is working again.

We occasionally come across LPs whose bridges just don't boot - if you connect a UART, sometimes you can see uBoot start, then it dies even before Linux can start. I now suspect that flash corruption is the cause here too. I wish there was a way to reflash these chips. Maybe via JTAG? I'm out of my depth at that point. Maybe somebody knows.