epoxy_client reboot retry logic is broken

[nkinkade]

epoxy_client is supposed to attempt to contact ePoxy repeatedly, and after some time, if it can't, it should reboot the machine. This reboot logic is broken, apparently something to do with sysrq not being enabled by default in Ubuntu:

https://github.com/m-lab/epoxy/blob/master/cmd/epoxy_client/main.go#L93

[nkinkade]

Investigating this, Ubuntu is in fact configured to support SYSRQ commands. I forced mlab1-lga0t to get stuck in stage1 (deleted ePoxy GCD entity), and then manually caused the machine to boot from the USB. Logging into the machine, I saw that epoxy_client was failing to get a reply from ePoxy. I then SSH'd into the machine and ran:

root@mlab1-lga0t:~# cat /proc/sys/kernel/sysrq
176
root@mlab1-lga0t:~# echo 1 > /proc/sys/kernel/sysrq
root@mlab1-lga0t:~# echo b > /proc/sysrq-trigger

The machine promptly rebooted. The problem must be somewhere else.