A single interface failures makes l2 stop working

This was raised on slack https://kubernetes.slack.com/archives/C8FSTGU56/p1668605044508709
If a node is v4 only but some interfaces have v6 addresses, we stop creating the responders and l2 is not working because the list of interfaces we check is empty.