Retry SSH keyscan command

[stefanprodan]

Prevent CircleCI build from failing due to transient network errors

Fix: #1969

[squaremo]

Could known_hosts.sh and verify_known_hosts.sh be combined?

[squaremo]

(I don't know why I did that as a review)

[stefanprodan]

I'm going to rerun the e2e tests a couple of times and see if the number of retries needs to be increased.

[stefanprodan]

Hmm looks like shh-keyscan doesn't fail on host timeout:

Step 5/17 : RUN sh /home/flux/known_hosts.sh /etc/ssh/ssh_known_hosts &&     rm /home/flux/known_hosts.sh
 ---> Running in 5f438543d24b
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# ssh.dev.azure.com:22 SSH-2.0-SSHBlackbox.10
# vs-ssh.visualstudio.com:22 SSH-2.0-SSHBlackbox.10
# vs-ssh.visualstudio.com:22 SSH-2.0-SSHBlackbox.10
# vs-ssh.visualstudio.com:22 SSH-2.0-SSHBlackbox.10
# github.com:22 SSH-2.0-babeld-873429c4
# github.com:22 SSH-2.0-babeld-873429c4
# github.com:22 SSH-2.0-babeld-873429c4
--- -
+++ /tmp/tmp.GPfpPo
@@ -2,6 +2,5 @@
 2048 SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 github.com (RSA)
 2048 SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og ssh.dev.azure.com (RSA)
 2048 SHA256:ohD8VZEXGWo6Ez8GSEJQ9WpafgLFsOfLOtGGQCQo6Og vs-ssh.visualstudio.com (RSA)
-2048 SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A bitbucket.org (RSA)
 256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw gitlab.com (ECDSA)
 256 SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8 gitlab.com (ED25519)
The command '/bin/sh -c sh /home/flux/known_hosts.sh /etc/ssh/ssh_known_hosts &&     rm /home/flux/known_hosts.sh' returned a non-zero code: 1
Makefile:64: recipe for target 'build/.helm-operator.done' failed

The approach with two scripts was better, it would've retry both actions

[stefanprodan]

@squaremo @hiddeco now it works as expected, both operations will retry if any of them fail.

[hiddeco]

🚢