Add timeout to connector pod init container command

[pmossman]

Addresses #10587

[cgardens]

is there any sane way to heartbeat to the orchestrator?

timeouts could be such a headache here. if we do need a time out, are we confident that in 99% of cases that if it is taking longer than 5 min it is that something is broken and that it isn't a big config?

also, if we ever do exit due to timeout, I think it is worth explicitly logging that the error was due to a timeout to make it easier to debug.

[pmossman]

I'm looking through how the heartbeat server works now, I just wanted to put this PR up to build a version of the timeout because it was pretty low effort to throw together. 5 minutes is arbitrary but yeah, you make a good point that no matter how high we set it, there'd always be some concern that we just have a massive config to copy over and we end up killing a working container.

[jrhizor]

A kube cluster internal kubectl cp should be moving data at >1MB/s. Are we expecting 300mb+ configs?

[jrhizor]

Another way we could approach this is to check to see if the disk usage in the folder is increasing over time. If it's increasing, we could keep on resetting our timeout, but if it's stalled and not increasing for 1min that seems safe.

If we do want to allow this to use heartbeating, I think the approach would be to get rid of the init container entirely and instead move this waiting logic to the main container. Then we'd also need to change the KubePodProcess logic to copy files into the main container instead of the init container, and change the watches for pod status to look at the other containers.

For the orchestrator, we don't want a consistent connection between the two so we either need the heartbeat as it exists today or some additional check that the transfer is actually occurring.

[cgardens]

Also, fwiw, I'm fine with going with the timeout now (as long as it logs when the timeout is the thing that terminates it) and create a follow up issue to be smarter later.

[pmossman]

@jrhizor @cgardens updated this PR with disk usage checking and a timeout of 1 minute if disk utilization hasn't changed.

A few notes:

1. the init container uses a version of busybox that doesn't support du -b, so we can't be as granular as I would have liked. I don't think this will be an issue since there should be plenty of copied data before the timeout to cause du to report a new value.
2. I wasn't able to come up with an elegant way to write an integration test for this. Would need to somehow interrupt the file copy and wait for it to timeout. Open to suggestions there.

I ran this locally to make sure the script was working as expected, and added some verbose logging that I didn't end up committing:

❯ kl logs busybox-sync-some-id-0-wawht -c init -f
iteration: 1
iteration: 2
iteration: 3
iteration: 4
iteration: 5
iteration: 6
iteration: 7
iteration: 8
disk usage was 988, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 1976, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 2964, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 3952, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 4940, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 5928, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 6916, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 7904, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 8892, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 9880, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 10868, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 11856, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 12844, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 13832, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 14820, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 15808, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 16796, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 17784, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 18772, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 19760, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 20748, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 21736, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 22724, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 23712, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 24700, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 25688, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 26676, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 27664, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 28652, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 29640, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 30628, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 31616, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 32604, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 33592, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 34580, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 35568, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 36556, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 37544, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 38532, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 39520, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 40508, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 41496, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 42484, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 43472, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 44460, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 45448, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 46436, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 47424, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 48412, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 49400, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 50388, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 51376, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 52364, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 53352, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 54340, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 55328, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 56316, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 57304, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 58292, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 59280, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 60268, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 61256, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 62244, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 63232, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 64220, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 65208, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 66196, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 67184, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 68172, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 69160, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 70148, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 71136, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 72124, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 73112, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 74100, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 74932, setting iteration back to 0
iteration: 1
disk usage was 75088, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 76076, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 77064, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 78052, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 79040, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 80028, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 81016, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 82004, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 82992, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 83980, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 84968, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 85956, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 86944, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 87932, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 88920, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 89908, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 90896, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 91884, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 92872, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 93860, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 94848, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 95836, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 96824, setting iteration back to 0
iteration: 1
iteration: 2
iteration: 3
disk usage was 97812, setting iteration back to 0
iteration: 1
iteration: 2
disk usage was 98800, setting iteration back to 0
iteration: 1
All files copied successfully, exiting with code 0...

And here's what the pod manifest looks like when described:

Command:
      sh
      -c
      USES_STDIN=false

      mkfifo /pipes/stdout
      mkfifo /pipes/stderr

      if [ "$USES_STDIN" = true ]; then
        mkfifo /pipes/stdin
      fi

      ITERATION=0
      MAX_ITERATION=600
      DISK_USAGE=$(du -s /config | awk '{print $1;}')

      until [ -f FINISHED_UPLOADING -o $ITERATION -ge $MAX_ITERATION ]; do
        ((ITERATION=ITERATION+1))
        echo "iteration: ${ITERATION}"
        LAST_DISK_USAGE=$DISK_USAGE
        DISK_USAGE=$(du -s /config | awk '{print $1;}')
        if [ $DISK_USAGE -gt $LAST_DISK_USAGE ]; then
          echo "disk usage was ${DISK_USAGE}, setting iteration back to 0"
          ITERATION=0
        fi
        sleep 0.1
      done

      if [ -f FINISHED_UPLOADING ]; then
        echo "All files copied successfully, exiting with code 0..."
        exit 0
      else
        echo "Timeout while attempting to copy to init container, exiting with code 1..."
        exit 1
      fi

[cgardens]

cool! looks good to me.