only update etcd when preempted

[asm582]

Issue link

#632

What changes have been made

Verification steps

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • Testing is not required for this change

[metalcycling]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: metalcycling

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [metalcycling]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jbusche]

Hi @asm582 I was able to build and test this PR with the perf-tests.

AppWrappers     Total Time (seconds)    MCAD Image      Cluster Info    Comments
5       30 seconds      Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633
20      80 seconds      Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633
50      153 seconds     Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633
100     353 seconds     Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633
200     720 seconds     Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633
300     1116 seconds    Abishek PR 633  Fyre MEDIUM OC 412 3 Nodes, 8 cpu/16GB mem       quay.io/jbusche/mcad-controller:fix-preempt1-try1 from his pr 633

This compares very similar to release-v1.35.0 and faster than the combined codeflare-operator:v1.0.0-rc.1

I'm storing my performance runs here: https://ibm.ent.box.com/file/1182185239246

[astefanutti]

/lgtm