API Verification

[abutcher]

Wait until the API becomes available

• before starting the controllers service w/ native ha.
• before node registration.
• after api service restarts

[detiber]

@sdodson would this solve your containerized problem if we trigger the test for non-ha installs as well?

It might require a bit of rework to make it so that the ordering is correct for ha and non-ha deployments, but I think it's worth it to notify the user of a failed api server before getting a failure from a later role (usually openshift_examples)

[sdodson]

The containerization problem is unique to having to restart docker during node initialization which leads to up to a minute of master downtime while systemd starts the master services again after docker has been restarted. We need to fix both. Should I limit my fix to the node restarting docker in #1137 ?

[detiber]

@sdodson hmm, I think considering they are slightly different issues, it probably wouldn't hurt to leave the additional check in place for #1137

[sdodson]

👍 to needing this and the other, LGTM

[abutcher]

@wshearn is testing this w/ online

[detiber]

👍

[wshearn]

So the curl part is failing for me, it is like the api_available_output is not getting updated.

TASK: [openshift_master | Wait for API to become available] ******************* 
ok: [XXXXXXXXXX]

TASK: [openshift_master | fail ] ********************************************** 
failed: [XXXXXXXXXX] => {"failed": true}
msg: Unable to contact master API at https://nope.com

FATAL: all hosts have already failed -- aborting

But the ELB is saying InService around the time it fails.

[detiber]

@wshearn how is the elb health check configured? If I'm remembering the default settings right 120s might time out before re-adding a previously failed host.

[wshearn]

Ping Target TCP:443
Timeout 5 seconds
Interval 30 seconds
Unhealthy Threshold 2
Healthy Threshold 2

And I bumped it up to 180 retries.

[detiber]

how about increasing the retry interval, I believe I vaguely remember reading a bug about using 0 or 1 as the retry interval?

[abutcher]

@wshearn the fail conditional was reversed. Updated and this is working for me locally.

[abutcher]

aos-ci-test

[wshearn]

👍 works for me.

[abutcher]

Addressing the nosetests failure in #1169

[abutcher]

aos-ci-test

[abutcher]

Melded @sdodson and my changes together and I'm still testing native ha with the combo.

[abutcher]

aos-ci-test

[abutcher]

@detiber @wshearn PTAL

[detiber]

Why not move this down below the set_fact and make it conditional on start_result?

[abutcher]

Good idea

[abutcher]

aos-ci-test

[abutcher]

Closes #1137

[abutcher]

@detiber Moved the wait below set_fact

[detiber]

@brenton 👍