POST request to http://localhost:8100/mgmt/shared/cloud-failover/trigger fails with 500 error

[y-myk]

BIG-IP HA cluster in Azure deployed from https://github.com/F5Networks/f5-azure-arm-templates/tree/v7.4.0.0/supported/failover/same-net/via-api/n-nic/existing-stack/byol

Deployment completed with no errors. However failover fails.

/var/log/cloud/azure/onboard.log

2020-03-17T14:21:54.300Z info: Onboard starting.
...
2020-03-17T14:22:10.411Z info: Licensing.
2020-03-17T14:22:34.619Z info: Provisioning modules {"ltm":"nominal"}
2020-03-17T14:23:07.305Z info: Installing package at path: /var/config/rest/downloads/f5-cloud-failover-1.1.0-0.noarch.rpm
2020-03-17T14:23:07.306Z info: Installing package at path: /var/config/rest/downloads/f5-appsvcs-3.5.1-5.noarch.rpm
2020-03-17T14:23:23.368Z info: Saving config.
2020-03-17T14:23:35.766Z info: Waiting for device to be active.
2020-03-17T14:23:40.228Z info: Sending metrics
2020-03-17T14:23:41.126Z info: Device onboard complete.
2020-03-17T14:23:44.433Z info: Onboard finished.

/var/log/cloud/azure/cluster.log

2020-03-17T14:24:08.961Z info: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/cluster.js called with /usr/bin/f5-rest-node /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/cluster.js --output /var/log/cloud/azure/cluster.log --log-level info --host 10.10.0.11 --port 443 -u svc_user --password-url file:///config/cloud/.passwd --password-encrypted --config-sync-ip 10.10.2.11 --join-group --device-group Sync --sync --remote-host 10.10.0.10 --remote-user svc_user --remote-password-url file:///config/cloud/.passwd
2020-03-17T14:24:08.970Z info: Cluster starting.
...
2020-03-17T14:24:48.624Z info: Adding to remote trust.
2020-03-17T14:25:04.010Z info: Adding to remote device group.
2020-03-17T14:25:10.712Z info: Checking for datasync-global-dg.
2020-03-17T14:25:21.554Z info: Telling remote to sync.
2020-03-17T14:25:53.117Z info: Waiting for sync to complete.
2020-03-17T14:25:54.571Z info: Sync complete.
2020-03-17T14:25:54.573Z info: Waiting for BIG-IP to be active.
2020-03-17T14:25:56.619Z info: Cluster finished.

/var/log/ltm

Mar 18 10:40:54 f5jp010.eastus.cloudapp.azure.com notice sod[7364]: 010c006d:5: Leaving Standby for Active: Next Active, peers agree on config.
Mar 18 10:40:54 f5jp010.eastus.cloudapp.azure.com notice sod[7364]: 010c0053:5: Active for traffic group traffic-group-1.
Mar 18 10:40:54 f5jp010.eastus.cloudapp.azure.com notice sod[7364]: 010c0019:5: Active

/var/log/restnoded/restnoded.log

Wed, 18 Mar 2020 10:40:54 GMT - finest: socket 202 opened
Wed, 18 Mar 2020 10:40:59 GMT - info: [f5-cloud-failover] Performing failover - execute
Wed, 18 Mar 2020 10:40:59 GMT - info: [f5-cloud-failover] Performing Failover - recovery
Wed, 18 Mar 2020 10:40:59 GMT - warning: [f5-cloud-failover] Recovering previous task: {"addresses":null,"routes":null}
Wed, 18 Mar 2020 10:40:59 GMT - info: [f5-cloud-failover] Performing Failover - update
Wed, 18 Mar 2020 10:40:59 GMT - info: [f5-cloud-failover] No localAddresses/failoverAddresses to discover
Wed, 18 Mar 2020 10:40:59 GMT - severe: [f5-cloud-failover] failover.execute() error: Cannot read property 'disassociate' of undefined TypeError: Cannot read property 'disassociate' of undefined
at _discoverAddressOperations.then.operations (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/azure/cloud.js:129:66)
at tryCatcher (/usr/share/rest/node/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:512:31)
at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:569:18)
at Promise._settlePromiseCtx (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:606:10)
at Async._drainQueue (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:138:12)
at Async._drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:143:10)
at Immediate.Async.drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:794:20)
at tryOnImmediate (timers.js:752:5)
at processImmediate [as _immediateCallback] (timers.js:729:5)
Wed, 18 Mar 2020 10:40:59 GMT - info: [f5-cloud-failover] No route operations to run
Wed, 18 Mar 2020 10:41:04 GMT - finest: socket 202 closed

/var/log/restjavad-audit.0.log

[I][243][18 Mar 2020 10:40:59 UTC][ForwarderPassThroughWorker] {"user":"local/admin","method":"POST","uri":"http://localhost:8100/mgmt/shared/cloud-failover/trigger","status":500,"from":"Unknown"}

CFE endpoint https://localhost/mgmt/shared/cloud-failover/info is responding on each HA peer, however GET request to https://localhost/mgmt/shared/cloud-failover/trigger fails:

[admin@f5jp011:Active:In Sync] ~ # curl -ik -u admin https://localhost/mgmt/shared/cloud-failover/info
HTTP/1.1 200 OK
Date: Wed, 18 Mar 2020 10:56:05 GMT
Server: Jetty(9.2.22.v20170606)
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/json; charset=UTF-8
X-Powered-By: Express
Pragma: no-cache
Cache-Control: no-store
Cache-Control: no-cache
Cache-Control: must-revalidate
Expires: -1
Content-Length: 81
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data: http://127.4.1.1 http://127.4.2.1

[admin@f5jp011:Active:In Sync] ~ # curl -ik -u admin https://localhost/mgmt/shared/cloud-failover/trigger
HTTP/1.1 400 Bad Request
Date: Wed, 18 Mar 2020 10:56:32 GMT
Server: Jetty(9.2.22.v20170606)
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=16070400; includeSubDomains
Content-Type: application/json; charset=UTF-8
Pragma: no-cache
Cache-Control: no-store
Cache-Control: no-cache
Cache-Control: must-revalidate
Expires: -1
Content-Length: 1349
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; img-src 'self' data: http://127.4.1.1 http://127.4.2.1
Connection: close

{"taskState":"FAILED","message":"Failover failed because of failover.execute() error: Cannot read property 'disassociate' of undefined TypeError: Cannot read property 'disassociate' of undefined\n at _discoverAddressOperations.then.operations (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/azure/cloud.js:129:66)\n at tryCatcher (/usr/share/rest/node/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:512:31)\n at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:569:18)\n at Promise._settlePromiseCtx (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:606:10)\n at Async._drainQueue (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:138:12)\n at Async._drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:143:10)\n at Immediate.Async.drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:17:14)\n at runCallback (timers.js:794:20)\n at tryOnImmediate (timers.js:752:5)\n at processImmediate [as _immediateCallback] (timers.js:729:5)","timestamp":"2020-03-18T10:54:18.040Z","instance":"f5jp011.eastus.cloudapp.azure.com","failoverOperations":{"addresses":null,"routes":null},"code":400}[admin@f5jp011:Active:In Sync] ~ #

[shyawnkarim]

Thanks for supplying us with these logs. I've filed a bug report, internal ID AUTOSDK-258, to have this investigated.

[alaari-f5]

Closing this issue

As of release CFE 1.2 we moved this CFE repo under F5Networks. Your issue was recreated there. To follow-up on this issue visit:

F5Networks/f5-cloud-failover-extension#7