Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot read property name of undefined fwdRules.forEach #26

Closed
JeffGiroux opened this issue Jun 14, 2020 · 4 comments
Closed

Cannot read property name of undefined fwdRules.forEach #26

JeffGiroux opened this issue Jun 14, 2020 · 4 comments
Labels
enhancement New feature or request
Milestone
Release 1.7

Comments

@JeffGiroux
Copy link

JeffGiroux commented Jun 14, 2020 •
edited
Loading

Hitting an error when performing failover in GCP. I stood up my environment with terraform, proper labels, pre-reqs, etc. I installed CFE and the POST declaration is 200 OK. I perform a failover, but items are not moving. Routes do not move, alias IPs do not move, and forwarding rules are not moving. It appears from the restnoded logs that the forwarding rule logic is causing issues.

specific error from unit 1...

Sun, 14 Jun 2020 18:39:43 GMT - severe: [f5-cloud-failover] Cannot read property 'name' of undefined TypeError: Cannot read property 'name' of undefined
    at fwdRules.forEach (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/gcp/cloud.js:827:80)
    at Array.forEach (<anonymous>)
    at Promise.all.then (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/gcp/cloud.js:808:31)
    at tryCatcher (/usr/share/rest/node/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/usr/share/rest/node/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/usr/share/rest/node/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)

Full Logs

unit 1 during failover to become active...

Sun, 14 Jun 2020 18:39:40 GMT - fine: [f5-cloud-failover] HTTP Request - POST /trigger
Sun, 14 Jun 2020 18:39:40 GMT - fine: [f5-cloud-failover] Performing failover - initialization
Sun, 14 Jun 2020 18:39:40 GMT - fine: [f5-cloud-failover] config: {"class":"Cloud_Failover","environment":"gcp","externalStorage":{"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverAddresses":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverRoutes":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"},"scopingAddressRanges":[{"range":"0.0.0.0/0"}],"defaultNextHopAddresses":{"discoveryType":"static","items":["10.1.10.31","10.1.10.30"]}},"controls":{"class":"Controls","logLevel":"silly"},"schemaVersion":"1.3.0"}
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] bucket name: f5-bigip-storage-jchambers-failover bucket labels: [{"f5_cloud_failover_label":"jchambers-failover","goog-dm":"jchambers-failover"}]
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] bucket name: giroux77-storage bucket labels: [{"f5_cloud_failover_label":"mydeployment"}]
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] bucket name: jgiroux123 bucket labels: [{"f5_cloud_failover_label":"jg-f5-tf-ha"}]
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] deployment bucket name: giroux77-storage
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] Getting GCP resources
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] getFwdRules called with nextPageToken 
Sun, 14 Jun 2020 18:39:41 GMT - fine: [f5-cloud-failover] Cloud provider found vms: {"0":{"id":"2631417376537304696","creationTimestamp":"2020-06-14T10:38:31.698-07:00","name":"giroux5-f5vm01","tags":{"items":["appfw-giroux5","mgmtfw-giroux5"],"fingerprint":"EzXXUUv52l0="},"machineType":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/machineTypes/n1-standard-8","status":"RUNNING","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","canIpForward":true,"networkInterfaces":[{"network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/giroux77-net-ext","subnetwork":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/subnetworks/giroux77-subnet-ext","networkIP":"10.1.10.30","name":"nic0","accessConfigs":[{"type":"ONE_TO_ONE_NAT","name":"external-nat","natIP":"34.83.60.181","networkTier":"PREMIUM","kind":"compute#accessConfig"}],"fingerprint":"BeZpU0Gaeeg=","kind":"compute#networkInterface"},{"network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/giroux77-net-mgmt","subnetwork":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/subnetworks/giroux77-subnet-mgmt","networkIP":"10.1.1.30","name":"nic1","accessConfigs":[{"type":"ONE_TO_ONE_NAT","name":"external-nat","natIP":"34.82.153.142","networkTier":"PREMIUM","kind":"compute#accessConfig"}],"fingerprint":"NRO7HIAH7ug=","kind":"compute#networkInterface"}],"disks":[{"type":"PERSISTENT","mode":"READ_WRITE","source":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/disks/giroux5-f5vm01","deviceName":"persistent-disk-0","index":0,"boot":true,"autoDelete":true,"licenses":["https://www.googleapis.com/compute/v1/projects/f5-7626-networks-public/global/licenses/f5-big-ip-adc-hourly-best-1gbps-updated"],"interface":"SCSI","diskSizeGb":"128","kind":"compute#attachedDisk"}],"metadata":{"fingerprint":"6syWP22fb64=","items":[{"key":"startup-script",
--snippet...lots of data from VM metadata script--
Sun, 14 Jun 2020 18:39:41 GMT - fine: [f5-cloud-failover] Cloud provider found fwdRules: {"0":{"id":"8978073301247843922","creationTimestamp":"2020-06-14T10:39:09.863-07:00","name":"giroux5-forwarding-rule","description":"","region":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1","IPAddress":"34.105.114.14","IPProtocol":"TCP","portRange":"1-65535","target":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/forwardingRules/giroux5-forwarding-rule","loadBalancingScheme":"EXTERNAL","networkTier":"PREMIUM","fingerprint":"7Val-dXL4oM=","kind":"compute#forwardingRule"}}
Sun, 14 Jun 2020 18:39:41 GMT - fine: [f5-cloud-failover] Cloud provider found targetInstances: {"0":{"id":"8868401732253947533","creationTimestamp":"2020-06-04T09:59:14.883-07:00","name":"bigip2-jchambers-failover-ti","description":"bigip2-jchambers-failover","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","natPolicy":"NO_NAT","instance":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/instances/bigip2-jchambers-failover","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/bigip2-jchambers-failover-ti","kind":"compute#targetInstance"},"1":{"id":"1074036512448568926","creationTimestamp":"2020-06-14T10:38:57.734-07:00","name":"giroux5-ti","description":"","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","natPolicy":"NO_NAT","instance":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/instances/giroux5-f5vm02","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","kind":"compute#targetInstance"}}
Sun, 14 Jun 2020 18:39:41 GMT - finest: [f5-cloud-failover] Cloud Provider initialization complete
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] Device initialization complete
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] Failover initialization complete
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] Download stateFile: {"taskState":"SUCCEEDED","message":"Failover Complete","timestamp":"2020-06-14T18:38:22.345Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] taskState: {"taskState":"SUCCEEDED","message":"Failover Complete","timestamp":"2020-06-14T18:38:22.345Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
Sun, 14 Jun 2020 18:39:42 GMT - fine: [f5-cloud-failover] Address operations enabled?  true
Sun, 14 Jun 2020 18:39:42 GMT - fine: [f5-cloud-failover] Route operations enabled?  true
Sun, 14 Jun 2020 18:39:42 GMT - info: [f5-cloud-failover] Performing failover - execute
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] State file data:  {"taskState":"SUCCEEDED","message":"Failover Complete","timestamp":"2020-06-14T18:38:22.345Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] Data will be uploaded to f5cloudfailoverstate.json:  {"taskState":"RUNNING","message":"Failover running","timestamp":"2020-06-14T18:39:42.616Z","instance":"giroux5-f5vm01.c.f5-4136-mspteam-dev.internal","failoverOperations":{}}
Sun, 14 Jun 2020 18:39:42 GMT - info: [f5-cloud-failover] Performing Failover - discovery
Sun, 14 Jun 2020 18:39:42 GMT - fine: [f5-cloud-failover] Getting failover addresses using selfAddresses  {"0":{"address":"10.1.10.30","trafficGroup":"/Common/traffic-group-local-only","trafficGroupMatch":false}}  and floatingAddresses  {"0":{"address":"10.1.10.105"},"1":{"address":"34.105.114.14"}}
Sun, 14 Jun 2020 18:39:42 GMT - fine: [f5-cloud-failover] Retrieved local addresses {"0":"10.1.10.30"}
Sun, 14 Jun 2020 18:39:42 GMT - fine: [f5-cloud-failover] Retrieved failover addresses  {"0":"10.1.10.105","1":"34.105.114.14"}
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] updateAddresses:  {"localAddresses":["10.1.10.30"],"failoverAddresses":["10.1.10.105","34.105.114.14"],"discoverOnly":true}
Sun, 14 Jun 2020 18:39:42 GMT - finest: [f5-cloud-failover] getRoutes called with nextPageToken 
Sun, 14 Jun 2020 18:39:43 GMT - fine: [f5-cloud-failover] Filtered Routes [object Object]
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] Discovered routes:  {"0":{"id":"1937764816610122535","creationTimestamp":"2020-06-14T11:00:40.677-07:00","name":"jg-route2-external","description":"f5_cloud_failover_labels={\"f5_cloud_failover_label\":\"mydeployment\"}","network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext","destRange":"0.0.0.0/0","priority":1000,"nextHopIp":"10.1.10.31","warnings":[{"code":"NEXT_HOP_ADDRESS_NOT_ASSIGNED","message":"Next hop ip address '10.1.10.31' is not assigned to the primary IP address of any instance on 'https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext'.  Please ensure that the address is assigned to the primary IP address of an instance on the route's network.","data":[{"key":"ip_address","value":"10.1.10.31"},{"key":"route_network","value":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext"}]}],"selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/routes/jg-route2-external","kind":"compute#route"}}
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] Next hop address: 10.1.10.30
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] Route to be updated {"id":"1937764816610122535","creationTimestamp":"2020-06-14T11:00:40.677-07:00","name":"jg-route2-external","description":"f5_cloud_failover_labels={\"f5_cloud_failover_label\":\"mydeployment\"}","network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext","destRange":"0.0.0.0/0","priority":1000,"nextHopIp":"10.1.10.31","warnings":[{"code":"NEXT_HOP_ADDRESS_NOT_ASSIGNED","message":"Next hop ip address '10.1.10.31' is not assigned to the primary IP address of any instance on 'https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext'.  Please ensure that the address is assigned to the primary IP address of an instance on the route's network.","data":[{"key":"ip_address","value":"10.1.10.31"},{"key":"route_network","value":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/jgiroux-net-ext"}]}],"selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/routes/jg-route2-external","kind":"compute#route"}
Sun, 14 Jun 2020 18:39:43 GMT - fine: [f5-cloud-failover] Failover addresses to discover {"0":"10.1.10.105","1":"34.105.114.14"}
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] VM name: giroux5-f5vm02
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] getFwdRules called with nextPageToken 
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] updateFwdRules matched rule: {"id":"8978073301247843922","creationTimestamp":"2020-06-14T10:39:09.863-07:00","name":"giroux5-forwarding-rule","description":"","region":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1","IPAddress":"34.105.114.14","IPProtocol":"TCP","portRange":"1-65535","target":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/forwardingRules/giroux5-forwarding-rule","loadBalancingScheme":"EXTERNAL","networkTier":"PREMIUM","fingerprint":"7Val-dXL4oM=","kind":"compute#forwardingRule"}
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] Discovered our target instance  undefined
Sun, 14 Jun 2020 18:39:43 GMT - severe: [f5-cloud-failover] Cannot read property 'name' of undefined TypeError: Cannot read property 'name' of undefined
    at fwdRules.forEach (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/gcp/cloud.js:827:80)
    at Array.forEach (<anonymous>)
    at Promise.all.then (/var/config/rest/iapps/f5-cloud-failover/nodejs/providers/gcp/cloud.js:808:31)
    at tryCatcher (/usr/share/rest/node/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:638:18)
    at PromiseArray._resolve (/usr/share/rest/node/node_modules/bluebird/js/release/promise_array.js:126:19)
    at PromiseArray._promiseFulfilled (/usr/share/rest/node/node_modules/bluebird/js/release/promise_array.js:144:14)
    at Promise._settlePromise (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:574:26)
    at Promise._settlePromise0 (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/share/rest/node/node_modules/bluebird/js/release/promise.js:693:18)
    at Async._drainQueue (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:133:16)
    at Async._drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:143:10)
    at Immediate.Async.drainQueues (/usr/share/rest/node/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:794:20)
    at tryOnImmediate (timers.js:752:5)
    at processImmediate [as _immediateCallback] (timers.js:729:5)
Sun, 14 Jun 2020 18:39:43 GMT - finest: [f5-cloud-failover] Data will be uploaded to f5cloudfailoverstate.json:  {"taskState":"FAILED","message":"Failover failed because Cannot read property 'name' of undefined","timestamp":"2020-06-14T18:39:43.499Z","instance":"giroux5-f5vm01.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":null,"routes":null}}

unit 2 during failover to become active...

Sun, 14 Jun 2020 19:04:36 GMT - finest: socket 240 opened
Sun, 14 Jun 2020 19:04:36 GMT - fine: [f5-cloud-failover] HTTP Request - POST /trigger
Sun, 14 Jun 2020 19:04:36 GMT - fine: [f5-cloud-failover] Performing failover - initialization
Sun, 14 Jun 2020 19:04:36 GMT - fine: [f5-cloud-failover] config: {"class":"Cloud_Failover","environment":"gcp","externalStorage":{"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverAddresses":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"}},"failoverRoutes":{"enabled":true,"scopingTags":{"f5_cloud_failover_label":"mydeployment"},"scopingAddressRanges":[{"range":"192.0.2.0/24"}],"defaultNextHopAddresses":{"discoveryType":"static","items":["10.1.10.31","10.1.10.30"]}},"controls":{"class":"Controls","logLevel":"silly"},"schemaVersion":"1.3.0"}
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] bucket name: f5-bigip-storage-jchambers-failover bucket labels: [{"f5_cloud_failover_label":"jchambers-failover","goog-dm":"jchambers-failover"}]
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] bucket name: giroux77-storage bucket labels: [{"f5_cloud_failover_label":"mydeployment"}]
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] bucket name: jgiroux123 bucket labels: [{"f5_cloud_failover_label":"jg-f5-tf-ha"}]
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] deployment bucket name: giroux77-storage
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] Getting GCP resources
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] getFwdRules called with nextPageToken 
Sun, 14 Jun 2020 19:04:37 GMT - fine: [f5-cloud-failover] Cloud provider found vms: {"0":{"id":"2631417376537304696","creationTimestamp":"2020-06-14T10:38:31.698-07:00","name":"giroux5-f5vm01","tags":{"items":["appfw-giroux5","mgmtfw-giroux5"],"fingerprint":"EzXXUUv52l0="},"machineType":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/machineTypes/n1-standard-8","status":"RUNNING","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","canIpForward":true,"networkInterfaces":[{"network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/giroux77-net-ext","subnetwork":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/subnetworks/giroux77-subnet-ext","networkIP":"10.1.10.30","name":"nic0","accessConfigs":[{"type":"ONE_TO_ONE_NAT","name":"external-nat","natIP":"34.83.60.181","networkTier":"PREMIUM","kind":"compute#accessConfig"}],"fingerprint":"BeZpU0Gaeeg=","kind":"compute#networkInterface"},{"network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/giroux77-net-mgmt","subnetwork":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/subnetworks/giroux77-subnet-mgmt","networkIP":"10.1.1.30","name":"nic1","accessConfigs":[{"type":"ONE_TO_ONE_NAT","name":"external-nat","natIP":"34.82.153.142","networkTier":"PREMIUM","kind":"compute#accessConfig"}],"fingerprint":"NRO7HIAH7ug=","kind":"compute#networkInterface"}],"disks":[{"type":"PERSISTENT","mode":"READ_WRITE","source":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/disks/giroux5-f5vm01","deviceName":"persistent-disk-0","index":0,"boot":true,"autoDelete":true,"licenses":["https://www.googleapis.com/compute/v1/projects/f5-7626-networks-public/global/licenses/f5-big-ip-adc-hourly-best-1gbps-updated"],"interface":"SCSI","diskSizeGb":"128","kind":"compute#attachedDisk"}],"metadata":{"fingerprint":"6syWP22fb64=","items":[{"key":"startup-script"
--snippet...lots of data from VM metadata script--
Sun, 14 Jun 2020 19:04:37 GMT - fine: [f5-cloud-failover] Cloud provider found fwdRules: {"0":{"id":"8978073301247843922","creationTimestamp":"2020-06-14T10:39:09.863-07:00","name":"giroux5-forwarding-rule","description":"","region":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1","IPAddress":"34.105.114.14","IPProtocol":"TCP","portRange":"1-65535","target":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/forwardingRules/giroux5-forwarding-rule","loadBalancingScheme":"EXTERNAL","networkTier":"PREMIUM","fingerprint":"7Val-dXL4oM=","kind":"compute#forwardingRule"}}
Sun, 14 Jun 2020 19:04:37 GMT - fine: [f5-cloud-failover] Cloud provider found targetInstances: {"0":{"id":"8868401732253947533","creationTimestamp":"2020-06-04T09:59:14.883-07:00","name":"bigip2-jchambers-failover-ti","description":"bigip2-jchambers-failover","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","natPolicy":"NO_NAT","instance":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/instances/bigip2-jchambers-failover","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/bigip2-jchambers-failover-ti","kind":"compute#targetInstance"},"1":{"id":"1074036512448568926","creationTimestamp":"2020-06-14T10:38:57.734-07:00","name":"giroux5-ti","description":"","zone":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b","natPolicy":"NO_NAT","instance":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/instances/giroux5-f5vm02","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","kind":"compute#targetInstance"}}
Sun, 14 Jun 2020 19:04:37 GMT - finest: [f5-cloud-failover] Cloud Provider initialization complete
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Device initialization complete
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Failover initialization complete
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Download stateFile: {"taskState":"SUCCEEDED","message":"Failover state file was reset","timestamp":"2020-06-14T19:04:28.230Z","instance":"none","failoverOperations":{}}
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] taskState: {"taskState":"SUCCEEDED","message":"Failover state file was reset","timestamp":"2020-06-14T19:04:28.230Z","instance":"none","failoverOperations":{}}
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Address operations enabled?  true
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Route operations enabled?  true
Sun, 14 Jun 2020 19:04:38 GMT - info: [f5-cloud-failover] Performing failover - execute
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] State file data:  {"taskState":"SUCCEEDED","message":"Failover state file was reset","timestamp":"2020-06-14T19:04:28.230Z","instance":"none","failoverOperations":{}}
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Data will be uploaded to f5cloudfailoverstate.json:  {"taskState":"RUNNING","message":"Failover running","timestamp":"2020-06-14T19:04:38.555Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{}}
Sun, 14 Jun 2020 19:04:38 GMT - info: [f5-cloud-failover] Performing Failover - discovery
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Getting failover addresses using selfAddresses  {"0":{"address":"10.1.10.31","trafficGroup":"/Common/traffic-group-local-only","trafficGroupMatch":false}}  and floatingAddresses  {"0":{"address":"10.1.10.105"},"1":{"address":"34.105.114.14"}}
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Retrieved local addresses {"0":"10.1.10.31"}
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Retrieved failover addresses  {"0":"10.1.10.105","1":"34.105.114.14"}
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] updateAddresses:  {"localAddresses":["10.1.10.31"],"failoverAddresses":["10.1.10.105","34.105.114.14"],"discoverOnly":true}
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] getRoutes called with nextPageToken 
Sun, 14 Jun 2020 19:04:38 GMT - fine: [f5-cloud-failover] Filtered Routes [object Object]
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Discovered routes:  {"0":{"id":"266466546359716558","creationTimestamp":"2020-06-14T12:01:53.239-07:00","name":"jg-route2-external","description":"f5_cloud_failover_labels={\"f5_cloud_failover_label\":\"mydeployment\"}","network":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/networks/giroux77-net-ext","destRange":"192.0.2.0/24","priority":1000,"nextHopIp":"10.1.10.31","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/global/routes/jg-route2-external","kind":"compute#route"}}
Sun, 14 Jun 2020 19:04:38 GMT - finest: [f5-cloud-failover] Next hop address: 10.1.10.31
Sun, 14 Jun 2020 19:04:39 GMT - fine: [f5-cloud-failover] Failover addresses to discover {"0":"10.1.10.105","1":"34.105.114.14"}
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] VM name: giroux5-f5vm01
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] getFwdRules called with nextPageToken 
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] updateFwdRules matched rule: {"id":"8978073301247843922","creationTimestamp":"2020-06-14T10:39:09.863-07:00","name":"giroux5-forwarding-rule","description":"","region":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1","IPAddress":"34.105.114.14","IPProtocol":"TCP","portRange":"1-65535","target":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/regions/us-west1/forwardingRules/giroux5-forwarding-rule","loadBalancingScheme":"EXTERNAL","networkTier":"PREMIUM","fingerprint":"7Val-dXL4oM=","kind":"compute#forwardingRule"}
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] Discovered our target instance  {"name":"giroux5-ti","selfLink":"https://www.googleapis.com/compute/v1/projects/f5-4136-mspteam-dev/zones/us-west1-b/targetInstances/giroux5-ti"}
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] fwdRulesToUpdate:  {}
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] Data will be uploaded to f5cloudfailoverstate.json:  {"taskState":"RUNNING","message":"Failover running","timestamp":"2020-06-14T19:04:39.458Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
Sun, 14 Jun 2020 19:04:39 GMT - info: [f5-cloud-failover] Performing Failover - update
Sun, 14 Jun 2020 19:04:39 GMT - fine: [f5-cloud-failover] Address discovery: {"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}}
Sun, 14 Jun 2020 19:04:39 GMT - finest: [f5-cloud-failover] updateAddresses:  {"updateOperations":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}}}
Sun, 14 Jun 2020 19:04:39 GMT - fine: [f5-cloud-failover] Route discovery: {"operations":[]}
Sun, 14 Jun 2020 19:04:39 GMT - fine: [f5-cloud-failover] updateRoutes operations:  {}
Sun, 14 Jun 2020 19:04:39 GMT - info: [f5-cloud-failover] No route operations to run
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] _updateAddresses interface operations:  {"disassociate":[],"associate":[]}
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] _updateAddresses forwarding rules operations:  {"operations":[]}
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] updateAddresses disassociate operations:  {}
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] updateAddresses associate operations:  {}
Sun, 14 Jun 2020 19:04:40 GMT - info: [f5-cloud-failover] Disassociate NICs successful.
Sun, 14 Jun 2020 19:04:40 GMT - info: [f5-cloud-failover] Updated forwarding rules successfully
Sun, 14 Jun 2020 19:04:40 GMT - info: [f5-cloud-failover] Associate NICs successful.
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] Data will be uploaded to f5cloudfailoverstate.json:  {"taskState":"SUCCEEDED","message":"Failover Complete","timestamp":"2020-06-14T19:04:40.191Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
Sun, 14 Jun 2020 19:04:40 GMT - info: [f5-cloud-failover] Failover Complete
Sun, 14 Jun 2020 19:04:40 GMT - finest: [f5-cloud-failover] Download stateFile: {"taskState":"SUCCEEDED","message":"Failover Complete","timestamp":"2020-06-14T19:04:40.191Z","instance":"giroux5-f5vm02.c.f5-4136-mspteam-dev.internal","failoverOperations":{"addresses":{"publicAddresses":{},"interfaces":{"disassociate":[],"associate":[]},"loadBalancerAddresses":{"operations":[]}},"routes":{"operations":[]}}}
@JeffGiroux
Copy link
Author

JeffGiroux commented Jun 14, 2020 •
edited
Loading

I removed the virtual server IP that matched the forwarding rule from my BIG-IP configs and tested failover again. This way, the forwarding IP should NOT match...and therefore the script should NOT fail. Sure enough, I have success.

My workaround...

  1. Remove IP address for virtual server that matches forwarding rule IP
  2. test failover
  3. routes now move, alias IPs now move

Why are forwarding rules causing a problem?

@JeffGiroux
Copy link
Author

This was my own fault. I forgot to have my terraform plan create TWO target instances: one containing each BIG-IP. I only had ONE target instance which caused the 'name' undefined error.

Suggest that better logging be added to the code to better defined this type of error.

for reference :)

Sun, 14 Jun 2020 20:00:23 GMT - info: [f5-cloud-failover] Performing failover - execute
Sun, 14 Jun 2020 20:00:24 GMT - info: [f5-cloud-failover] Performing Failover - discovery
Sun, 14 Jun 2020 20:00:25 GMT - info: [f5-cloud-failover] Performing Failover - update
Sun, 14 Jun 2020 20:00:29 GMT - info: [f5-cloud-failover] Disassociate NICs successful.
Sun, 14 Jun 2020 20:00:31 GMT - info: [f5-cloud-failover] Updated routes successfully
Sun, 14 Jun 2020 20:00:34 GMT - info: [f5-cloud-failover] Associate NICs successful.
Sun, 14 Jun 2020 20:00:52 GMT - info: [f5-cloud-failover] Updated forwarding rules successfully
Sun, 14 Jun 2020 20:00:53 GMT - info: [f5-cloud-failover] Failover Complete

@JeffGiroux JeffGiroux mentioned this issue Jun 14, 2020
Closed
@shyawnkarim
Copy link

Thanks for suggesting this improvement to our logging. We are now tracking this logging enhancement request with internal issue ID AUTOSDK-362.

@shyawnkarim shyawnkarim added the enhancement New feature or request label Jun 17, 2020
@shyawnkarim shyawnkarim added this to the backlog milestone Jun 17, 2020
@shyawnkarim
Copy link

Closing.

This enhancement was included with Release 1.7.

@shyawnkarim shyawnkarim modified the milestones: backlog, Release 1.7 Jan 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
Release 1.7
Development

No branches or pull requests
2 participants
@shyawnkarim @JeffGiroux