Unreachable NATS leads to unreasonable amount of HTTP metadata service requests

[voelzmo]

When for some reason the BOSH NATS isn't available (Director update, network problems, etc), the agent exits and re-starts every few seconds. Reason is that heartbeats cannot be sent to the HM.
Here is an example from the agent logs that we see every few seconds:

2017-02-14_13:15:41.94560 [main] 2017/02/14 13:15:41 DEBUG - Starting agent
2017-02-14_13:15:41.94565 [File System] 2017/02/14 13:15:41 DEBUG - Reading file /var/vcap/bosh/agent.json
2017-02-14_13:15:41.94566 [File System] 2017/02/14 13:15:41 DEBUG - Read content
2017-02-14_13:15:41.94566 ********************
--
2017-02-14_13:15:55.06415 [Cmd Runner] 2017/02/14 13:15:55 DEBUG - Stdout:
2017-02-14_13:15:55.06415 [Cmd Runner] 2017/02/14 13:15:55 DEBUG - Stderr: SIOCDARP(dontpub): Network is unreachable
2017-02-14_13:15:55.06416 [Cmd Runner] 2017/02/14 13:15:55 DEBUG - Successful: false (255)
2017-02-14_13:15:55.06416 [NATS Handler] 2017/02/14 13:15:55 ERROR - Cleaning ip-mac address cache for: 192.168.1.11
2017-02-14_13:15:55.06617 [main] 2017/02/14 13:15:55 ERROR - App run Running agent: Message Bus Handler: Starting nats handler: Connecting: dial tcp 192.168.1.11:4222: getsockopt: connection refused
2017-02-14_13:15:55.06618 [main] 2017/02/14 13:15:55 ERROR - Agent exited with error: Running agent: Message Bus Handler: Starting nats handler: Connecting: dial tcp 192.168.1.11:4222: getsockopt: connection refused

Each agent startup makes 4 (or 5?) calls to the metadata service, which makes up for a pretty big amount for huge CF installations using HTTP metadata service: Multiply that by VMs and by Director downtime during an bosh-init update.

I'm open for suggestions how to approach this issue. Possible workarounds are:

• using config-drive instead of HTTP metadata service (disk reads are probably less of a problem than HTTP access). However, there might be other reasons why people chose HTTP metadata service over config-drive.
• Installing NATS separately from the Director to reduce downtime. Possibly clustered? Not sure what the state is here?
• Some kind of exponential backoff in the agent instead of just exiting when NATSHandler cannot connect. This would avoid the source of frequent metadata access due to NATS being not available
• something else?

I'd prefer the exponential backoff solution, what do you think?

[voelzmo]

@cppforlife care to elaborate on the "planned-enhancement" part? Which part are you planning for? :)

[aashah]

Hello,

It looks like we have not responded to this in a reasonable time, and unfortunately a year has passed. Given this, we are going to close the issue, but please feel free to re-open the issue. In doing so, a new Pivotal Tracker story will be generated, and we can revisit this.

Thanks again for submitting this, and apologies for closing without a proper resolution at this moment.

BOSH Systems Team - @aashah @luan @jaresty

[voelzmo]

Hey @aashah,

Thanks for replying! While the original issue still exists, we have found a workaround in our environments to keep the Director downtime during updates as small as possible, thereby mitigating this.

I'll leave it up to you if you want to have this issue for documenting this behavior in case someone analyzes their network traffic or even look at one of the possible solution suggestions above – or if this should remain closed.

[aashah]

Seems reasonable to keep this open given the original issue remains.

Side-question: What was your workaround?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/164934963

The labels on this github issue will be updated when the story is started.

[voelzmo]

We create-env an 'outer director' which is just used to install and 'inner director'. As with the other regular bosh deployments, the inner director is only down on stemcell updates, not on every configuration and release change as with create-env. That was good enough for us at this time.

[edwardstudy]

Hi, any update about this IMP? I just saw tracker removed this story.

[bosh-admin-bot]

This issue was marked as Stale because it has been open for 21 days without any activity. If no activity takes place in the coming 7 days it will automatically be close. To prevent this from happening remove the Stale label or comment below.