pre-start script fails more quickly than I think it needs to

[youngm]

Issue

Using CF Release 238. Bosh deploy fails at pre-start script if DB is only temporarily down as part of the same deploy.

Context

We deploy cf-mysql as part of the same deployment of CF. So quite often the DB is being restarted the same time the cloud_controller is starting. This will quite often cause the pre-start script to fail and by extension the deployment to fail more often then I'd like.

Steps to Reproduce

Try deploying CF when the database is down and notice how quickly the CC instance_group fails and how long you need to wait for everything else to fail before you can try again.

Expected result

I'd like to see CC pre-start script fail less often.

Current result

pre-start fails quickly making a re-deploy necessary and a longer deploy time.

Possible Fix

Perhaps the CC pre-start script could wait and retry a couple (configurable?) of times before deciding the DB is really down and cannot succeed. This would give time for the DB to come back up.

I don't think if kept short enough this would not have a negative impact on overall failure time in case of a real failure.

Thoughts?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/126275653

The labels on this github issue will be updated when the story is started.

[thausler786]

Hey @youngm,

Thanks for raising this issue! We've added two five-second retries to the migrations script in 48eea2b and have added a story to our icebox to make the retries configurable (https://www.pivotaltracker.com/story/show/127574901).

Hopefully your CC pre-start script will start failing less often! Let us know if your problem persists.

@thausler & @utako, CF CAPI team

[youngm]

Thanks! I'll let you know. Feel free to close if you want. I can reopen if it isn't good enough.