Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

binding existing deployment: Timed out sending `get_state' #154

Closed
lvnilesh opened this Issue Apr 7, 2013 · 12 comments

Comments

Projects
None yet
3 participants

lvnilesh commented Apr 7, 2013

vcap@ip-10-245-24-211:~$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...

Director task 3

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment (00:00:00)
  binding resource pools (00:00:00)
  binding stemcells (00:00:00)
  binding templates (00:00:00)
  binding properties (00:00:00)
  binding unallocated VMs (00:00:00)
  binding instance networks (00:00:00)
Done                    9/9 00:00:00

Preparing package compilation
  finding packages to compile (00:00:00)
Done                    1/1 00:00:00

Compiling packages
  dea_ruby19/7: Invalid availability zone: [us-east-1a] (00:00:01)
  imagemagick/2: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_jvm7/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_ruby18/8: Invalid availability zone: [us-east-1a] (00:00:00)
  git/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node04/3: Invalid availability zone: [us-east-1a] (00:00:00)
  redis22/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node08/1: Invalid availability zone: [us-east-1a] (00:00:00)
  dea_jvm/3.1-dev: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node06/3: Invalid availability zone: [us-east-1a] (00:00:00)
Error                   10/30 00:00:01

Error 100: Invalid availability zone: [us-east-1a]

Task 3 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'
vcap@ip-10-245-24-211:~$

lvnilesh commented Apr 7, 2013

I edited /var/vcap/store/systems/production/deployments/production-core.yml and manually added this line.

     region: us-west-2

per comment by @mrdavidlaing StarkAndWayne#100 (comment)

and ran bosh cf deploy again to run into timeout issue.

vcap@ip-10-245-24-211:~$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Compiling deployment manifest...

Director task 4

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment: Timed out sending `get_state' to 509eaeca-2b4a-4c96-b486-fa25705a3e5a after 30 seconds (00:01:31)
Error                   3/9 00:01:31

Error 450002: Timed out sending `get_state' to 509eaeca-2b4a-4c96-b486-fa25705a3e5a after 30 seconds

Task 4 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'

lvnilesh commented Apr 8, 2013

I ran through another set of steps in us-east-1 and ran into similar issue. @drnic would you know what may be causing this time out issue?

vcap@domU-12-31-39-04-65-C4:~$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...

Director task 3

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment (00:00:00)
  binding resource pools (00:00:00)
  binding stemcells (00:00:00)
  binding templates (00:00:00)
  binding properties (00:00:00)
  binding unallocated VMs (00:00:00)
---
  binding instance networks (00:00:00)
Done                9/9 00:00:00

Preparing package compilation
  finding packages to compile (00:00:00)
Done                1/1 00:00:00

Compiling packages
  imagemagick/2: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_ruby19/7: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_jvm7/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_ruby18/8: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node04/3: Invalid availability zone: [us-east-1a] (00:00:01)
  git/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_jvm/3.1-dev: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node06/3: Invalid availability zone: [us-east-1a] (00:00:01)
  redis22/1: Invalid availability zone: [us-east-1a] (00:00:01)
  dea_node08/1: Invalid availability zone: [us-east-1a] (00:00:01)
Error               10/30 00:00:01

Error 100: Invalid availability zone: [us-east-1a]

Task 3 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'
vcap@domU-12-31-39-04-65-C4:~$ vi /var/vcap/store/systems/production/deployments/production-core.yml
vcap@domU-12-31-39-04-65-C4:~$ bosh cf deployCurrent CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Compiling deployment manifest...

Director task 4

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment: Timed out sending `get_state' to 1c0b01ce-3345-4a3d-a7a7-dac3565b1b43 after 30 seconds (00:01:30)
Error               3/9 00:01:30

Error 450002: Timed out sending `get_state' to 1c0b01ce-3345-4a3d-a7a7-dac3565b1b43 after 30 seconds

Task 4 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'
Contributor

drnic commented Apr 8, 2013

@pmenglund ever seen this "Invalid availability zone: [us-east-1a]" issue before? Is it fixed in 1.5.0.pre?

Yes, ive seen that before - the zone could be blocked for that account. They do that every now and then...

lvnilesh commented Apr 8, 2013

I tried us-west-2 in StarkAndWayne#154 (comment)
and us-east-1 in StarkAndWayne#154 (comment)

Contributor

drnic commented Apr 8, 2013

Is 1.5.0 doing more to retry and use alternate zones? Or does it require human intervention to force it to use specific zones still?

Contributor

drnic commented Apr 8, 2013

@lvnilesh life doesn't get much better in other regions (as of today) - eventually you'll hit a bosh-cloudfoundry bug that doesn't support that region.

Sorry!!

Can I suggest you try the https://github.com/StarkAndWayne/bosh-bootstrap/tree/bosh-from-jenkins-builds branch and use bosh-bootstrap deploy --edge-prebuilt? See the bosh-users mailing list for my email on this.

Contributor

drnic commented Apr 8, 2013

@lvnilesh hopefully I'll make progress on bosh-cloudfoundry this week to make it work with the new bosh 1.5.0 & the new light stemcell (another pre-built AMI, yay!)

lvnilesh commented Apr 8, 2013

You mean gem uninstall bosh-bootstrap and git checkout bosh-from-jenkins-builds and then bundle install and then run bin/bosh-bootstrap? or is there a better way?

Region == us-east-1
Availability zone == us-east-1a

You can still use the region us-east-1, but not the zone us-east-1a. Try changing the az for your compilation VMs...

Contributor

drnic commented Apr 8, 2013

@lvnilesh that sounds about right

Contributor

drnic commented Jul 18, 2013

Closing and hoping this has been resolved upstream in bosh. If not then please please re-open for v0.7 (as I'm sure we'll have the same issues again there).

@drnic drnic closed this Jul 18, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment