Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

bosh micro AWS/VPC does not survive a power-off/power-on #92

Closed
james-masson opened this Issue Mar 22, 2013 · 1 comment

Comments

Projects
None yet
1 participant

Yesterday I deployed a new micro BOSH with git build gems ....

(last commit 8600696)

... to Amazon EC2 VPC, using ami-874c4af3

This worked, and happily accepted a stemcell+release upload, but is broken this morning when freshly powered up.

bosh "current" log on the system reports.

2013-03-22_09:33:42.61890 #[4226] INFO: Starting agent 0.6.4...
2013-03-22_09:33:42.61901 #[4226] INFO: Configuring agent...
2013-03-22_09:33:42.62622 #[4226] INFO: Configuring instance
2013-03-22_09:33:42.66613 /var/vcap/bosh/agent/lib/agent/infrastructure/aws/registry.rb:70:in rescue in get_json_from_url': Error requesting registry information #<Errno::ECONNREFUSED: Connection refused - connect(2) (http://localhost:25888)> (RuntimeError) 2013-03-22_09:33:42.66617 from /var/vcap/bosh/agent/lib/agent/infrastructure/aws/registry.rb:43:inget_json_from_url'
2013-03-22_09:33:42.66617 from /var/vcap/bosh/agent/lib/agent/infrastructure/aws/registry.rb:91:in get_settings' 2013-03-22_09:33:42.66618 from /var/vcap/bosh/agent/lib/agent/infrastructure/aws/settings.rb:33:inload_settings'
2013-03-22_09:33:42.66619 from /var/vcap/bosh/agent/lib/agent/infrastructure/aws.rb:10:in load_settings' 2013-03-22_09:33:42.66620 from /var/vcap/bosh/agent/lib/agent/bootstrap.rb:60:inload_settings'
2013-03-22_09:33:42.66621 from /var/vcap/bosh/agent/lib/agent/bootstrap.rb:34:in configure' 2013-03-22_09:33:42.66621 from /var/vcap/bosh/agent/lib/agent.rb:93:instart'
2013-03-22_09:33:42.66622 from /var/vcap/bosh/agent/lib/agent.rb:72:in run' 2013-03-22_09:33:42.66623 from /var/vcap/bosh/agent/bin/agent:97:in

'

According to the AWS UI, it has two disks available - sda and sdf, but neither seem mounted. So I guess it's missing a lot of what it needs to run.

Nothing seems to be listening on ports 25888 and 6868

I attempted a bosh micro deploy ami-874c4af3 --update to recover it, but got the following error.

Updating micro BOSH instance test-amazon/micro_bosh.yml' toUnknown Director' (type 'yes' to continue): yes
WARNING: Sinatra 1.2.x has reached its EOL. Please upgrade.

Prepare for update
stopping agent services (00:00:00)
Unmount disk |oooo | 1/5 00:00:00 ETA: 00:00:00/usr/local/lib64/ruby/gems/1.9.1/gems/agent_client-1.5.0.pre.3/lib/agent_client/http_client.rb:44:in rescue in request': Request details: (Bosh::Agent::Error) uri: http://vcap:b00tstrap@x.x.x.x:6868/agent payload: {"method":"list_disk","arguments":[],"reply_to":"bm-857f9fe5-8293-47c7-9b10-276c2ef8b2bd"} user: vcap password: b00tstrap Errno::ECONNREFUSED: Connection refused - connect(2) (http://x.x.x.x:6868) from /usr/local/lib64/ruby/gems/1.9.1/gems/agent_client-1.5.0.pre.3/lib/agent_client/http_client.rb:27:inrequest'
from /usr/local/lib64/ruby/gems/1.9.1/gems/agent_client-1.5.0.pre.3/lib/agent_client/http_client.rb:53:in post_json' from /usr/local/lib64/ruby/gems/1.9.1/gems/agent_client-1.5.0.pre.3/lib/agent_client/http_client.rb:21:inhandle_method'
from /usr/local/lib64/ruby/gems/1.9.1/gems/agent_client-1.5.0.pre.3/lib/agent_client/base.rb:17:in method_missing' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:258:indisk_info'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:241:in block in unmount_disk' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:84:instep'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:240:in unmount_disk' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:301:indetach_disk'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:177:in update' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:110:inblock in update_deployment'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:97:in with_lifecycle' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/deployer/instance_manager.rb:109:inupdate_deployment'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_deployer-1.5.0.pre.3/lib/bosh/cli/commands/micro.rb:171:in perform' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_cli-1.5.0.pre.3/lib/cli/command_handler.rb:57:inrun'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_cli-1.5.0.pre.3/lib/cli/runner.rb:59:in run' from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_cli-1.5.0.pre.3/lib/cli/runner.rb:18:inrun'
from /usr/local/lib64/ruby/gems/1.9.1/gems/bosh_cli-1.5.0.pre.3/bin/bosh:7:in <top (required)>' from /usr/local/bin/bosh:23:inload'
from /usr/local/bin/bosh:23:in `

'

I'm happy to let you have SSH access to the broken instance to debug if you wish.

regards

James M

this seems to work for me now on 1.5.0-pre-3

closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment