This repository has been archived by the owner. It is now read-only.

bosh cf deploy assumes us-east-1 #100

Closed
marcus233 opened this Issue Feb 12, 2013 · 34 comments

Comments

Projects
None yet
7 participants
@marcus233

i set my microbosh up on aws us west 2 (oregon). bosh cf prepare system installed bosh on us-west-2 as well.

upon running bosh cf deploy, i receive this error:

vcap@:~$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...

Director task 3

Preparing deployment
  binding deployment (00:00:00)                                                                     
  binding releases (00:00:00)                                                                       
  binding existing deployment (00:00:00)                                                            
  binding resource pools (00:00:00)                                                                 
  binding stemcells (00:00:00)                                                                      
  binding templates (00:00:00)                                                                      
  binding properties (00:00:00)                                                                     
  binding unallocated VMs (00:00:00)                                                                
  binding instance networks (00:00:00)                                                              
Done                    9/9 00:00:00                                                                

Preparing package compilation
  finding packages to compile (00:00:00)                                                            
Done                    1/1 00:00:00                                                                

Compiling packages
  dea_node04/3: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_node06/3: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_jvm7/1: Invalid availability zone: [us-east-1a] (00:00:01)                                    
  redis22/1: Invalid availability zone: [us-east-1a] (00:00:01)                                     
  dea_node08/1: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_ruby19/7: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_jvm/3: Invalid availability zone: [us-east-1a] (00:00:01)                                     
  imagemagick/2: Invalid availability zone: [us-east-1a] (00:00:01)                                 
  dea_ruby18/8: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  git/1: Invalid availability zone: [us-east-1a] (00:00:01)                                         
Error                   10/29 00:00:01                                                              

Error 100: Invalid availability zone: [us-east-1a]

Task 3 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:618:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:153:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:151:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:151:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'
@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Feb 12, 2013

Contributor

Same issue for me; but using bosh-cloudfoundry-0.5.1

Contributor

mrdavidlaing commented Feb 12, 2013

Same issue for me; but using bosh-cloudfoundry-0.5.1

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 12, 2013

Contributor

Firstly, congrats on creating issue #100 :)

Contributor

drnic commented Feb 12, 2013

Firstly, congrats on creating issue #100 :)

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 12, 2013

Contributor

Can either of you paste in your /var/vcap/store/microboshes/deployments/*/micro_bosh.yml?

Contributor

drnic commented Feb 12, 2013

Can either of you paste in your /var/vcap/store/microboshes/deployments/*/micro_bosh.yml?

@marcus233

This comment has been minimized.

Show comment
Hide comment
@marcus233

marcus233 Feb 12, 2013

---
name: microbosh-aws-us-west-2
env:
  bosh:
    password: xxxxx
logging:
  level: DEBUG
  network:  type: dynamic
  vip: xxxxxx
resources:  
  persistent_disk: 16384  
  cloud_properties:  
      instance_type: m1.medium
cloud:
  plugin: aws
  properties:    aws:
      access_key_id: xxxxxxxxxxx
      secret_access_key: xxxxxxxxxxxxx
      region: us-west-2
      ec2_endpoint: ec2.us-west-2.amazonaws.com
      default_security_groups:
      - microbosh-aws-us-west-2
      default_key_name: microbosh-aws-us-west-2
      ec2_private_key: /home/vcap/.ssh/microbosh-aws-us-west-2.pem
apply_spec:
  agent:
    blobstore:
      address: xxxxxx
    nats:
      address: xxxxxx
  properties:
    aws_registry:
      address: xxxxxxxxxxx
---
name: microbosh-aws-us-west-2
env:
  bosh:
    password: xxxxx
logging:
  level: DEBUG
  network:  type: dynamic
  vip: xxxxxx
resources:  
  persistent_disk: 16384  
  cloud_properties:  
      instance_type: m1.medium
cloud:
  plugin: aws
  properties:    aws:
      access_key_id: xxxxxxxxxxx
      secret_access_key: xxxxxxxxxxxxx
      region: us-west-2
      ec2_endpoint: ec2.us-west-2.amazonaws.com
      default_security_groups:
      - microbosh-aws-us-west-2
      default_key_name: microbosh-aws-us-west-2
      ec2_private_key: /home/vcap/.ssh/microbosh-aws-us-west-2.pem
apply_spec:
  agent:
    blobstore:
      address: xxxxxx
    nats:
      address: xxxxxx
  properties:
    aws_registry:
      address: xxxxxxxxxxx
@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 12, 2013

Contributor

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Contributor

drnic commented Feb 12, 2013

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Feb 12, 2013

Contributor

I've been grepping around looking for us-east-1, but I'm not having any
luck. Grr.

On 12 February 2013 16:32, Dr Nic Williams notifications@github.com wrote:

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-13441281.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

Contributor

mrdavidlaing commented Feb 12, 2013

I've been grepping around looking for us-east-1, but I'm not having any
luck. Grr.

On 12 February 2013 16:32, Dr Nic Williams notifications@github.com wrote:

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-13441281.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 12, 2013

Contributor

In my deployment, which was in us-east-1, so it didn't fail, also put the
core/0 VM into us-east-1a. So perhaps the AZ is being specifically
overridden/defaulted.

Contributor

drnic commented Feb 12, 2013

In my deployment, which was in us-east-1, so it didn't fail, also put the
core/0 VM into us-east-1a. So perhaps the AZ is being specifically
overridden/defaulted.

@marcus233

This comment has been minimized.

Show comment
Hide comment
@marcus233

marcus233 Feb 12, 2013

i ended up changing DEFAULT_AVAILABILITY_ZONE in /usr/local/lib/ruby/gems/1.9.1/gems/bosh_aws_cpi-0.7.0/lib/cloud/aws/cloud.rb and got a bit further (then ran into another bug). perhaps this helps? i'll join the grep search in a moment.

i ended up changing DEFAULT_AVAILABILITY_ZONE in /usr/local/lib/ruby/gems/1.9.1/gems/bosh_aws_cpi-0.7.0/lib/cloud/aws/cloud.rb and got a bit further (then ran into another bug). perhaps this helps? i'll join the grep search in a moment.

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Feb 12, 2013

Contributor

Could this be related to a similar default_availability_zone bug in bosh-bootstrap ?

Perhaps we should be adding

availability_zone: eu-west-1b to /var/vcap/store/systems/production/deployments/production-core.yml ?

Contributor

mrdavidlaing commented Feb 12, 2013

Could this be related to a similar default_availability_zone bug in bosh-bootstrap ?

Perhaps we should be adding

availability_zone: eu-west-1b to /var/vcap/store/systems/production/deployments/production-core.yml ?

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 12, 2013

Contributor

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).

Contributor

drnic commented Feb 12, 2013

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Feb 12, 2013

Contributor

Does it makes sense to split a single CF cluster over multiple AZs?

Wouldn't you run the risk of having a specific app allocated to a router in
one, DEA(s) in another and backend services services in yet another?

I'd assumed you would want them all as close together in network hop terms
as possible.

On Tuesday, February 12, 2013, Dr Nic Williams wrote:

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-13453533.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

Contributor

mrdavidlaing commented Feb 12, 2013

Does it makes sense to split a single CF cluster over multiple AZs?

Wouldn't you run the risk of having a specific app allocated to a router in
one, DEA(s) in another and backend services services in yet another?

I'd assumed you would want them all as close together in network hop terms
as possible.

On Tuesday, February 12, 2013, Dr Nic Williams wrote:

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-13453533.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Feb 13, 2013

Contributor

Aha! It looks like

/var/vcap/store/systems/production/deployments/production-core.yml is missing the region from the compilation::cloud_properties

compilation:
  cloud_properties:  
      instance_type: m1.medium
      region: us-west-2

You might also want to specify the availability_zone: us-west-2b explicitly, to ensure it ends up in the same AZ as your micro-bosh

Manually correcting your production-core.yml with that extra property allows the compilation step to proceed.

Contributor

mrdavidlaing commented Feb 13, 2013

Aha! It looks like

/var/vcap/store/systems/production/deployments/production-core.yml is missing the region from the compilation::cloud_properties

compilation:
  cloud_properties:  
      instance_type: m1.medium
      region: us-west-2

You might also want to specify the availability_zone: us-west-2b explicitly, to ensure it ends up in the same AZ as your micro-bosh

Manually correcting your production-core.yml with that extra property allows the compilation step to proceed.

@marcus233

This comment has been minimized.

Show comment
Hide comment
@marcus233

marcus233 Feb 13, 2013

i have no section compilation_workers. adding region: us-west-2 to the cloud properties section under compilation doesn't get me past the error.

i have no section compilation_workers. adding region: us-west-2 to the cloud properties section under compilation doesn't get me past the error.

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

He's now talking about the deployment manifest at
/var/vcap/store/systems/NAME/deployments/...

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Contributor

drnic commented Feb 13, 2013

He's now talking about the deployment manifest at
/var/vcap/store/systems/NAME/deployments/...

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

@marcus233

This comment has been minimized.

Show comment
Hide comment
@marcus233

marcus233 Feb 13, 2013

This file?

vcap@:~$ cat /var/vcap/store/systems/production/deployments/production-core.yml 
--- 
name: production-core
director_uuid: xxxxxxxxxxxxx
release: 
  name: appcloud-staging
  version: latest
compilation: 
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
update: 
  canaries: 1
  canary_watch_time: 30000-150000
  update_watch_time: 30000-150000
  max_in_flight: 4
  max_errors: 1
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    security_groups: 
    - cloudfoundry-production
...

This file?

vcap@:~$ cat /var/vcap/store/systems/production/deployments/production-core.yml 
--- 
name: production-core
director_uuid: xxxxxxxxxxxxx
release: 
  name: appcloud-staging
  version: latest
compilation: 
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
update: 
  canaries: 1
  canary_watch_time: 30000-150000
  update_watch_time: 30000-150000
  max_in_flight: 4
  max_errors: 1
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    security_groups: 
    - cloudfoundry-production
...
@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

That's the one. compilation.cloud_properties is what Dave was referring to
above.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Contributor

drnic commented Feb 13, 2013

That's the one. compilation.cloud_properties is what Dave was referring to
above.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

The region will need to be discovered via the micro_bosh.yml; like we currently discover the AWS API keys (to offer to provide automatic IP address provisioning).

Contributor

drnic commented Feb 13, 2013

The region will need to be discovered via the micro_bosh.yml; like we currently discover the AWS API keys (to offer to provide automatic IP address provisioning).

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

Specifically, via microbosh_config.rb #fog_connection_properties

Contributor

drnic commented Feb 13, 2013

Specifically, via microbosh_config.rb #fog_connection_properties

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

A fix for this (compilation VMs needing a region) is in branch https://github.com/StarkAndWayne/bosh-cloudfoundry/tree/compilation_region

The branch is also working on adding region to all cloud_properties (core, dea, & service jobs).

Contributor

drnic commented Feb 13, 2013

A fix for this (compilation VMs needing a region) is in branch https://github.com/StarkAndWayne/bosh-cloudfoundry/tree/compilation_region

The branch is also working on adding region to all cloud_properties (core, dea, & service jobs).

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

@pmenglund @frodenas do know if it recently became necessary for cloud_properties to include a "region" key for AWS? Is it just for the compilation VMs or all resource pools? When was this introduced?

Contributor

drnic commented Feb 13, 2013

@pmenglund @frodenas do know if it recently became necessary for cloud_properties to include a "region" key for AWS? Is it just for the compilation VMs or all resource pools? When was this introduced?

@pmenglund

This comment has been minimized.

Show comment
Hide comment
@pmenglund

pmenglund Feb 13, 2013

cloudfoundry/bosh@c8e3224

It is for all VMs, as you neet to pass it in the CPI options.

cloudfoundry/bosh@c8e3224

It is for all VMs, as you neet to pass it in the CPI options.

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

Does this mean that BOSH is going to become multi-region? Else why are we
now needing to provide the region information that the AWS CPI already had
from the micro_bosh.yml?

Contributor

drnic commented Feb 13, 2013

Does this mean that BOSH is going to become multi-region? Else why are we
now needing to provide the region information that the AWS CPI already had
from the micro_bosh.yml?

@pmenglund

This comment has been minimized.

Show comment
Hide comment
@pmenglund

pmenglund Feb 13, 2013

It might become multi region - I'm not making any promises on when...

You might want to have your micro bosh in one region and your bosh in another - my job is to make bosh configurable, yours is to pick good defaults ;)

It might become multi region - I'm not making any promises on when...

You might want to have your micro bosh in one region and your bosh in another - my job is to make bosh configurable, yours is to pick good defaults ;)

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 13, 2013

Contributor

Sorry, I was under the impression from last year that BOSH can live in and
provision in one region. I am very happy for the lifting of that constraint!

Contributor

drnic commented Feb 13, 2013

Sorry, I was under the impression from last year that BOSH can live in and
provision in one region. I am very happy for the lifting of that constraint!

@pmenglund

This comment has been minimized.

Show comment
Hide comment
@pmenglund

pmenglund Feb 13, 2013

once I'm done with the fake stemcell I'm working on, you can - we are restricted to the current region when we upload create a stemcell, as it we need to attach an ebs volume to the current instance, but that will go away when I'm done (and we can just reference existing AMIs)

once I'm done with the fake stemcell I'm working on, you can - we are restricted to the current region when we upload create a stemcell, as it we need to attach an ebs volume to the current instance, but that will go away when I'm done (and we can just reference existing AMIs)

@drnic

This comment has been minimized.

Show comment
Hide comment
@drnic

drnic Feb 14, 2013

Contributor

Ok, so if a user creates a custom stemcell or uses a normal stemcell (for which there is not a fleet of AWS AMI backing it), then we need to set the region to the region of the BOSH. If its AWS, and the chosen stemcell has a set of AMIs mapped to it, then we could prompt for which region to deploy to (based on whats available in the fake stemcell)

Contributor

drnic commented Feb 14, 2013

Ok, so if a user creates a custom stemcell or uses a normal stemcell (for which there is not a fleet of AWS AMI backing it), then we need to set the region to the region of the BOSH. If its AWS, and the chosen stemcell has a set of AMIs mapped to it, then we could prompt for which region to deploy to (based on whats available in the fake stemcell)

@anderssv

This comment has been minimized.

Show comment
Hide comment
@anderssv

anderssv Feb 20, 2013

Contributor

I think you're aware of this but running bosh cf change deas 1 removed the additional information put in about zone, so you'll need to add it back.

Contributor

anderssv commented Feb 20, 2013

I think you're aware of this but running bosh cf change deas 1 removed the additional information put in about zone, so you'll need to add it back.

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Mar 5, 2013

Contributor

There are actually 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml that need to be corrected.

The following grep shows how the "corrected" sections of your production-core.yml should look:

vcap@ip-10-245-25-35:~$ grep cloud_properties /var/vcap/store/systems/production/deployments/production-core.yml -3
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
    region: us-west-2
    availability_zone: us-west-2b
--
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
    - cloudfoundry-production
- name: vip_network
  type: vip
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
--
  stemcell: 
    name: bosh-stemcell
    version: 0.7.0
  cloud_properties: 
    instance_type: m1.large
    region: us-west-2
    availability_zone: us-west-2b
Contributor

mrdavidlaing commented Mar 5, 2013

There are actually 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml that need to be corrected.

The following grep shows how the "corrected" sections of your production-core.yml should look:

vcap@ip-10-245-25-35:~$ grep cloud_properties /var/vcap/store/systems/production/deployments/production-core.yml -3
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
    region: us-west-2
    availability_zone: us-west-2b
--
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
    - cloudfoundry-production
- name: vip_network
  type: vip
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
--
  stemcell: 
    name: bosh-stemcell
    version: 0.7.0
  cloud_properties: 
    instance_type: m1.large
    region: us-west-2
    availability_zone: us-west-2b
@nand2

This comment has been minimized.

Show comment
Hide comment
@nand2

nand2 Mar 8, 2013

I confirm the bug; and I confirm the fix in the comment above (eu-west-1 zone).

nand2 commented Mar 8, 2013

I confirm the bug; and I confirm the fix in the comment above (eu-west-1 zone).

@qch1843

This comment has been minimized.

Show comment
Hide comment
@qch1843

qch1843 Mar 19, 2013

Hi,

I had same "invalid availability zone" issue and followed above comments and modified 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml and re-run the bosh cf deploy, following errors appears, any suggestions?

$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Compiling deployment manifest...

Director task 4

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds (00:01:30)
Error               3/9 00:01:30

Error 450002: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds

Task 4 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
        from /usr/local/bin/bosh:23:in `load'
        from /usr/local/bin/bosh:23:in `<main>'

qch1843 commented Mar 19, 2013

Hi,

I had same "invalid availability zone" issue and followed above comments and modified 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml and re-run the bosh cf deploy, following errors appears, any suggestions?

$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Compiling deployment manifest...

Director task 4

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds (00:01:30)
Error               3/9 00:01:30

Error 450002: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds

Task 4 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
        from /usr/local/bin/bosh:23:in `load'
        from /usr/local/bin/bosh:23:in `<main>'

@anderssv

This comment has been minimized.

Show comment
Hide comment
@anderssv

anderssv Mar 19, 2013

Contributor

It first failed, then you corrected and ran again? I have never really understood how to recover from failures like this. You could try running bosh cloudcheck and choosing to delete the VM reference. If my understanding is correct this error stems from the fact that BOSH retains a reference to the VM it tries to create, but is unable to. So it will have to remove that reference before running again.

Contributor

anderssv commented Mar 19, 2013

It first failed, then you corrected and ran again? I have never really understood how to recover from failures like this. You could try running bosh cloudcheck and choosing to delete the VM reference. If my understanding is correct this error stems from the fact that BOSH retains a reference to the VM it tries to create, but is unable to. So it will have to remove that reference before running again.

@qch1843

This comment has been minimized.

Show comment
Hide comment
@qch1843

qch1843 Mar 19, 2013

Yes, it failed first, and I modified the configuration file and run it again. And it failed with above error. With no other options, I have to start it over from very beginning (boot-bootstrap).

I will try the "bosh cloudcheck" next time...

Thanks!
Steven

qch1843 commented Mar 19, 2013

Yes, it failed first, and I modified the configuration file and run it again. And it failed with above error. With no other options, I have to start it over from very beginning (boot-bootstrap).

I will try the "bosh cloudcheck" next time...

Thanks!
Steven

@qch1843

This comment has been minimized.

Show comment
Hide comment
@qch1843

qch1843 Mar 19, 2013

BTW, in case of a failure, such as bosh cf deploy failed, how can I cleanup the previous deployment and start it from beginning (not from where it left off from last run)?

qch1843 commented Mar 19, 2013

BTW, in case of a failure, such as bosh cf deploy failed, how can I cleanup the previous deployment and start it from beginning (not from where it left off from last run)?

@mrdavidlaing

This comment has been minimized.

Show comment
Hide comment
@mrdavidlaing

mrdavidlaing Mar 19, 2013

Contributor

bosh cloudcheck - and say yes fix any problems

Then bosh delete deployment production-core --force

This should leave you with just your inception vm and your micro bosh. You
can then bosh cf deploy again

On Tuesday, March 19, 2013, Steven wrote:

BTW, in case of a failure, such as bosh cf deploy failed, how can I
cleanup the previous deployment and start it from beginning (not from where
it left off from last run)?


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-15100958
.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

Contributor

mrdavidlaing commented Mar 19, 2013

bosh cloudcheck - and say yes fix any problems

Then bosh delete deployment production-core --force

This should leave you with just your inception vm and your micro bosh. You
can then bosh cf deploy again

On Tuesday, March 19, 2013, Steven wrote:

BTW, in case of a failure, such as bosh cf deploy failed, how can I
cleanup the previous deployment and start it from beginning (not from where
it left off from last run)?


Reply to this email directly or view it on GitHubhttps://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100#issuecomment-15100958
.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.