bosh cf deploy assumes us-east-1 #100

marcus233 · 2013-02-12T07:27:58Z

i set my microbosh up on aws us west 2 (oregon). bosh cf prepare system installed bosh on us-west-2 as well.

upon running bosh cf deploy, i receive this error:

vcap@:~$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...

Director task 3

Preparing deployment
  binding deployment (00:00:00)                                                                     
  binding releases (00:00:00)                                                                       
  binding existing deployment (00:00:00)                                                            
  binding resource pools (00:00:00)                                                                 
  binding stemcells (00:00:00)                                                                      
  binding templates (00:00:00)                                                                      
  binding properties (00:00:00)                                                                     
  binding unallocated VMs (00:00:00)                                                                
  binding instance networks (00:00:00)                                                              
Done                    9/9 00:00:00                                                                

Preparing package compilation
  finding packages to compile (00:00:00)                                                            
Done                    1/1 00:00:00                                                                

Compiling packages
  dea_node04/3: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_node06/3: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_jvm7/1: Invalid availability zone: [us-east-1a] (00:00:01)                                    
  redis22/1: Invalid availability zone: [us-east-1a] (00:00:01)                                     
  dea_node08/1: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_ruby19/7: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  dea_jvm/3: Invalid availability zone: [us-east-1a] (00:00:01)                                     
  imagemagick/2: Invalid availability zone: [us-east-1a] (00:00:01)                                 
  dea_ruby18/8: Invalid availability zone: [us-east-1a] (00:00:01)                                  
  git/1: Invalid availability zone: [us-east-1a] (00:00:01)                                         
Error                   10/29 00:00:01                                                              

Error 100: Invalid availability zone: [us-east-1a]

Task 3 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
    from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:618:in `bosh_cmd'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:153:in `block in deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:151:in `each'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.5.0/lib/bosh/cli/commands/cf.rb:151:in `deploy'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
    from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
    from /usr/local/bin/bosh:23:in `load'
    from /usr/local/bin/bosh:23:in `<main>'

The text was updated successfully, but these errors were encountered:

mrdavidlaing · 2013-02-12T15:35:48Z

Same issue for me; but using bosh-cloudfoundry-0.5.1

drnic · 2013-02-12T15:45:54Z

Firstly, congrats on creating issue #100 :)

drnic · 2013-02-12T15:51:42Z

Can either of you paste in your /var/vcap/store/microboshes/deployments/*/micro_bosh.yml?

marcus233 · 2013-02-12T16:29:05Z

---
name: microbosh-aws-us-west-2
env:
  bosh:
    password: xxxxx
logging:
  level: DEBUG
  network:  type: dynamic
  vip: xxxxxx
resources:  
  persistent_disk: 16384  
  cloud_properties:  
      instance_type: m1.medium
cloud:
  plugin: aws
  properties:    aws:
      access_key_id: xxxxxxxxxxx
      secret_access_key: xxxxxxxxxxxxx
      region: us-west-2
      ec2_endpoint: ec2.us-west-2.amazonaws.com
      default_security_groups:
      - microbosh-aws-us-west-2
      default_key_name: microbosh-aws-us-west-2
      ec2_private_key: /home/vcap/.ssh/microbosh-aws-us-west-2.pem
apply_spec:
  agent:
    blobstore:
      address: xxxxxx
    nats:
      address: xxxxxx
  properties:
    aws_registry:
      address: xxxxxxxxxxx

drnic · 2013-02-12T16:32:46Z

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

mrdavidlaing · 2013-02-12T17:58:33Z

I've been grepping around looking for us-east-1, but I'm not having any
luck. Grr.

On 12 February 2013 16:32, Dr Nic Williams notifications@github.com wrote:

Ok thanks. This looks good to me; but perhaps it doesn't look good to
deployer gem.

Next, if you have time, look in bosh source (say deployer/ project) for
where it might override or default to us-east-1 region. It's ignoring the
region we're requesting!

Else I'll try to look later.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/100#issuecomment-13441281.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

drnic · 2013-02-12T18:13:27Z

In my deployment, which was in us-east-1, so it didn't fail, also put the
core/0 VM into us-east-1a. So perhaps the AZ is being specifically
overridden/defaulted.

marcus233 · 2013-02-12T18:53:50Z

i ended up changing DEFAULT_AVAILABILITY_ZONE in /usr/local/lib/ruby/gems/1.9.1/gems/bosh_aws_cpi-0.7.0/lib/cloud/aws/cloud.rb and got a bit further (then ran into another bug). perhaps this helps? i'll join the grep search in a moment.

mrdavidlaing · 2013-02-12T19:58:32Z

Could this be related to a similar default_availability_zone bug in bosh-bootstrap ?

Perhaps we should be adding

availability_zone: eu-west-1b to /var/vcap/store/systems/production/deployments/production-core.yml ?

drnic · 2013-02-12T20:01:49Z

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).

mrdavidlaing · 2013-02-12T21:29:51Z

Does it makes sense to split a single CF cluster over multiple AZs?

Wouldn't you run the risk of having a specific app allocated to a router in
one, DEA(s) in another and backend services services in yet another?

I'd assumed you would want them all as close together in network hop terms
as possible.

On Tuesday, February 12, 2013, Dr Nic Williams wrote:

I'm interested in starting to do that because some AWS users have AZs that
are not available to them; and the BOSH team (martin) won't support a fix
for that.

Also, if we start managing the AZ allocation manually, then we can start
supporting multi-AZ deployments (say 3 dea resource pools, one per AZ, and
allocate the DEA job VMs to them equally).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/100#issuecomment-13453533.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

mrdavidlaing · 2013-02-13T01:16:16Z

Aha! It looks like

/var/vcap/store/systems/production/deployments/production-core.yml is missing the region from the compilation::cloud_properties

compilation:
  cloud_properties:  
      instance_type: m1.medium
      region: us-west-2

You might also want to specify the availability_zone: us-west-2b explicitly, to ensure it ends up in the same AZ as your micro-bosh

Manually correcting your production-core.yml with that extra property allows the compilation step to proceed.

marcus233 · 2013-02-13T04:36:03Z

i have no section compilation_workers. adding region: us-west-2 to the cloud properties section under compilation doesn't get me past the error.

drnic · 2013-02-13T04:41:46Z

He's now talking about the deployment manifest at
/var/vcap/store/systems/NAME/deployments/...

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

marcus233 · 2013-02-13T04:47:59Z

This file?

vcap@:~$ cat /var/vcap/store/systems/production/deployments/production-core.yml 
--- 
name: production-core
director_uuid: xxxxxxxxxxxxx
release: 
  name: appcloud-staging
  version: latest
compilation: 
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
update: 
  canaries: 1
  canary_watch_time: 30000-150000
  update_watch_time: 30000-150000
  max_in_flight: 4
  max_errors: 1
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    security_groups: 
    - cloudfoundry-production
...

drnic · 2013-02-13T05:03:57Z

That's the one. compilation.cloud_properties is what Dave was referring to
above.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

drnic · 2013-02-13T06:49:50Z

The region will need to be discovered via the micro_bosh.yml; like we currently discover the AWS API keys (to offer to provide automatic IP address provisioning).

drnic · 2013-02-13T06:50:15Z

Specifically, via microbosh_config.rb #fog_connection_properties

drnic · 2013-02-13T08:13:19Z

A fix for this (compilation VMs needing a region) is in branch https://github.com/StarkAndWayne/bosh-cloudfoundry/tree/compilation_region

The branch is also working on adding region to all cloud_properties (core, dea, & service jobs).

drnic · 2013-02-13T08:22:09Z

@pmenglund @frodenas do know if it recently became necessary for cloud_properties to include a "region" key for AWS? Is it just for the compilation VMs or all resource pools? When was this introduced?

pmenglund · 2013-02-13T18:29:58Z

cloudfoundry/bosh@c8e3224

It is for all VMs, as you neet to pass it in the CPI options.

drnic · 2013-02-13T18:34:41Z

Does this mean that BOSH is going to become multi-region? Else why are we
now needing to provide the region information that the AWS CPI already had
from the micro_bosh.yml?

pmenglund · 2013-02-13T18:45:43Z

It might become multi region - I'm not making any promises on when...

You might want to have your micro bosh in one region and your bosh in another - my job is to make bosh configurable, yours is to pick good defaults ;)

drnic · 2013-02-13T18:49:17Z

Sorry, I was under the impression from last year that BOSH can live in and
provision in one region. I am very happy for the lifting of that constraint!

pmenglund · 2013-02-13T19:35:48Z

once I'm done with the fake stemcell I'm working on, you can - we are restricted to the current region when we upload create a stemcell, as it we need to attach an ebs volume to the current instance, but that will go away when I'm done (and we can just reference existing AMIs)

drnic · 2013-02-14T07:26:35Z

Ok, so if a user creates a custom stemcell or uses a normal stemcell (for which there is not a fleet of AWS AMI backing it), then we need to set the region to the region of the BOSH. If its AWS, and the chosen stemcell has a set of AMIs mapped to it, then we could prompt for which region to deploy to (based on whats available in the fake stemcell)

anderssv · 2013-02-20T15:12:23Z

I think you're aware of this but running bosh cf change deas 1 removed the additional information put in about zone, so you'll need to add it back.

mrdavidlaing · 2013-03-05T17:43:00Z

There are actually 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml that need to be corrected.

The following grep shows how the "corrected" sections of your production-core.yml should look:

vcap@ip-10-245-25-35:~$ grep cloud_properties /var/vcap/store/systems/production/deployments/production-core.yml -3
  workers: 10
  network: default
  reuse_compilation_vms: true
  cloud_properties: 
    instance_type: m1.medium
    region: us-west-2
    availability_zone: us-west-2b
--
networks: 
- name: default
  type: dynamic
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
    - cloudfoundry-production
- name: vip_network
  type: vip
  cloud_properties: 
    region: us-west-2
    availability_zone: us-west-2b
    security_groups: 
--
  stemcell: 
    name: bosh-stemcell
    version: 0.7.0
  cloud_properties: 
    instance_type: m1.large
    region: us-west-2
    availability_zone: us-west-2b

nand2 · 2013-03-08T15:08:54Z

I confirm the bug; and I confirm the fix in the comment above (eu-west-1 zone).

qch1843 · 2013-03-19T02:22:54Z

Hi,

I had same "invalid availability zone" issue and followed above comments and modified 4 instances of cloud_properties in /var/vcap/store/systems/production/deployments/production-core.yml and re-run the bosh cf deploy, following errors appears, any suggestions?

$ bosh cf deploy
Current CloudFoundry system is '/var/vcap/store/systems/production'
Deployment set to `/var/vcap/store/systems/production/deployments/production-core.yml'
bosh -n --color deploy
Getting deployment properties from director...
Compiling deployment manifest...

Director task 4

Preparing deployment
  binding deployment (00:00:00)
  binding releases (00:00:00)
  binding existing deployment: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds (00:01:30)
Error               3/9 00:01:30

Error 450002: Timed out sending `get_state' to b68ff6b0-6a2e-45f2-9108-422b9bd56a10 after 30 seconds

Task 4 error
/usr/local/lib/ruby/1.9.1/rake/file_utils.rb:53:in `block in create_shell_runner': Command failed with status (1): [bosh -n --color deploy...] (RuntimeError)
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `call'
        from /usr/local/lib/ruby/1.9.1/rake/file_utils.rb:45:in `sh'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:632:in `bosh_cmd'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:156:in `block in deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `each'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh-cloudfoundry-0.6.1/lib/bosh/cli/commands/cf.rb:154:in `deploy'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/command_handler.rb:57:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:61:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/lib/cli/runner.rb:18:in `run'
        from /usr/local/lib/ruby/gems/1.9.1/gems/bosh_cli-1.0.3/bin/bosh:16:in `<top (required)>'
        from /usr/local/bin/bosh:23:in `load'
        from /usr/local/bin/bosh:23:in `<main>'

anderssv · 2013-03-19T06:02:31Z

It first failed, then you corrected and ran again? I have never really understood how to recover from failures like this. You could try running bosh cloudcheck and choosing to delete the VM reference. If my understanding is correct this error stems from the fact that BOSH retains a reference to the VM it tries to create, but is unable to. So it will have to remove that reference before running again.

qch1843 · 2013-03-19T07:18:01Z

Yes, it failed first, and I modified the configuration file and run it again. And it failed with above error. With no other options, I have to start it over from very beginning (boot-bootstrap).

I will try the "bosh cloudcheck" next time...

Thanks!
Steven

qch1843 · 2013-03-19T07:37:30Z

BTW, in case of a failure, such as bosh cf deploy failed, how can I cleanup the previous deployment and start it from beginning (not from where it left off from last run)?

mrdavidlaing · 2013-03-19T11:39:30Z

bosh cloudcheck - and say yes fix any problems

Then bosh delete deployment production-core --force

This should leave you with just your inception vm and your micro bosh. You
can then bosh cf deploy again

On Tuesday, March 19, 2013, Steven wrote:

BTW, in case of a failure, such as bosh cf deploy failed, how can I
cleanup the previous deployment and start it from beginning (not from where
it left off from last run)?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/100#issuecomment-15100958
.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

mrdavidlaing mentioned this issue Feb 16, 2013

Bosh deploy fails on "Invalid availability zone" #110

Closed

anderssv mentioned this issue Feb 22, 2013

Issue Installing #113

Closed

drnic mentioned this issue Mar 18, 2013

"bosh cf deploy" failed with "Invalid availability zone: [us-east-1a]" #142

Closed

lvnilesh mentioned this issue Apr 7, 2013

binding existing deployment: Timed out sending `get_state' #154

Closed

drnic closed this Aug 2, 2013

cloudfoundry-community-attic / bosh-cloudfoundry Archived

bosh cf deploy assumes us-east-1 #100

bosh cf deploy assumes us-east-1 #100

marcus233 commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

drnic commented Feb 12, 2013

marcus233 commented Feb 12, 2013

drnic commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

marcus233 commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

mrdavidlaing commented Feb 13, 2013

marcus233 commented Feb 13, 2013

drnic commented Feb 13, 2013

marcus233 commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 14, 2013

anderssv commented Feb 20, 2013

mrdavidlaing commented Mar 5, 2013

nand2 commented Mar 8, 2013

qch1843 commented Mar 19, 2013

anderssv commented Mar 19, 2013

qch1843 commented Mar 19, 2013

qch1843 commented Mar 19, 2013

mrdavidlaing commented Mar 19, 2013

cloudfoundry-community-attic / bosh-cloudfoundry Archived

bosh cf deploy assumes us-east-1 #100

bosh cf deploy assumes us-east-1 #100

Comments

marcus233 commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

drnic commented Feb 12, 2013

marcus233 commented Feb 12, 2013

drnic commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

marcus233 commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

drnic commented Feb 12, 2013

mrdavidlaing commented Feb 12, 2013

mrdavidlaing commented Feb 13, 2013

marcus233 commented Feb 13, 2013

drnic commented Feb 13, 2013

marcus233 commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 13, 2013

pmenglund commented Feb 13, 2013

drnic commented Feb 14, 2013

anderssv commented Feb 20, 2013

mrdavidlaing commented Mar 5, 2013

nand2 commented Mar 8, 2013

qch1843 commented Mar 19, 2013

anderssv commented Mar 19, 2013

qch1843 commented Mar 19, 2013

qch1843 commented Mar 19, 2013

mrdavidlaing commented Mar 19, 2013