Properties in spec file are lost when collocating jobs #57

Closed
mheath opened this Issue Feb 27, 2013 · 10 comments

Comments

Projects
None yet
7 participants

mheath commented Feb 27, 2013

I just tried to collocate cloud_controller_ng with another job (in my case I was collocating with NATS but that doesn't matter). When deploying ccng, Bosh errored out saying it couldn't find the property 'ccng.resource_pool.minimum_size'. Removing the collocated job from the equation fixed the problem.

After doing some additional tests, it became clear that properties declared in the job's spec file get lost whenever jobs are collocated.

Owner

mkocher commented Feb 27, 2013

Hmm, are you using a mixture of old and new style properties?

mheath commented Feb 27, 2013

What do you mean by "old and new style properties"? I'm deploying ccng as is in cf-release.

Member

drnic commented Feb 27, 2013

It's a bug (or user error) if core cf-release components cannot
be collocated.

Contributor

Amit-PivotalLabs commented Mar 29, 2013

Hey @mheath, if you compare cf-release/jobs/cloud_controller_ng/spec to cf-release/jobs/nats/spec you'll see the difference between "new style" (ccng) and "old style" (nats). Bascially the new style requires you to specify all the types of properties (and optional defaults) that the job templates will need, and makes them easier to access from the templates using the helper method p. I'll ask around if there's any good documentation on using the new style, in the meantime you can check out this thread from the bosh-users group:

https://groups.google.com/a/cloudfoundry.org/forum/?fromgroups=#!searchin/bosh-users/bosh$20properties$20revamp/bosh-users/FNnxwwwm2HY/U6Y3ZQu8RzkJ

This could be caused by trying to colocate two jobs where one has new style and the other has old style, but that's just a guess. Spinning up a highly collocated CF deployment is the next thing we're working on, so if it's a real bug we'll definitely have to fix it.

i have seen this too. tried to colocate UAA and CCNG. doesn't work because bosh doesn't give CCNG its default properties.

Contributor

mreider commented Apr 3, 2013

Ok. We talked about this a lot today and yesterday. Basically we'll have to convert jobs to use the new property format for anything in CF that needs to be colocated. I talked to @abic and he said it took him a day and a half to convert the BOSH Director over. He also said @kowshik was the first one to convert a kernel job over last year. So the first thing we need to do is to answer the question "What jobs might be colocated?" and then update those jobs to use the new format. Otherwise we just need to go through all jobs and update them, which is a major exercise, but important nonetheless.

I am also curious what @dsboulder and @kushmerick think about all this - since colocating services is important? Are they all just using old style properties so the problem never surfaced?

for sure services jobs don't spec their properties today.

it makes sense in smallish CFs to colocate service-gateways, so for sure all foobar_gateway jobs should be upgraded.

but IMHO it is a bad idea to colocate service nodes, except perhaps for toy smoke-test scenarios. the colocated services will step all over each other. not in the sense of (eg) writing to the same files [ie, the reason you can't colocate jobs that use vcap_registrar]. Rather, VMs can become full because service nodes assume they are the only consumers of disk/memory. One can argue we should remove this assumption [though we thought long & hard about how to achieve lights-out operation of each service and I don't think see an alternative]. in the meantime, we don't need to bother to upgrade any of the foobar_node_ng jobs.

Member

drnic commented Apr 3, 2013

Perhaps in a wardenized world, services could be colocated - but the
services wouldn't know about it, so technically they aren't colocated (I'm
almost confusing myself).

Nic

On Wed, Apr 3, 2013 at 11:22 AM, Dr Nic Williams drnicwilliams@gmail.comwrote:

FWIW bosh-cloudfoundry adheres to Nick's recommendation - service nodes
are on dedicated VMs; and not on the core/0 or shared VMs with other
service nodes.

Nic

On Wed, Apr 3, 2013 at 11:19 AM, kushmerick notifications@github.comwrote:

for sure services jobs don't spec their properties today.

it makes sense in smallish CFs to colocate service-gateways, so for sure
all foobar_gateway jobs should be upgraded.

but IMHO it is a bad idea to colocate service nodes, except perhaps for
toy smoke-test scenarios. the colocated services will step all over each
other. not in the sense of (eg) writing to the same files [ie, the reason
you can't colocate jobs that use vcap_registrar]. Rather, VMs can become
full because service nodes assume they are the only consumers of
disk/memory. One can argue we should remove this assumption [though we
thought long & hard about how to achieve lights-out operation of each
service and I don't think see an alternative]. in the meantime, we don't
need to bother to upgrade any of the foobar_node_ng jobs.


Reply to this email directly or view it on GitHubhttps://github.com/cloudfoundry/bosh/issues/57#issuecomment-15854118
.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Member

drnic commented Apr 3, 2013

FWIW bosh-cloudfoundry adheres to Nick's recommendation - service nodes are
on dedicated VMs; and not on the core/0 or shared VMs with other service
nodes.

Nic

On Wed, Apr 3, 2013 at 11:19 AM, kushmerick notifications@github.comwrote:

for sure services jobs don't spec their properties today.

it makes sense in smallish CFs to colocate service-gateways, so for sure
all foobar_gateway jobs should be upgraded.

but IMHO it is a bad idea to colocate service nodes, except perhaps for
toy smoke-test scenarios. the colocated services will step all over each
other. not in the sense of (eg) writing to the same files [ie, the reason
you can't colocate jobs that use vcap_registrar]. Rather, VMs can become
full because service nodes assume they are the only consumers of
disk/memory. One can argue we should remove this assumption [though we
thought long & hard about how to achieve lights-out operation of each
service and I don't think see an alternative]. in the meantime, we don't
need to bother to upgrade any of the foobar_node_ng jobs.


Reply to this email directly or view it on GitHubhttps://github.com/cloudfoundry/bosh/issues/57#issuecomment-15854118
.

Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic

Member

oppegard commented May 29, 2013

We've added code that will give a better error when trying to mix old-style properties with new-style properties. We also have tasks underway to convert jobs in cf-release to new-style properties.

oppegard closed this May 29, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment