CC deploys successfully and then flaps when domains overlap

[Amit-PivotalLabs]

Unfortunately I don't remember what state my manifest was in when I had this problem, but I know that given my domain, system_domain, and apps_domains, I was seeing overlapping_domain validation errors in the CC logs. My BOSH deploy of CF succeeded, but then watching bosh vms I could see my api_zN jobs kept flapping between failing and starting.

It would be nice for it to fail fast, and stop the deploy, since it needs to be corrected before anything can work.

This is related to this issue: cloudfoundry/docs-deploying-cf#86

(This issue was driven out by these stories: https://www.pivotaltracker.com/story/show/114144105, https://www.pivotaltracker.com/story/show/85293086)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/114838603.

[dieucao]

@Amit-PivotalLabs I believe all three things should be fine to be the same domain from cloud controller's point of view even though we recommend people use a separate system domain than apps domain.

We can take a look into how it's possible you got into the overlapping domains state. As far as I know, overlapping domains would only occur with apps domains. Not 100% sure, but I thought apps domains in the manifest were only used as seed values for a new deployment.

[Amit-PivotalLabs]

You get an overlapping error if one domain extends the other, e.g.
domain.com and apps.domain.com. You're allowed to have domain.com and
domain.com, or you can have apps.domain.com and sys.domain.com. But you
can't have domain.com and apps.domain.com.

cloud_controller_ng/app/models/runtime/private_domain.rb

Line 46 in 39da42f

errors.add(:name, :overlapping_domain) if exclude_domains_from_same_org.filter(Sequel.like(:name, "%.#{name}")).count > 0

Whether or not that's the desired validation, the main problem was that CC
deploy wasn't failing when hitting this validation error. It succeeded,
the rest of the deploy kept going successfully, and then eventually CC got
itself into a flapping state. After the "successful" deploy, I couldn't
push an app, or even target the api, because CC was flapping by then.

On Wed, Mar 2, 2016 at 12:26 AM, Dieu Cao notifications@github.com wrote:

@Amit-PivotalLabs https://github.com/Amit-PivotalLabs I believe all
three things should be fine to be the same domain from cloud controller's
point of view even though we recommend people use a separate system domain
than apps domain.

We can take a look into how it's possible you got into the overlapping
domains state. As far as I know, overlapping domains would only occur with
apps domains. Not 100% sure, but I thought apps domains in the manifest
were only used as seed values for a new deployment.

—
Reply to this email directly or view it on GitHub
#552 (comment)
.

[dieucao]

Got it. So you had multiple domains specified in apps_domains that overlapped each other? I could see that running into an overlapping error.

[Amit-PivotalLabs]

It wasn't that actually. My system_domain was foo.com, and my apps_domain
was just one domain, apps.foo.com.

On Wed, Mar 2, 2016 at 12:45 AM, Dieu Cao notifications@github.com wrote:

Got it. So you had multiple domains specified in apps_domains that
overlapped each other? I could see that running into an overlapping error.

—
Reply to this email directly or view it on GitHub
#552 (comment)
.

[ameowlia]

We were able to reproduce this by editing the bosh manifest to include

system_domain: bosh-lite.com
app_domains: 
- app.bosh-lite.com

The deployment succeeds but api_z1 doesn't come up.

[ronakbanka]

This is failing because app domains shouldn't overlap with system domain,
but if you add the shared domains after the deployment then it won't create
any problem.

Not sure if it is a bug or a feature. But last week on cf-dev mailing list
there was a proposal for the same, to avoid this.

On Tuesday, 5 April 2016, Amelia Downs notifications@github.com wrote:

We were able to reproduce this by editing the bosh manifest to include

system_domain: bosh-lite.comapp_domains: - app.bosh-lite.com

The deployment succeeds but api_z1 doesn't come up.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#552 (comment)

[ameowlia]

@ronakbanka can you forward that email to me please? adowns@pivotal.io

Thanks,
Amelia

[Amit-PivotalLabs]

@adowns01 https://lists.cloudfoundry.org/archives/list/cf-dev@lists.cloudfoundry.org/thread/IGSGZBOQLTV5VAPNP76K7EAK3FRLNM2K/

But it does not solve the issue mentioned here. In fact, the example in "Solution 1" suffers the same overlapping domains problem. Furthermore, the issue I wanted to highlight is the operator UX. If domains are overlapping, the deploy should fail fast so I can check the logs and quickly figure out and fix my error. Instead the deploy succeeds, and only after the fact do I notice that the VM is flapping.

[rizwanreza]

I agree. I think a better solution here is to add validations on deployment, so it fails fast and notifies the user that the domains are overlapping.

[petergtz]

Fixed as a bosh pre-start script in cloudfoundry/capi-release@222711a

This should make it fail fast. The error message is in /var/vcap/sys/log/cloud_controller_ng/pre-start.stdout.log. Unfortunately there seems to be no way to make this visible on the CLI console.