AWS replace VM does not swap the ops manager private IPs

[datianshi]

Ops Manager has a private IP
Customer has VPN/Direct connect to the VPC
DNS is assigned to the private IP

When replace the VM the private IPs are not swapped

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[DonBower]

This is significant issue that needs to be addressed.
It prevents anyone using fixed IPs from using this pipeline.

[svrc]

the workaround i've had to do for this for now is to use a CNAME record from opsman.mydomain to ec2-xx-xx-xx-xx.compute-1.amazonaws.com to be able to access the OpsMan through the private network as AWS will do split horizon DNS resolution on the underlying ec2 DNS A record to the private IP

[datianshi]

@svrc-pivotal did not get you... after the new VM stood up... it will get a new ec2-xx-xx-xx-xx.compute-1.amazonaws.com. You run into the same issue that you have to change DNS records? I am thinking of have an ELB associate with the opsman. Then we can disassociate old VM and re-associate new VM through automation without waiting DNS records change.

[svrc]

@datianshi Let me clarify, I noticed this (and the Upgrade Pipelines) requires an EIP attached to Ops Manager. So I assigned the EIP to Ops Manager, 52.3.68.14. My Opsman is available at https://opsman.td.ycf.link, which was normally a DNS A record within my private network at 10.0.0.218. as I don't access this Ops Man from the Internet (I use a VPN).

Unfortunately now, EIPs are not normally accessible within the private network unless I setup some kind of Hairpin NAT with the EC2 NAT instance.

So what I did instead was:

1. Created a CNAME record for opsman.td.ycf.link to ec2-52-3-68-14.compute-1.amazonaws.com
2. This has the effect of allowing all internal users when they resolve opsman.td.ycf.link , it will CNAME resolve to ec2-52-3-68-14.compute-1.amazonaws.com , whose A record will automatically translate to 10.0.0.218 as AWS is using split-horizon DNS.

Now the Upgrade pipeline will work, as the underlying private IP can change at upgrade time, and internal users can access the Ops Man. The public internet can be blocked from access via Security Group or Network ACL policy. Obviously though a stable private IP is preferable.

[datianshi]

@svrc-pivotal I see.... So you are saying to create an EIP just for the split-horizon DNS purpose.

Another question is: What is the split Based on? Source IP address? How do we guarantee that the machines on premises will resolve to the private IP instead of public IP

[svrc]

@datianshi I only created the EIP because cliaas currently requires it :)

The split horizon is based AWS' internal DNS server policy, which is looking at the source IP. Normally I have a DNSmasq setup but it recurses to 10.0.0.2 for my VPC. If you use 8.8.8.8 instead this doesn't work.

[datianshi]

Then I guess this is not a solution for a real environment? I am confused.....

…

On Thu, May 11, 2017 at 11:17 AM, Stu Charlton ***@***.***> wrote: @datianshi <https://github.com/datianshi> I only created the EIP because cliaas currently requires it :) The split horizon is based AWS' internal DNS server policy, which is looking at the source IP. Normally I have a DNSmasq setup but it recurses to 10.0.0.2 for my VPC. If you use 8.8.8.8 instead this doesn't work. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABjiTCJqm-DJYuqp2sKIUybyrz5qq707ks5r41DVgaJpZM4NRDUK> .

[svrc]

What can I help explain?

It is a workaround that allows people with private AWS networks to use cliaas/upgrade pipelines as-is today until it can be patched to preserve a stable private IP. Most real environments this would work fine for.

[datianshi]

@svrc-pivotal The DNS is controlled by on premise DNS server (Not 10.0.0.2)
Assume AWS VPC is 10.0.0.0/24
on premise src IP is 10.10.0.5..... trying to access ops manager through VPN. When it resolve the DNS -> ec2-52-3-68-14.compute-1.amazonaws.com. Will AWS give the public EIP address or private IP address.... How to make sure all the IP address space in my on premise environment able to resolve to the private IP address space?

[svrc]

I don't know all the details of your VPC, but something has to forward requests for at least some sub domains to the internal AWS DNS service for this to work. This is a common setup on most direct-connected clouds (eg Azure has similar issues).

Firstly , DNS should always have a resolver in the same LAN as the client so it doesn't have to traverse the WAN for the UDP 53 request. Eg. a bind or dnsmasq server. Going over the WAN a for all DNS requests will lead to a world of pain .

Beyond that, you have a couple options

Option A:
Your local LAN DNS resolver should forward all public zone requests to the native cloud DNS server (eg 10.0.0.2) and only forward to an on-premise DNS server for private zones (ie. those not on the internet DNS).

This way all AWS DNS will resolve correctly per their rules here:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html
And
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_DHCP_Options.html#AmazonDNS

Option B:

Per the latter link, either your LAN DNS resolver or your on premise DNS could conditionally forward queries for "region-name.compute.internal" to the Amazon internal DNS server to ensure the DNS addresses resolve properly.

But this may all vary depending on your VPC settings, and you might as well just wait for Cliaas to get patched for a stable private IP...