cloud-init selects wrong mirror with dns server redirection

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #974509

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Precise)']
assignee = None
assignee_name = None
date_closed = 2012-10-01T18:24:12.328357+00:00
date_created = 2012-04-05T18:31:55.414711+00:00
date_fix_committed = 2012-08-03T18:57:31.912006+00:00
date_fix_released = 2012-10-01T18:24:12.328357+00:00
id = 974509
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/974509
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = fix_released
submitter = zulcss
submitter_name = Chuck Short
tags = ['verification-done']
duplicates = [1021395]

Launchpad user Chuck Short(zulcss) wrote on 2012-04-05T18:31:55.414711+00:00

=== Begin SRU Information ===
[Impact]
 * If a user launches an cloud-image in an environment where the DNS
   server does DNS redirection (also known as DNS hijacking), then the
   system will configure itself to use a mirror at
   http://ubuntu-mirror/ubuntu .

   This behavior was by design in cloud-init. It was intended to allow
   a cloud provider to set up a mirror at 'ubuntu-mirror' and have
   cloud-init select the mirror transparently. However, this causes
   failure if dns hijacking ins being used.

 * The fix is two fold:
   a.) cloud-init's code that checks for DNS entries is now protected
       by logic that detects the dns hijacking and does not consider
       such entries as valid.
   b.) the selection of the "search dns for 'ubuntu-mirror'" behavior
       has been disabled by default.

[Test Case]
 * download cloud image from cloud-images.ubuntu.com, and convert for use
   $ url="http://cloud-images.ubuntu.com/server/releases/precise/release-20121026.1/"
   $ wget "$url/ubuntu-12.04-server-cloudimg-i386-disk1.img" -O disk.img.orig
   $ qemu-img convert -O raw disk.img.orig disk.raw.dist

 * have some way to add 'ubuntu-mirror' to the dns for kvm guests (or
   just have a service provider that uses dns hijacking)

   I used dnsmasq on a server system, and can control this by adding entries
   to /etc/hosts. You need to be able to configure your system such
   that 'host ubuntu-mirror' returns something:
   $ host ubuntu-mirror
   ubuntu-mirror has address 192.168.1.1

 * boot kvm guest (cloud-localds from 12.10 cloud-utils)
$ qemu-img create -f qcow2 -b disk.raw.dist disk.img
   # this user-data just sets password so you can log in
   $ cat user-data.txt
   #cloud-config
   password: passw0rd
   chpasswd: { expire: False }
   ssh_pwauth: True

   $ cloud-localds seed.img user-data.txt
   $ kvm -m 512 -curses -drive file=seed.img,if=virtio
      -drive file=disk.img,if=virtio

 * login and see problem.
   looking at sources.list will show 'ubuntu-mirror' entry

[Regression Potential]
 * A regression is possible due to this designed change in behavior. If
   someone was expecting the 'ubuntu-mirror' mirror to be automatically
   located they will subsequently have to take different means to
   accomplish this. That can be either:
    a.) modifying the image to set 'apt_mirror_search_dns: true'
    b.) doing 'a' through user-data user-data
 * The change made in quantal was tested for regression as described in
   comment 5 below.

[Other Info]
 * The changes here also enable 2 other fixes
    * allowing region/availability-zone to be part of mirror (bug 1037727)
    * making mirror selection arch aware (bug #1028501)

=== End SRU Information ===

=== original bug report ===
Hi,

I have Rogers as an ISP in the great white north, and use their DNS servers. However they run DNS redirectors so that when you get a bad domain then it does bogus things to the hostname. Anyways this resolves in unresovalble hosts in my /etc/apt/sources.list when Im running an openstack instance.

ubuntu@server-5:/var/log$ host nov.ec2.archive.ubuntu.com
nov.ec2.archive.ubuntu.com has address 8.15.7.107
nov.ec2.archive.ubuntu.com has address 63.251.179.17
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)
Host nov.ec2.archive.ubuntu.com not found: 3(NXDOMAIN)

The console output is the following:

http://paste.ubuntu.com/916324/

If you have any questions please let me know.

Regards
chuck

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2012-05-09T19:36:14.073546+00:00

Status changed to 'Confirmed' because the bug affects multiple users.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-05-09T21:24:34.162131+00:00

The clear solution here is to use google dns (8.8.8.8) or opendns or some other dns server that does not provide bogus answers for non-existing domains.

That said, I think we'll likely be finding a better solution for discovery of a local mirror by an instance.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-08-03T18:57:31.041306+00:00

This was fixed in trunk in revno 612.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2012-08-03T19:00:13.087024+00:00

This bug was fixed in the package cloud-init - 0.7.0~bzr614-0ubuntu1

---

cloud-init (0.7.0~bzr614-0ubuntu1) quantal; urgency=low

• New upstream snapshot.
  • disable searching for 'ubuntu-mirror' in local dns to find a
    local mirror (LP: #974509)
  • emit the cloud-config event (LP: #1028674)
  • write timestamps to console on reboot and shutdown (LP: #1018554)
    -- Scott Moser smoser@ubuntu.com Fri, 03 Aug 2012 14:55:37 -0400

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-08-06T15:09:10.311912+00:00

This is fixed in cloud-init trunk and in the ubuntu quantal package listed in comment 4.
I've tested this by:

• setup:

  • have some way to add 'ubuntu-mirror' to the dns for kvm guests
    I used dnsmasq on a server system, and can control this by adding entries
    to /etc/hosts. You need to be able to configure your system such
    that 'host ubuntu-mirror' returns something:
    $ host ubuntu-mirror
    ubuntu-mirror has address 192.168.1.1
  • use make-seed-disk [1] to create a seed disk for cloud-init.
    $ cat user-data.txt
    #cloud-config
    password: passw0rd
    chpasswd: { expire: False }
    ssh_pwauth: True
    $ make-seed-disk seed.img user-data.txt

• verify issue in released version of precise or quantal alpha3:

  • add 'ubuntu-mirror' entry to dns
  • boot kvm guest
  • check mirror in /etc/apt/sources by:
    $ grep "^deb .*main$" /etc/apt/sources.list
  • quantal-alpha3: ubuntu-mirror is selected
  • precise 20120728: ubuntu-mirror is selected

• verify fix in quantal 20120804 or later:

  • add 'ubuntu-mirror' entry to dns
  • boot kvm guest of quantal 20120804 or later.
  • check mirror in /etc/apt/sources by:
    $ grep "^deb .*main$" /etc/apt/sources.list

• verify no regression:

  • remove 'ubuntu-mirror' entry from dns, boot, check /etc/apt/sources.list

• verify explicit enablement works:

  • add ubuntu-mirror
  • create seed disk with 'apt_mirror_search_dns: True' in user-data
  • verify that /etc/apt/sources.list shows ubuntu-mirror

• Notes, boot kvm with:
  kvm -m 512 -curses -drive file=seed.img,if=virtio
  -drive file=disk.img,if=virtio

--
[1] http://smoser.brickies.net/git/?p=tildabin.git;a=blob;f=make-seed-disk;hb=HEAD

[ubuntu-server-builder]

Launchpad user Steve Langasek(vorlon) wrote on 2012-12-10T23:05:50.399351+00:00

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-12-12T21:07:17.056339+00:00

following steps in description, i booted a cloud-image with new cloud-init and verified that it did not have 'ubuntu-mirror' in the /etc/apt/sources.list entries.

[ubuntu-server-builder]

Launchpad user Clint Byrum(clint-fewbar) wrote on 2013-01-08T19:16:32.303134+00:00

Hello Chuck, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Colin Watson(cjwatson) wrote on 2013-01-16T14:20:09.648043+00:00

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2013-01-16T14:20:17.525104+00:00

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.3

---

cloud-init (0.6.3-0ubuntu1.3) precise-proposed; urgency=low

• debian/patches/lp-1070345-landscape-restart-after-change.patch,
  debian/patches/lp-1066115-landscape-install-fix-perms.patch:
  fix missing or incorrect imports (LP: #1070345, LP: #1066115).

cloud-init (0.6.3-0ubuntu1.2) precise-proposed; urgency=low

• debian/patches/lp-978127-maas-oauth-fix-bad-clock.patch: fix usage of
  oauth in maas data source if local system has a bad clock (LP: #978127)
• debian/cloud-init.preinst: fix bug where user data scripts re-ran on
  upgrade from 10.04 versions (LP: #1049146)
• debian/patches/lp-974509-detect-dns-server-redirection.patch: detect dns
  server redirection and disable searching dns for a mirror named
  'ubuntu-mirror' (LP: #974509)
• debian/patches/lp-1018554-shutdown-message-to-console.patch: write a
  message to the console on system shutdown. (LP: #1018554)
• debian/patches/lp-1066115-landscape-install-fix-perms.patch: install
  landscape package if needed which will ensure proper permissions on config
  file (LP: #1066115).
• debian/patches/lp-1070345-landscape-restart-after-change.patch: restart
  landscape after modifying config (LP: #1070345)
• debian/patches/lp-1073077-zsh-workaround-for-locale_warn.patch: avoid
  warning when user's shell is zsh (LP: #1073077)
• debian/patches/rework-mirror-selection.patch: improve mirror selection by:
  • allowing region/availability-zone to be part of mirror (LP: #1037727)
  • making mirror selection arch aware (LP: #1028501)
  • allow specification of a security mirror (LP: #1006963)
    -- Scott Moser smoser@ubuntu.com Thu, 13 Dec 2012 12:16:56 -0500