Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Convert lxc_hosts role to use simple download URL
For a very long time we've been parsing and using the lxc images as provided by upstream lxc. While these images are functional there are by no means optimal. In general they're quite a bit larger than they need to be and contian a lot of little sharp edges that have cut us over the years. This change removes all of the lxc image cache parsing and meta-data linking and simply downloads the rootfs a given url. To maintain compatibility with the legacy images a script has been created to parse the image index and return the legacy image url. The result of this change: * Access to smaller more optimal base image which is well known by the corresponding communities. * Deployers now have the ability to set and forget the download url for an internal image instead of having to create a cache infrastructure compatible with the lxc download template. * Any rootfs tarball will work as an image. * Fewer tasks are executed and less memory is consumed resulting in faster deployment times. * The base cache has a uniform meta-data setup giving all container types the same access to config, devices, and templating. Change-Id: I1775e775bbb7fe86bdffdd8296c2cff5ebc5bac8 Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
- Loading branch information
Showing
16 changed files
with
231 additions
and
108 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
--- | ||
features: | ||
- The option ``lxc_hosts_container_image_url`` has been added allowing | ||
deployers to define their base image url to whatever it needs to be | ||
removing the requirement for operators to maintain an internal LXC | ||
index in the event they want to host a private repository. | ||
- The option ``lxc_hosts_container_image_download_legacy`` has been | ||
added allowing a deployer to enable the use of the legacy lxc image | ||
repository. This option is a Boolean and has a default of **false**. | ||
deprecations: | ||
- The variable ``lxc_image_cache_server_mirrors`` has been deprecated | ||
in the "lxc_hosts" role. This option has been replaced by the static | ||
variable ``lxc_hosts_container_image_url``. This variable will continue | ||
to function as a single element list allowing existing automation to | ||
function when in legacy image mode but should not be considered in use | ||
by default. | ||
- The variable ``lxc_image_cache_server`` has been deprecated in the | ||
``lxc_hosts`` role. This option has been replaced by the static | ||
variable ``lxc_hosts_container_image_url``. | ||
other: | ||
- The use of **images.linuxcontainers.org** is no longer required. While | ||
the images provided by that build system are perfectly functional they | ||
have been less than optimal in a lot ways for a very long time. The | ||
``lxc_hosts`` role will now pull a base image from the upstream distro | ||
being deployed. If a deployer wishes to continue using the images from | ||
**images.linuxcontainers.org** they are welcome to but it is no longer | ||
forced. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,120 @@ | ||
#!/usr/bin/env python | ||
|
||
# Copyright 2017, Rackspace US, Inc. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
try: | ||
import httplib | ||
except ImportError: | ||
import http.client as httplib | ||
import ssl | ||
try: | ||
import urlparse | ||
except ImportError: | ||
import urllib.parse as urlparse | ||
import sys | ||
|
||
|
||
__DOC__ = """ | ||
USAGE: | ||
This script will take arguments passed into it to discover the url | ||
of a given container image using the distro, release, architecture, | ||
and variant as the parameters. | ||
EXAMPLE: | ||
# python lxc-image-fetch-url ubuntu xenial amd64 default | ||
""" | ||
|
||
LXC_CACHE_SERVER = '{{ lxc_image_cache_server_mirrors[0].strip() }}' | ||
LXC_INDEX = '{0}/meta/1.0/index-system'.format(LXC_CACHE_SERVER) | ||
|
||
|
||
def get_image_url(url, depth=0): | ||
if depth > 10: | ||
raise SystemExit('Too many redirects') | ||
|
||
url_path = urlparse.urlparse(url, allow_fragments=True) | ||
if url_path.scheme == 'https': | ||
conn = httplib.HTTPSConnection( | ||
host=url_path.netloc, | ||
context=ssl._create_unverified_context() | ||
) | ||
else: | ||
conn = httplib.HTTPConnection(host=url_path.netloc) | ||
|
||
try: | ||
conn.request('GET', url_path.path) | ||
except httplib.BadStatusLine: | ||
raise SystemExit('Connection Failure') | ||
else: | ||
resp = conn.getresponse() | ||
headers = dict(resp.getheaders()) | ||
check_redirect = headers.get('location', None) | ||
if not check_redirect: | ||
check_redirect = headers.get('Location', None) | ||
|
||
if check_redirect: | ||
depth += 1 | ||
return get_image_url( | ||
url=check_redirect, | ||
depth=depth | ||
) | ||
else: | ||
return ( | ||
str(url), | ||
resp.read().decode('UTF-8').splitlines() | ||
) | ||
finally: | ||
conn.close() | ||
|
||
|
||
def main(): | ||
try: | ||
distro = sys.argv[1] | ||
release = sys.argv[2] | ||
arch = sys.argv[3] | ||
variant = sys.argv[4] | ||
except IndexError: | ||
print('Missing argument, Please see the documentation.') | ||
raise SystemExit(__DOC__) | ||
|
||
_, meta = get_image_url(url=LXC_INDEX, depth=0) | ||
image_hint = '{0};{1};{2};{3}'.format( | ||
distro, | ||
release, | ||
arch, | ||
variant | ||
) | ||
|
||
images = sorted([i for i in meta if i.strip().startswith(image_hint)]) | ||
if not images: | ||
print(__DOC__ + '\nAvailable options:') | ||
for line in meta: | ||
print(';'.join(line.split(';')[:-2])) | ||
|
||
raise SystemExit( | ||
'No Image found with image hint "{0}"'.format(image_hint) | ||
) | ||
|
||
container_url = urlparse.urljoin( | ||
urlparse.urljoin( | ||
LXC_CACHE_SERVER, | ||
images[0].split(';')[-1] | ||
), | ||
'rootfs.tar.xz' | ||
) | ||
print(container_url) | ||
|
||
if __name__ == '__main__': | ||
main() |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
|
||
# System containers will require mount points for the following | ||
mkdir -p /dev | ||
mkdir -p /proc | ||
mkdir -p /sys/fs/cgroup | ||
|
||
# In order for the package manager to function /dev/null must exist. This is | ||
# is being run here because some images do not create /dev/null by default. | ||
if [[ ! -e /dev/null ]]; then | ||
mknod /dev/null c 1 3 | ||
chmod 0666 /dev/null | ||
fi | ||
|
||
# Create the directory where local facts will be stored | ||
mkdir -p /etc/ansible/facts.d | ||
|
||
# Create the cache resolvers | ||
rm /etc/resolv.conf || true | ||
{% for resolver in lxc_cache_prep_dns %} | ||
echo "nameserver {{ resolver }}" >> /etc/resolv.conf | ||
{% endfor %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.