Skip to content
Tooling for efficient consistent infrastructure mirroring
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
config
README.md
create-release.sh
promote-repo.sh
sync-packages.sh

README.md

YUM mirrors tools

When operating infrastructure at large scale, ensuring that all servers maintain the same level of packages can be a tricky issue. As newer servers are added, they likely would get a newer version of kernel and other packages than the older machines that have not been updated yet.

This is a few scripts which help in managing mirroring infrastructure that can be staged (to be tested independently) then promoted. This will mean that you will have an always functional and common list of packages across all your fleet regardless when they were deployed.

There are a few tools that help with doin this already, however, they are too much (require client agents, or other tooling) for the simple task we're trying to accomplish here. This implies you'll be updating your /etc/yum.repos.d/* files via some existing configuration management, therefore no need for all that extra tooling.

This toolset include a configuration which can be used for deploying OpenStack Ansible. You can create your own configuration by looking at what the config/openstack-ansible.conf file looks like. Contributions are welcome for new 'profiles'.

Configuration

You can control a few parameters where you'll be storing your data. You can do this by overriding the following environment variables

  • MIRROR_LOCATION: Storage path for all downloaded packages (defaults to /var/lib/mirrors)
  • REPO_LOCATION: Path to store XML metadata for releases (defaults to /var/www/html). This should be exposed by a webserver.

In addition, there is a few profiles available by default for mirroring

  • base.conf: CentOS packages
  • openstack-ansible.conf: CentOS packages + OpenStack-Ansible dependencies

Usage

  1. Clone this repository on the machine that is going to be running the mirrors

    $ git clone https://github.com/vexxhost/yum-mirror-tools /opt/yum-mirror-tools
  2. Install dependencies (tooling to speed things up & create repositories). You should also setup an HTTP server which will be hosting your REPO_LOCATION.

    $ yum -y install createrepo httpd
    $ rm -rfv /etc/httpd/conf.d/welcome.conf
    $ systemctl enable httpd
    $ systemctl start httpd
  3. You can start the initial mirror (this will take quite sometime depending on your connection speed). In this example, we're using the openstack-ansible.conf profile.

    $ /opt/yum-mirror-tools/sync-packages.sh config/openstack-ansible.conf

    The tool does not currently support running multiple configurations at once, patches welcome!

  4. At this stage, you do not have a usable repository that you can point your users to. You need to make your first release of repositories. This shows an example of releasing your 1.0.0 repository.

    $ /opt/yum-mirror-tools/create-release.sh 1.0.0

    You should now have an accessible, frozen set of packages which you can use inside your yum repository configuration at http://host/1.0.0/repoid. If you followed these instructions, you should be able to see all the repositories listed by opening that path in your browser.

  5. Once you've tested tha teverything is working by deploying that path against a subset of machines, you can now setup a symbolic link (that always changes) towards the latest tested release.

    The promote-repo.sh script has a few more things added there which shutdown the web service during the promotion to make sure it kills off any HTTP clients and forces them to either fail or retry with the nwe package list.

    $ /opt/yum-mirror-tools/promote-repo.sh 1.0.0
You can’t perform that action at this time.