Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add awx as an embedded ansible plugin #16205

Merged
merged 12 commits into from Dec 6, 2017

Conversation

@carbonin
Copy link
Member

carbonin commented Oct 13, 2017

This fixes #15975

This PR builds on #16168 by adding a new EmbeddedAnsible plugin which handles running AWX.

AWX is the upstream of Ansible Tower and is distributed in a collection of containers rather than rpms.

The DockerEmbeddedAnsible class manages these containers using the docker-api gem.

When the role is enabled on an upstream appliance or on a development machine, we will pull the latest images, and run the containers using the same credential management system as we would in the appliance case.

The class also handles setting up the container environment required. This is mostly a ruby version of the playbook role that the AWX project uses to install the containers without all the image build and PG bits.

@carbonin carbonin requested review from Fryguy and bdunne Oct 13, 2017
@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch Oct 18, 2017
Copy link
Member

gtanzillo left a comment

Very nice! 👍

lib/embedded_ansible/docker_embedded_ansible.rb Outdated
require 'docker'
Docker.validate_version!
rescue RuntimeError
false

This comment has been minimized.

Copy link
@gtanzillo

gtanzillo Nov 3, 2017

Member

Maybe we should log something here in case the error is not what we are expecting?

This comment has been minimized.

Copy link
@carbonin

carbonin Nov 17, 2017

Author Member

I wanted to avoid logging here because this could get called whenever we do EmbeddedAnsible.new which could be quite a lot in production.

I will look into if we can rescue a more specific error though.

@miq-bot

This comment has been minimized.

Copy link
Member

miq-bot commented Nov 6, 2017

This pull request is not mergeable. Please rebase and repush.

@miq-bot miq-bot added the unmergeable label Nov 6, 2017
@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch Nov 7, 2017
@miq-bot miq-bot removed the unmergeable label Nov 7, 2017
lib/embedded_ansible/docker_embedded_ansible.rb Outdated
end

def awx_task_image_name
"ansible/awx_task:latest"

This comment has been minimized.

Copy link
@carbonin

carbonin Nov 17, 2017

Author Member

Looking for suggestions around how to make this configurable.

Should I go for creating a new key in Settings?

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

👍 for Settings

@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch to f04352a Nov 17, 2017
@@ -43,9 +43,45 @@ def api_connection_raw(host, port)
)
end

def find_or_create_secret_key
miq_database.ansible_secret_key || miq_database.ansible_secret_key = SecureRandom.hex(16)

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

Will this be saved?

This comment has been minimized.

Copy link
@bdunne

bdunne Dec 6, 2017

Member

Can I persuade you to miq_database.ansible_secret_key ||= SecureRandom.hex(16)?

end

def database_configuration
@db_config ||= Rails.configuration.database_configuration[Rails.env]

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

Prefer ActiveRecord::Base.configurations. See #15269 for more info

def start
run_rabbitmq_container
run_memcached_container
sleep(15)

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

Is there something better that we could do? I assume we're waiting for rabbitmq and memcached to start, right?

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

Do the other containers have any of their own checks for these services?

This comment has been minimized.

Copy link
@carbonin

carbonin Nov 21, 2017

Author Member

I actually yanked this directly from the playbook in ansible/awx

Because these are not "services" it seems harder to check their status.
Maybe we can pull the ncat checks into ManageIQ/manageiq in some way and reuse those here and in the pod in a follow up?

expect(connection).to receive(:select_value).with("CREATE DATABASE awx OWNER \"awx\" ENCODING 'utf8'")

auth = subject.send(:find_or_create_database_authentication)
expect(auth.userid).to eq("awx")

This comment has been minimized.

Copy link
@bdunne

bdunne Nov 20, 2017

Member

You can use have_attributes to reduce the number of expectations.

end

URI::HTTPS.build(:host => host, :path => path).to_s

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 27, 2017

Member

Minor, but you can also code this up such that each path in the conditional creates a hash, with :scheme as one of the keys, then you can DRY up the URI, build, to_s bits down here into a single line.

auth
end

def generate_password

This comment has been minimized.

Copy link
@Fryguy

Fryguy Nov 27, 2017

Member

I can't tell if they aren't already, but these should probably be private methods from here down.

This comment has been minimized.

Copy link
@carbonin

carbonin Nov 28, 2017

Author Member

Yup, they're private starting at line 34

@Fryguy

This comment has been minimized.

Copy link
Member

Fryguy commented Nov 27, 2017

Looks good! Sorry about the delayed review.

@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch 2 times, most recently from e42f543 to 13891f9 Nov 28, 2017
@carbonin carbonin removed the wip label Nov 29, 2017
@carbonin carbonin changed the title [WIP] Add awx as an embedded ansible plugin Add awx as an embedded ansible plugin Nov 29, 2017
@carbonin carbonin changed the title Add awx as an embedded ansible plugin [WIP] Add awx as an embedded ansible plugin Nov 29, 2017
@carbonin

This comment has been minimized.

Copy link
Member Author

carbonin commented Nov 29, 2017

Leaving this as WIP until ManageIQ/manageiq-appliance-build#250 is merged which adds docker to the appliance.

@carbonin

This comment has been minimized.

Copy link
Member Author

carbonin commented Dec 4, 2017

end

def find_or_create_admin_authentication
miq_database.ansible_admin_authentication || miq_database.set_ansible_admin_authentication(:password => generate_password)

This comment has been minimized.

Copy link
@bdunne

bdunne Dec 6, 2017

Member

Can these also be moved to miq_database.ansible_admin_authentication= then collapsed to ||= like 52bab34#r155362349 ?

@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch from 74396f2 to 574a6c4 Dec 6, 2017
carbonin added 10 commits Oct 12, 2017
This allows the "fetch from the database or generate and save"
behavior to be shared across different embedded ansible platforms
This will run the containers which make up AWX (https://github.com/ansible/awx)
and configure our app to use that for the embedded ansible feature.

This class uses the docker-api gem to communicate with the locally
running docker daemon to pull and launch the containers.

We use port 54321 as the host port so that this can be used
seamlessly in place of ApplianceEmbeddedAnsible when ansible tower
is not installed locally
When we have "localhost" in our database configuration, we have
to change that to the local machine's IP on the docker NIC
This also adds stubs for all of the subclass availability in each of
the specs to avoid sporadic test failures depending on the order
the subclasses are evaluated for availability.
This sorts the subclasses and instantiates the first available one
This really just assumes that a dev environment isn't multi-appliance
and isn't fronted by our httpd configuration.

This means that we always go to localhost, use http over https and
hardcode the port and path.
This error will be raised when the containers are just started.
Every API end point during the initial migration will return an
html page rather than a json payload.

This accounts for that specific situation by assuming if we don't
get a valid json response the service is not ready to serve requests
@carbonin carbonin force-pushed the carbonin:add_awx_as_embedded_ansible_backend branch from 574a6c4 to b015723 Dec 6, 2017
@miq-bot

This comment has been minimized.

Copy link
Member

miq-bot commented Dec 6, 2017

Checked commits carbonin/manageiq@a754fa8~...b015723 with ruby 2.3.3, rubocop 0.47.1, haml-lint 0.20.0, and yamllint 1.10.0
10 files checked, 1 offense detected

app/models/embedded_ansible_worker/runner.rb

end
end

private_constant :DockerDaemon

This comment has been minimized.

Copy link
@bdunne

bdunne Dec 6, 2017

Member

:neckbeard: 💯


def docker_bridge_gateway
br = Docker::Network.get("bridge")
br.info["IPAM"]["Config"].first["Gateway"]

This comment has been minimized.

Copy link
@bdunne

bdunne Dec 6, 2017

Member

fetch_path FTW

@bdunne
bdunne approved these changes Dec 6, 2017
@bdunne bdunne merged commit 5d27b97 into ManageIQ:master Dec 6, 2017
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage decreased (-0.03%) to 71.47%
Details
@bdunne bdunne assigned bdunne and unassigned gtanzillo Dec 6, 2017
@carbonin carbonin deleted the carbonin:add_awx_as_embedded_ansible_backend branch Dec 6, 2017
simaishi added a commit that referenced this pull request Dec 11, 2017
…ackend

Add awx as an embedded ansible plugin
(cherry picked from commit 5d27b97)
@simaishi

This comment has been minimized.

Copy link
Contributor

simaishi commented Dec 11, 2017

Gaprindashvili backport details:

$ git log -1
commit 1c5c1589982d848788a55aad0105dfd9b73fb5e3
Author: Brandon Dunne <brandondunne@hotmail.com>
Date:   Wed Dec 6 17:52:22 2017 -0500

    Merge pull request #16205 from carbonin/add_awx_as_embedded_ansible_backend
    
    Add awx as an embedded ansible plugin
    (cherry picked from commit 5d27b9731bd9bb10bdf72a7058cd2d8ec128ce15)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.