diff --git a/.env b/.env index 827440c6aa6c..94b2b621ae2e 100644 --- a/.env +++ b/.env @@ -1,3 +1,3 @@ PYTHONUNBUFFERED=true SELENIUM_DOCKER_TAG=latest - +COMPOSE_PROJECT_NAME=tools diff --git a/.gitignore b/.gitignore index b0d0e0bcfac5..42275f9abb1e 100644 --- a/.gitignore +++ b/.gitignore @@ -36,7 +36,11 @@ awx/ui_next/.env.local awx/ui_next/instrumented rsyslog.pid tools/prometheus/data +tools/docker-compose/ansible/awx_dump.sql tools/docker-compose/Dockerfile +tools/docker-compose/_build +tools/docker-compose/_sources +tools/docker-compose/overrides/ # Tower setup playbook testing setup/test/roles/postgresql @@ -146,7 +150,6 @@ use_dev_supervisor.txt .idea/* *.unison.tmp *.# -/tools/docker-compose/overrides/ /awx/ui_next/.ui-built /Dockerfile /_build/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e311ecfa1c37..51d3334aa797 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -11,24 +11,15 @@ Have questions about this document or anything not covered here? Come chat with * [Prerequisites](#prerequisites) * [Docker](#docker) * [Docker compose](#docker-compose) - * [Node and npm](#node-and-npm) - * [Build the environment](#build-the-environment) + * [Frontend Development](#frontend-development) + * [Build and Run the Development Environment](#build-and-run-the-development-environment) * [Fork and clone the AWX repo](#fork-and-clone-the-awx-repo) - * [Create local settings](#create-local-settings) - * [Build the base image](#build-the-base-image) - * [Build the user interface](#build-the-user-interface) - * [Running the environment](#running-the-environment) - * [Start the containers](#start-the-containers) - * [Start from the container shell](#start-from-the-container-shell) - * [Post Build Steps](#post-build-steps) - * [Start a shell](#start-a-shell) - * [Create a superuser](#create-a-superuser) - * [Load the data](#load-the-data) - * [Building API Documentation](#build-api-documentation) + * [Building API Documentation](#building-api-documentation) * [Accessing the AWX web interface](#accessing-the-awx-web-interface) * [Purging containers and images](#purging-containers-and-images) * [What should I work on?](#what-should-i-work-on) * [Submitting Pull Requests](#submitting-pull-requests) +* [PR Checks run by Zuul](#pr-checks-run-by-zuul) * [Reporting Issues](#reporting-issues) ## Things to know prior to submitting code @@ -42,7 +33,7 @@ Have questions about this document or anything not covered here? Come chat with ## Setting up your development environment -The AWX development environment workflow and toolchain is based on Docker, and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also binds the local source tree into the development container, making it possible to observe and test changes in real time. +The AWX development environment workflow and toolchain uses Docker and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also bind-mounts the local source tree into the development container, making it possible to observe and test changes in real time. ### Prerequisites @@ -55,29 +46,19 @@ respectively. For Linux platforms, refer to the following from Docker: -**Fedora** +* **Fedora** - https://docs.docker.com/engine/installation/linux/docker-ce/fedora/ -> https://docs.docker.com/engine/installation/linux/docker-ce/fedora/ +* **CentOS** - https://docs.docker.com/engine/installation/linux/docker-ce/centos/ -**CentOS** +* **Ubuntu** - https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ -> https://docs.docker.com/engine/installation/linux/docker-ce/centos/ +* **Debian** - https://docs.docker.com/engine/installation/linux/docker-ce/debian/ -**Ubuntu** +* **Arch** - https://wiki.archlinux.org/index.php/Docker -> https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/ +#### Docker Compose -**Debian** - -> https://docs.docker.com/engine/installation/linux/docker-ce/debian/ - -**Arch** - -> https://wiki.archlinux.org/index.php/Docker - -#### Docker compose - -If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the Docker compose Python module separately, in which case you'll need to run the following: +If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the `docker-compose` Python module separately. ```bash (host)$ pip3 install docker-compose @@ -87,186 +68,15 @@ If you're not using Docker for Mac, or Docker for Windows, you may need, or choo See [the ui development documentation](awx/ui_next/CONTRIBUTING.md). - -### Build the environment - #### Fork and clone the AWX repo If you have not done so already, you'll need to fork the AWX repo on GitHub. For more on how to do this, see [Fork a Repo](https://help.github.com/articles/fork-a-repo/). -#### Create local settings - -AWX will import the file `awx/settings/local_settings.py` and combine it with defaults in `awx/settings/defaults.py`. This file is required for starting the development environment and startup will fail if it's not provided. - -An example is provided. Make a copy of it, and edit as needed (the defaults are usually fine): - -```bash -(host)$ cp awx/settings/local_settings.py.docker_compose awx/settings/local_settings.py -``` - -#### Build the base image - -The AWX base container image (defined in `tools/docker-compose/Dockerfile`) contains basic OS dependencies and symbolic links into the development environment that make running the services easy. - -Run the following to build the image: - -```bash -(host)$ make docker-compose-build -``` - -**NOTE** - -> The image will need to be rebuilt, if the Python requirements or OS dependencies change. - -Once the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows: - -```bash -(host)$ docker images - -REPOSITORY TAG IMAGE ID CREATED SIZE -ansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB -``` - -#### Build the user interface - -Run the following to build the AWX UI: - -```bash -(host) $ make ui-devel -``` -See [the ui development documentation](awx/ui/README.md) for more information on using the frontend development, build, and test tooling. - -### Running the environment - -#### Start the containers - -Start the development containers by running the following: - -```bash -(host)$ make docker-compose -``` - -The above utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the *awx* container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process. - -If you start a second terminal session, you can take a look at the running containers using the `docker ps` command. For example: - -```bash -# List running containers -(host)$ docker ps - -$ docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -44251b476f98 gcr.io/ansible-tower-engineering/awx_devel:devel "/entrypoint.sh /bin…" 27 seconds ago Up 23 seconds 0.0.0.0:6899->6899/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 22/tcp, 0.0.0.0:8888->8888/tcp tools_awx_run_9e820694d57e -40de380e3c2e redis:latest "docker-entrypoint.s…" 28 seconds ago Up 26 seconds -b66a506d3007 postgres:12 "docker-entrypoint.s…" 28 seconds ago Up 26 seconds 0.0.0.0:5432->5432/tcp tools_postgres_1 -``` -**NOTE** - -> The Makefile assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG=` in your environment. - -> For example, you might be working in a feature branch, but you want to run the containers using the `devel` image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose` - -##### Wait for migrations to complete - -The first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following: - -```bash -awx_1 | Operations to perform: -awx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic -awx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main -awx_1 | Synchronizing apps without migrations: -awx_1 | Creating tables... -awx_1 | Running deferred SQL... -awx_1 | Installing custom SQL... -awx_1 | Running migrations: -awx_1 | Rendering model states... DONE -awx_1 | Applying contenttypes.0001_initial... OK -awx_1 | Applying contenttypes.0002_remove_content_type_name... OK -awx_1 | Applying auth.0001_initial... OK -awx_1 | Applying auth.0002_alter_permission_name_max_length... OK -awx_1 | Applying auth.0003_alter_user_email_max_length... OK -awx_1 | Applying auth.0004_alter_user_username_opts... OK -awx_1 | Applying auth.0005_alter_user_last_login_null... OK -awx_1 | Applying auth.0006_require_contenttypes_0002... OK -awx_1 | Applying taggit.0001_initial... OK -awx_1 | Applying taggit.0002_auto_20150616_2121... OK -awx_1 | Applying main.0001_initial... OK -awx_1 | Applying main.0002_squashed_v300_release... OK -awx_1 | Applying main.0003_squashed_v300_v303_updates... OK -awx_1 | Applying main.0004_squashed_v310_release... OK -awx_1 | Applying conf.0001_initial... OK -awx_1 | Applying conf.0002_v310_copy_tower_settings... OK -... -``` - -Once migrations are completed, you can begin using AWX. - -#### Start from the container shell - -Often times you'll want to start the development environment without immediately starting all of the services in the *awx* container, and instead be taken directly to a shell. You can do this with the following: - -```bash -(host)$ make docker-compose-test -``` - -Using `docker exec`, this will create a session in the running *awx* container, and place you at a command prompt, where you can run shell commands inside the container. - -If you want to start and use the development environment, you'll first need to bootstrap it by running the following command: - -```bash -(container)# /usr/bin/bootstrap_development.sh -``` - -The above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it -will drop you back to the shell. - -In order to launch all developer services: - -```bash -(container)# /usr/bin/launch_awx.sh -``` - -`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't -need to call `bootstrap_development.sh` first. - - - -### Post Build Steps - -Before you can log in and use the system, you will need to create an admin user. Optionally, you may also want to load some demo data. - -##### Start a shell - -To create the admin user, and load demo data, you first need to start a shell session on the *awx* container. In a new terminal session, use the `docker exec` command as follows to start the shell session: - -```bash -(host)$ docker exec -it tools_awx_1 bash -``` -This creates a session in the *awx* containers, just as if you were using `ssh`, and allows you execute commands within the running container. - -##### Create an admin user - -Before you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command: - -```bash -(container)# awx-manage createsuperuser -``` -You will be prompted for a username, an email address, and a password, and you will be asked to confirm the password. The email address is not important, so just enter something that looks like an email address. Remember the username and password, as you will use them to log into the web interface for the first time. - -##### Load demo data - -You can optionally load some demo data. This will create a demo project, inventory, and job template. From within the container shell, run the following to load the data: - -```bash -(container)# awx-manage create_preload_data -``` - -**NOTE** +### Build and Run the Development Environment -> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate -this container, and thus the database, if the database schema changes in an upstream commit. +See the [README.md](./tools/docker-compose/README.md) for docs on how to build the awx_devel image and run the development environment. -##### Building API Documentation +### Building API Documentation AWX includes support for building [Swagger/OpenAPI documentation](https://swagger.io). To build the documentation locally, run: @@ -284,7 +94,7 @@ is an example of one such service. You can now log into the AWX web interface at [https://localhost:8043](https://localhost:8043), and access the API directly at [https://localhost:8043/api/](https://localhost:8043/api/). -To log in use the admin user and password you created above in [Create an admin user](#create-an-admin-user). +[Create an admin user](./tools/docker-compose/README.md#create-an-admin-user) if needed. ### Purging containers and images @@ -335,7 +145,7 @@ Sometimes it might take us a while to fully review your PR. We try to keep the ` All submitted PRs will have the linter and unit tests run against them via Zuul, and the status reported in the PR. -## PR Checks ran by Zuul +## PR Checks run by Zuul Zuul jobs for awx are defined in the [zuul-jobs](https://github.com/ansible/zuul-jobs) repo. Zuul runs the following checks that must pass: diff --git a/INSTALL.md b/INSTALL.md index d96c2a086e0c..1a63cc9b8975 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -12,7 +12,6 @@ This document provides a guide for installing AWX. + [System Requirements](#system-requirements) + [Choose a deployment platform](#choose-a-deployment-platform) + [Official vs Building Images](#official-vs-building-images) - * [Upgrading from previous versions](#upgrading-from-previous-versions) * [OpenShift](#openshift) + [Prerequisites](#prerequisites-1) + [Pre-install steps](#pre-install-steps) @@ -29,17 +28,6 @@ This document provides a guide for installing AWX. + [Post-install](#post-install-1) + [Accessing AWX](#accessing-awx-1) + [SSL Termination](#ssl-termination) - * [Docker-Compose](#docker-compose) - + [Prerequisites](#prerequisites-3) - + [Pre-install steps](#pre-install-steps-2) - - [Deploying to a remote host](#deploying-to-a-remote-host) - - [Inventory variables](#inventory-variables) - - [Docker registry](#docker-registry) - - [Proxy settings](#proxy-settings) - - [PostgreSQL](#postgresql-1) - + [Run the installer](#run-the-installer-2) - + [Post-install](#post-install-2) - + [Accessing AWX](#accessing-awx-2) - [Installing the AWX CLI](#installing-the-awx-cli) * [Building the CLI Documentation](#building-the-cli-documentation) @@ -83,10 +71,6 @@ Before you can run a deployment, you'll need the following installed in your loc - [GNU Make](https://www.gnu.org/software/make/) - [Git](https://git-scm.com/) Requires Version 1.8.4+ - Python 3.6+ -- [Node 14.x LTS version](https://nodejs.org/en/download/) - + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false` -- [NPM 6.x LTS](https://docs.npmjs.com/) - + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false` ### System Requirements @@ -100,14 +84,13 @@ The system that runs the AWX service will need to satisfy the following requirem ### Choose a deployment platform -We currently support running AWX as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. The remainder of this document will walk you through the process of building the images, and deploying them to either platform. +We currently support running AWX as a containerized application using Docker images deployed to either an OpenShift cluster or a Kubernetes cluster. The remainder of this document will walk you through the process of building the images, and deploying them to either platform. The [installer](./installer) directory contains an [inventory](./installer/inventory) file, and a playbook, [install.yml](./installer/install.yml). You'll begin by setting variables in the inventory file according to the platform you wish to use, and then you'll start the image build and deployment process by running the playbook. In the sections below, you'll find deployment details and instructions for each platform: - [OpenShift](#openshift) - [Kubernetes](#kubernetes) -- [Docker Compose](#docker-compose). ### Official vs Building Images @@ -130,28 +113,15 @@ If these variables are present then all deployments will use these hosted images > Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123 -*use_container_for_build* - -> Use a local distribution build container image for building the AWX package. This is helpful if you don't want to bother installing the build-time dependencies as it is taken care of already. - - -## Upgrading from previous versions - -Upgrading AWX involves rerunning the install playbook. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables. - -For convenience, you can create a file called `vars.yml`: +To build your own container use the `build.yml` playbook: ``` -admin_password: 'adminpass' -pg_password: 'pgpass' -secret_key: 'mysupersecret' +ansible-playbook tools/ansible/build.yml -e awx_version=test-build ``` -And pass it to the installer: +The resulting image will automatically be pushed to a registry if `docker_registry` is defined. + -``` -$ ansible-playbook -i inventory install.yml -e @vars.yml -``` ## OpenShift @@ -437,213 +407,6 @@ If your provider is able to allocate an IP Address from the Ingress controller t Unlike Openshift's `Route` the Kubernetes `Ingress` doesn't yet handle SSL termination. As such the default configuration will only expose AWX through HTTP on port 80. You are responsible for configuring SSL support until support is added (either to Kubernetes or AWX itself). -## Docker-Compose - -### Prerequisites - -- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the ``docker`` group, refer to the documentation for details) -- [docker-compose](https://pypi.org/project/docker-compose/) Python module. - + This also installs the `docker` Python module, which is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it. -- [Docker Compose](https://docs.docker.com/compose/install/). - -### Pre-install steps - -#### Deploying to a remote host - -By default, the delivered [installer/inventory](./installer/inventory) file will deploy AWX to the local host. It is possible, however, to deploy to a remote host. The [installer/install.yml](./installer/install.yml) playbook can be used to build images on the local host, and ship the built images to, and run deployment tasks on, a remote host. To do this, modify the [installer/inventory](./installer/inventory) file, by commenting out `localhost`, and adding the remote host. - -For example, suppose you wish to build images locally on your CI/CD host, and deploy them to a remote host named *awx-server*. To do this, add *awx-server* to the [installer/inventory](./installer/inventory) file, and comment out or remove `localhost`, as demonstrated by the following: - -```yaml -# localhost ansible_connection=local -awx-server - -[all:vars] -... -``` - -In the above example, image build tasks will be delegated to `localhost`, which is typically where the clone of the AWX project exists. Built images will be archived, copied to remote host, and imported into the remote Docker image cache. Tasks to start the AWX containers will then execute on the remote host. - -If you choose to use the official images then the remote host will be the one to pull those images. - -**Note** - -> You may also want to set additional variables to control how Ansible connects to the host. For more information about this, view [Behavioral Inventory Parameters](http://docs.ansible.com/ansible/latest/intro_inventory.html#id12). - -> As mentioned above, in [Prerequisites](#prerequisites-1), the prerequisites are required on the remote host. - -> When deploying to a remote host, the playbook does not execute tasks with the `become` option. For this reason, make sure the user that connects to the remote host has privileges to run the `docker` command. This typically means that non-privileged users need to be part of the `docker` group. - - -#### Inventory variables - -Before starting the install process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section: - -*admin_password* - -> Provide a strong password to prevent malicious logins after the installation. - -*postgres_data_dir* - -> If you're using the default PostgreSQL container (see [PostgreSQL](#postgresql-1) below), provide a path that can be mounted to the container, and where the database can be persisted. - -*host_port* - -> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. If undefined no port will be exposed. Defaults to *80*. - -*host_port_ssl* - -> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. If undefined no port will be exposed. Defaults to *443*, only works if you also set `ssl_certificate` (see below). - -*ssl_certificate* - -> Optionally, provide the path to a file that contains a certificate and its private key. This needs to be a .pem-file - -*docker_compose_dir* - -> When using docker-compose, the `docker-compose.yml` file will be created there (default `~/.awx/awxcompose`). - -*custom_venv_dir* - -> Adds the custom venv environments from the local host to be passed into the containers at install. - -*ca_trust_dir* - -> If you're using a non trusted CA, provide a path where the untrusted Certs are stored on your Host. - -#### Docker registry - -If you wish to tag and push built images to a Docker registry, set the following variables in the inventory file: - -*docker_registry* - -> IP address and port, or URL, for accessing a registry. - -*docker_registry_repository* - -> Namespace to use when pushing and pulling images to and from the registry. Defaults to *awx*. - -*docker_registry_username* - -> Username of the user that will push images to the registry. Defaults to *developer*. - -**Note** - -> These settings are ignored if using official images - - -#### Proxy settings - -*http_proxy* - -> IP address and port, or URL, for using an http_proxy. - -*https_proxy* - -> IP address and port, or URL, for using an https_proxy. - -*no_proxy* - -> Exclude IP address or URL from the proxy. - -#### PostgreSQL - -AWX requires access to a PostgreSQL database, and by default, one will be created and deployed in a container, and data will be persisted to a host volume. In this scenario, you must set the value of `postgres_data_dir` to a path that can be mounted to the container. When the container is stopped, the database files will still exist in the specified path. - -If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information. - -### Run the installer - -If you are not pushing images to a Docker registry, start the install by running the following: - -```bash -# Set the working directory to installer -$ cd installer - -# Run the Ansible playbook -$ ansible-playbook -i inventory install.yml -``` - -If you're pushing built images to a repository, then use the `-e` option to pass the registry password as follows, replacing *password* with the password of the username assigned to `docker_registry_username` (note that you will also need to remove `dockerhub_base` and `dockerhub_version` from the inventory file): - -```bash -# Set the working directory to installer -$ cd installer - -# Run the Ansible playbook -$ ansible-playbook -i inventory -e docker_registry_password=password install.yml -``` - -### Post-install - -After the playbook run completes, Docker starts a series of containers that provide the services that make up AWX. You can view the running containers using the `docker ps` command. - -If you're deploying using Docker Compose, container names will be prefixed by the name of the folder where the docker-compose.yml file is created (by default, `awx`). - -Immediately after the containers start, the *awx_task* container will perform required setup tasks, including database migrations. These tasks need to complete before the web interface can be accessed. To monitor the progress, you can follow the container's STDOUT by running the following: - -```bash -# Tail the awx_task log -$ docker logs -f awx_task -``` - -You will see output similar to the following: - -```bash -Using /etc/ansible/ansible.cfg as config file -127.0.0.1 | SUCCESS => { - "changed": false, - "db": "awx" -} -Operations to perform: - Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic - Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main -Synchronizing apps without migrations: - Creating tables... - Running deferred SQL... - Installing custom SQL... -Running migrations: - Rendering model states... DONE - Applying contenttypes.0001_initial... OK - Applying contenttypes.0002_remove_content_type_name... OK - Applying auth.0001_initial... OK - Applying auth.0002_alter_permission_name_max_length... OK - Applying auth.0003_alter_user_email_max_length... OK - Applying auth.0004_alter_user_username_opts... OK - Applying auth.0005_alter_user_last_login_null... OK - Applying auth.0006_require_contenttypes_0002... OK - Applying taggit.0001_initial... OK - Applying taggit.0002_auto_20150616_2121... OK - Applying main.0001_initial... OK -... -``` - -Once migrations complete, you will see the following log output, indicating that migrations have completed: - -```bash -Python 2.7.5 (default, Nov 6 2016, 00:28:07) -[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2 -Type "help", "copyright", "credits" or "license" for more information. -(InteractiveConsole) - ->>> ->>> Default organization added. -Demo Credential, Inventory, and Job Template added. -Successfully registered instance awx -(changed: True) -Creating instance group tower -Added instance awx to tower -(changed: True) -... -``` - -### Accessing AWX - -The AWX web server is accessible on the deployment host, using the *host_port* value set in the *inventory* file. The default URL is [http://localhost](http://localhost). - -You will prompted with a login dialog. The default administrator username is `admin`, and the password is `password`. - - # Installing the AWX CLI `awx` is the official command-line client for AWX. It: diff --git a/Makefile b/Makefile index 961d99f66b45..1e83ffb8aba9 100644 --- a/Makefile +++ b/Makefile @@ -25,6 +25,8 @@ SCL_PREFIX ?= CELERY_SCHEDULE_FILE ?= /var/lib/awx/beat.db DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering +DEVEL_IMAGE_NAME ?= $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) + # Python packages to install only from source (not from binary wheels) # Comma separated list SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio,pycurl @@ -64,7 +66,7 @@ I18N_FLAG_FILE = .i18n_built receiver test test_unit test_coverage coverage_html \ dev_build release_build release_clean sdist \ ui-docker-machine ui-docker ui-release ui-devel \ - ui-test ui-deps ui-test-ci VERSION + ui-test ui-deps ui-test-ci VERSION docker-compose-sources clean-tmp: rm -rf tmp/ @@ -511,6 +513,12 @@ ui-release: awx/ui_next/node_modules $(UI_BUILD_FLAG_FILE) ui-devel: awx/ui_next/node_modules @$(MAKE) -B $(UI_BUILD_FLAG_FILE) +ui-devel-instrumented: awx/ui_next/node_modules + $(NPM_BIN) --prefix awx/ui_next --loglevel warn run start-instrumented + +ui-devel-test: awx/ui_next/node_modules + $(NPM_BIN) --prefix awx/ui_next --loglevel warn run start + ui-zuul-lint-and-test: $(NPM_BIN) --prefix awx/ui_next install $(NPM_BIN) run --prefix awx/ui_next lint @@ -559,31 +567,31 @@ docker-auth: awx/projects: @mkdir -p $@ -# Docker isolated rampart -docker-compose-isolated: awx/projects - CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up - COMPOSE_UP_OPTS ?= -# Docker Compose Development environment -docker-compose: docker-auth awx/projects - CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml $(COMPOSE_UP_OPTS) up --no-recreate awx +docker-compose-sources: + ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/sources.yml \ + -e awx_image=$(DEV_DOCKER_TAG_BASE)/awx_devel \ + -e awx_image_tag=$(COMPOSE_TAG) + +docker-compose: docker-auth awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_UP_OPTS) up --no-recreate awx docker-compose-cluster: docker-auth awx/projects - CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up + docker-compose -f tools/docker-compose-cluster.yml up -docker-compose-credential-plugins: docker-auth awx/projects +docker-compose-credential-plugins: docker-auth awx/projects docker-compose-sources echo -e "\033[0;31mTo generate a CyberArk Conjur API key: docker exec -it tools_conjur_1 conjurctl account create quick-start\033[0m" - CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx + docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx -docker-compose-test: docker-auth awx/projects - cd tools && CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash +docker-compose-test: docker-auth awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx /bin/bash -docker-compose-runtest: awx/projects - cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh +docker-compose-runtest: awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx /start_tests.sh -docker-compose-build-swagger: awx/projects - cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports --no-deps awx /start_tests.sh swagger +docker-compose-build-swagger: awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports --no-deps awx /start_tests.sh swagger detect-schema-change: genschema curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json @@ -591,24 +599,14 @@ detect-schema-change: genschema diff -u -b reference-schema.json schema.json docker-compose-clean: awx/projects - cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf + docker-compose -f tools/docker-compose/_sources/docker-compose.yml rm -sf # Base development image build docker-compose-build: - ansible-playbook installer/dockerfile.yml -e build_dev=True - docker build -t ansible/awx_devel \ + ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True + docker build -t $(DEVEL_IMAGE_NAME) \ --build-arg BUILDKIT_INLINE_CACHE=1 \ --cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) . - docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) - #docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) - -# For use when developing on "isolated" AWX deployments -docker-compose-isolated-build: docker-compose-build - docker build -t ansible/awx_isolated \ - --build-arg BUILDKIT_INLINE_CACHE=1 \ - -f tools/docker-isolated/Dockerfile . - docker tag ansible/awx_isolated $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG) - #docker push $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG) docker-clean: $(foreach container_id,$(shell docker ps -f name=tools_awx -aq),docker stop $(container_id); docker rm -f $(container_id);) @@ -620,11 +618,11 @@ docker-clean-volumes: docker-compose-clean docker-refresh: docker-clean docker-compose # Docker Development Environment with Elastic Stack Connected -docker-compose-elk: docker-auth awx/projects - CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate +docker-compose-elk: docker-auth awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate -docker-compose-cluster-elk: docker-auth awx/projects - TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate +docker-compose-cluster-elk: docker-auth awx/projects docker-compose-sources + docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate prometheus: docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml @@ -643,11 +641,12 @@ psql-container: VERSION: @echo "awx: $(VERSION)" -Dockerfile: installer/roles/dockerfile/templates/Dockerfile.j2 - ansible-playbook installer/dockerfile.yml +Dockerfile: tools/ansible/roles/dockerfile/templates/Dockerfile.j2 + ansible-playbook tools/ansible/dockerfile.yml -Dockerfile.kube-dev: installer/roles/dockerfile/templates/Dockerfile.j2 - ansible-playbook installer/dockerfile.yml \ +# TODO: test kube-dev target after changing dockerfile.yml location - refer to https://github.com/ansible/awx/commit/7c8bd471980d26083d4c4e11067bb53730175496 +Dockerfile.kube-dev: tools/ansible/roles/dockerfile/templates/Dockerfile.j2 + ansible-playbook tools/ansible/dockerfile.yml \ -e dockerfile_name=Dockerfile.kube-dev \ -e kube_dev=True \ -e template_dest=_build_kube_dev diff --git a/awx/main/utils/external_logging.py b/awx/main/utils/external_logging.py index 403105edf48d..7213f6f7498f 100644 --- a/awx/main/utils/external_logging.py +++ b/awx/main/utils/external_logging.py @@ -116,7 +116,7 @@ def escape_quotes(x): def reconfigure_rsyslog(): tmpl = construct_rsyslog_conf_template() # Write config to a temp file then move it to preserve atomicity - with tempfile.TemporaryDirectory(prefix='rsyslog-conf-') as temp_dir: + with tempfile.TemporaryDirectory(dir='/var/lib/awx/rsyslog/', prefix='rsyslog-conf-') as temp_dir: path = temp_dir + '/rsyslog.conf.temp' with open(path, 'w') as f: os.chmod(path, 0o640) diff --git a/installer/install.yml b/installer/install.yml index fc36be770a2a..a0c6d71f90f2 100644 --- a/installer/install.yml +++ b/installer/install.yml @@ -3,7 +3,4 @@ hosts: all roles: - {role: check_vars} - - {role: image_build, when: "dockerhub_base is not defined"} - - {role: image_push, when: "docker_registry is defined and dockerhub_base is not defined"} - {role: kubernetes, when: "openshift_host is defined or kubernetes_context is defined"} - - {role: local_docker, when: "openshift_host is not defined and kubernetes_context is not defined"} diff --git a/installer/roles/check_vars/tasks/check_docker.yml b/installer/roles/check_vars/tasks/check_docker.yml deleted file mode 100644 index a08db3a6a058..000000000000 --- a/installer/roles/check_vars/tasks/check_docker.yml +++ /dev/null @@ -1,8 +0,0 @@ -# check_docker.yml ---- -- name: postgres_data_dir should be defined - assert: - that: - - postgres_data_dir is defined and postgres_data_dir != '' - msg: "Set the value of 'postgres_data_dir' in the inventory file." - when: pg_hostname is not defined or pg_hostname == '' diff --git a/installer/roles/check_vars/tasks/main.yml b/installer/roles/check_vars/tasks/main.yml index 61ffa1cb77ba..9ac079e560f9 100644 --- a/installer/roles/check_vars/tasks/main.yml +++ b/installer/roles/check_vars/tasks/main.yml @@ -8,6 +8,3 @@ - include_tasks: check_openshift.yml when: openshift_host is defined and openshift_host != '' - -- include_tasks: check_docker.yml - when: openshift_host is not defined or openshift_host == '' diff --git a/installer/roles/local_docker/defaults/main.yml b/installer/roles/local_docker/defaults/main.yml deleted file mode 100644 index 4b97d471968c..000000000000 --- a/installer/roles/local_docker/defaults/main.yml +++ /dev/null @@ -1,11 +0,0 @@ ---- -dockerhub_version: "{{ lookup('file', playbook_dir + '/../VERSION') }}" - -awx_image: "awx" -redis_image: "redis" - -postgresql_version: "12" -postgresql_image: "postgres:{{postgresql_version}}" - -compose_start_containers: true -upgrade_postgres: false diff --git a/installer/roles/local_docker/tasks/compose.yml b/installer/roles/local_docker/tasks/compose.yml deleted file mode 100644 index 59ba2623fffa..000000000000 --- a/installer/roles/local_docker/tasks/compose.yml +++ /dev/null @@ -1,74 +0,0 @@ ---- -- name: Create {{ docker_compose_dir }} directory - file: - path: "{{ docker_compose_dir }}" - state: directory - -- name: Create Redis socket directory - file: - path: "{{ docker_compose_dir }}/redis_socket" - state: directory - mode: 0777 - -- name: Create Docker Compose Configuration - template: - src: "{{ item.file }}.j2" - dest: "{{ docker_compose_dir }}/{{ item.file }}" - mode: "{{ item.mode }}" - loop: - - file: environment.sh - mode: "0600" - - file: credentials.py - mode: "0600" - - file: docker-compose.yml - mode: "0600" - - file: nginx.conf - mode: "0600" - - file: redis.conf - mode: "0664" - register: awx_compose_config - -- name: Render SECRET_KEY file - copy: - content: "{{ secret_key }}" - dest: "{{ docker_compose_dir }}/SECRET_KEY" - mode: 0600 - register: awx_secret_key - -- block: - - name: Remove AWX containers before migrating postgres so that the old postgres container does not get used - docker_compose: - project_src: "{{ docker_compose_dir }}" - state: absent - ignore_errors: true - - - name: Run migrations in task container - shell: docker-compose run --rm --service-ports task awx-manage migrate --no-input - args: - chdir: "{{ docker_compose_dir }}" - - - name: Start the containers - docker_compose: - project_src: "{{ docker_compose_dir }}" - restarted: "{{ awx_compose_config is changed or awx_secret_key is changed }}" - register: awx_compose_start - - - name: Update CA trust in awx_web container - command: docker exec awx_web '/usr/bin/update-ca-trust' - when: awx_compose_config.changed or awx_compose_start.changed - - - name: Update CA trust in awx_task container - command: docker exec awx_task '/usr/bin/update-ca-trust' - when: awx_compose_config.changed or awx_compose_start.changed - - - name: Wait for launch script to create user - wait_for: - timeout: 10 - delegate_to: localhost - - - name: Create Preload data - command: docker exec awx_task bash -c "/usr/bin/awx-manage create_preload_data" - when: create_preload_data|bool - register: cdo - changed_when: "'added' in cdo.stdout" - when: compose_start_containers|bool diff --git a/installer/roles/local_docker/tasks/main.yml b/installer/roles/local_docker/tasks/main.yml deleted file mode 100644 index e2b793e50eb4..000000000000 --- a/installer/roles/local_docker/tasks/main.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -- name: Generate broadcast websocket secret - set_fact: - broadcast_websocket_secret: "{{ lookup('password', '/dev/null length=128') }}" - run_once: true - no_log: true - when: broadcast_websocket_secret is not defined - -- import_tasks: upgrade_postgres.yml - when: - - postgres_data_dir is defined - - pg_hostname is not defined - -- import_tasks: set_image.yml -- import_tasks: compose.yml diff --git a/installer/roles/local_docker/tasks/set_image.yml b/installer/roles/local_docker/tasks/set_image.yml deleted file mode 100644 index 995214b651ae..000000000000 --- a/installer/roles/local_docker/tasks/set_image.yml +++ /dev/null @@ -1,46 +0,0 @@ ---- -- name: Manage AWX Container Images - block: - - name: Export Docker awx image if it isnt local and there isnt a registry defined - docker_image: - name: "{{ awx_image }}" - tag: "{{ awx_version }}" - archive_path: "{{ awx_local_base_config_path|default('/tmp') }}/{{ awx_image }}_{{ awx_version }}.tar" - when: inventory_hostname != "localhost" and docker_registry is not defined - delegate_to: localhost - - - name: Set docker base path - set_fact: - docker_deploy_base_path: "{{ awx_base_path|default('/tmp') }}/docker_deploy" - when: ansible_connection != "local" and docker_registry is not defined - - - name: Ensure directory exists - file: - path: "{{ docker_deploy_base_path }}" - state: directory - when: ansible_connection != "local" and docker_registry is not defined - - - name: Copy awx image to docker execution - copy: - src: "{{ awx_local_base_config_path|default('/tmp') }}/{{ awx_image }}_{{ awx_version }}.tar" - dest: "{{ docker_deploy_base_path }}/{{ awx_image }}_{{ awx_version }}.tar" - when: ansible_connection != "local" and docker_registry is not defined - - - name: Load awx image - docker_image: - name: "{{ awx_image }}" - tag: "{{ awx_version }}" - load_path: "{{ docker_deploy_base_path }}/{{ awx_image }}_{{ awx_version }}.tar" - timeout: 300 - when: ansible_connection != "local" and docker_registry is not defined - - - name: Set full image path for local install - set_fact: - awx_docker_actual_image: "{{ awx_image }}:{{ awx_version }}" - when: docker_registry is not defined - when: dockerhub_base is not defined - -- name: Set DockerHub Image Paths - set_fact: - awx_docker_actual_image: "{{ dockerhub_base }}/awx:{{ dockerhub_version }}" - when: dockerhub_base is defined diff --git a/installer/roles/local_docker/tasks/upgrade_postgres.yml b/installer/roles/local_docker/tasks/upgrade_postgres.yml deleted file mode 100644 index 7887960a578d..000000000000 --- a/installer/roles/local_docker/tasks/upgrade_postgres.yml +++ /dev/null @@ -1,64 +0,0 @@ ---- - -- name: Create {{ postgres_data_dir }} directory - file: - path: "{{ postgres_data_dir }}" - state: directory - -- name: Get full path of postgres data dir - shell: "echo {{ postgres_data_dir }}" - register: fq_postgres_data_dir - -- name: Register temporary docker container - set_fact: - container_command: "docker run --rm -v '{{ fq_postgres_data_dir.stdout }}:/var/lib/postgresql' centos:8 bash -c " - -- name: Check for existing Postgres data (run from inside the container for access to file) - shell: - cmd: | - {{ container_command }} "[[ -f /var/lib/postgresql/10/data/PG_VERSION ]] && echo 'exists'" - register: pg_version_file - ignore_errors: true - -- name: Record Postgres version - shell: | - {{ container_command }} "cat /var/lib/postgresql/10/data/PG_VERSION" - register: old_pg_version - when: pg_version_file is defined and pg_version_file.stdout == 'exists' - -- name: Determine whether to upgrade postgres - set_fact: - upgrade_postgres: "{{ old_pg_version.stdout == '10' }}" - when: old_pg_version.changed - -- name: Set up new postgres paths pre-upgrade - shell: | - {{ container_command }} "mkdir -p /var/lib/postgresql/12/data/" - when: upgrade_postgres | bool - -- name: Stop AWX before upgrading postgres - docker_compose: - project_src: "{{ docker_compose_dir }}" - stopped: true - when: upgrade_postgres | bool - -- name: Upgrade Postgres - shell: | - docker run --rm \ - -v {{ postgres_data_dir }}/10/data:/var/lib/postgresql/10/data \ - -v {{ postgres_data_dir }}/12/data:/var/lib/postgresql/12/data \ - -e PGUSER={{ pg_username }} -e POSTGRES_INITDB_ARGS="-U {{ pg_username }}" \ - tianon/postgres-upgrade:10-to-12 --username={{ pg_username }} - when: upgrade_postgres | bool - -- name: Copy old pg_hba.conf - shell: | - {{ container_command }} "cp /var/lib/postgresql/10/data/pg_hba.conf /var/lib/postgresql/12/data/pg_hba.conf" - when: upgrade_postgres | bool - -- name: Remove old data directory - shell: | - {{ container_command }} "rm -rf /var/lib/postgresql/10/data" - when: - - upgrade_postgres | bool - - compose_start_containers|bool diff --git a/installer/roles/local_docker/templates/docker-compose.yml.j2 b/installer/roles/local_docker/templates/docker-compose.yml.j2 deleted file mode 100644 index b18aa83408a9..000000000000 --- a/installer/roles/local_docker/templates/docker-compose.yml.j2 +++ /dev/null @@ -1,208 +0,0 @@ -#jinja2: lstrip_blocks: True -version: '2' -services: - - web: - image: {{ awx_docker_actual_image }} - container_name: awx_web - depends_on: - - redis - {% if pg_hostname is not defined %} - - postgres - {% endif %} - {% if (host_port is defined) or (host_port_ssl is defined) %} - ports: - {% if (host_port_ssl is defined) and (ssl_certificate is defined) %} - - "{{ host_port_ssl }}:8053" - {% endif %} - {% if host_port is defined %} - - "{{ host_port }}:8052" - {% endif %} - {% endif %} - hostname: {{ awx_web_hostname }} - user: root - restart: unless-stopped - {% if (awx_web_container_labels is defined) and (',' in awx_web_container_labels) %} - {% set awx_web_container_labels_list = awx_web_container_labels.split(',') %} - labels: - {% for awx_web_container_label in awx_web_container_labels_list %} - - {{ awx_web_container_label }} - {% endfor %} - {% elif awx_web_container_labels is defined %} - labels: - - {{ awx_web_container_labels }} - {% endif %} - volumes: - - supervisor-socket:/var/run/supervisor - - rsyslog-socket:/var/run/awx-rsyslog/ - - rsyslog-config:/var/lib/awx/rsyslog/ - - "{{ docker_compose_dir }}/SECRET_KEY:/etc/tower/SECRET_KEY" - - "{{ docker_compose_dir }}/environment.sh:/etc/tower/conf.d/environment.sh" - - "{{ docker_compose_dir }}/credentials.py:/etc/tower/conf.d/credentials.py" - - "{{ docker_compose_dir }}/nginx.conf:/etc/nginx/nginx.conf:ro" - - "{{ docker_compose_dir }}/redis_socket:/var/run/redis/:rw" - {% if project_data_dir is defined %} - - "{{ project_data_dir +':/var/lib/awx/projects:rw' }}" - {% endif %} - {% if custom_venv_dir is defined %} - - "{{ custom_venv_dir +':'+ custom_venv_dir +':rw' }}" - {% endif %} - {% if ca_trust_dir is defined %} - - "{{ ca_trust_dir +':/etc/pki/ca-trust/source/anchors:ro' }}" - {% endif %} - {% if (ssl_certificate is defined) and (ssl_certificate_key is defined) %} - - "{{ ssl_certificate +':/etc/nginx/awxweb.pem:ro' }}" - - "{{ ssl_certificate_key +':/etc/nginx/awxweb_key.pem:ro' }}" - {% elif (ssl_certificate is defined) and (ssl_certificate_key is not defined) %} - - "{{ ssl_certificate +':/etc/nginx/awxweb.pem:ro' }}" - {% endif %} - {% if (awx_container_search_domains is defined) and (',' in awx_container_search_domains) %} - {% set awx_container_search_domains_list = awx_container_search_domains.split(',') %} - dns_search: - {% for awx_container_search_domain in awx_container_search_domains_list %} - - {{ awx_container_search_domain }} - {% endfor %} - {% elif awx_container_search_domains is defined %} - dns_search: "{{ awx_container_search_domains }}" - {% endif %} - {% if (awx_alternate_dns_servers is defined) and (',' in awx_alternate_dns_servers) %} - {% set awx_alternate_dns_servers_list = awx_alternate_dns_servers.split(',') %} - dns: - {% for awx_alternate_dns_server in awx_alternate_dns_servers_list %} - - {{ awx_alternate_dns_server }} - {% endfor %} - {% elif awx_alternate_dns_servers is defined %} - dns: "{{ awx_alternate_dns_servers }}" - {% endif %} - {% if (docker_compose_extra_hosts is defined) and (':' in docker_compose_extra_hosts) %} - {% set docker_compose_extra_hosts_list = docker_compose_extra_hosts.split(',') %} - extra_hosts: - {% for docker_compose_extra_host in docker_compose_extra_hosts_list %} - - "{{ docker_compose_extra_host }}" - {% endfor %} - {% endif %} - environment: - http_proxy: {{ http_proxy | default('') }} - https_proxy: {{ https_proxy | default('') }} - no_proxy: {{ no_proxy | default('') }} - {% if docker_logger is defined %} - logging: - driver: {{ docker_logger }} - {% endif %} - - task: - image: {{ awx_docker_actual_image }} - container_name: awx_task - depends_on: - - redis - - web - {% if pg_hostname is not defined %} - - postgres - {% endif %} - command: /usr/bin/launch_awx_task.sh - hostname: {{ awx_task_hostname }} - user: root - restart: unless-stopped - volumes: - - supervisor-socket:/var/run/supervisor - - rsyslog-socket:/var/run/awx-rsyslog/ - - rsyslog-config:/var/lib/awx/rsyslog/ - - "{{ docker_compose_dir }}/SECRET_KEY:/etc/tower/SECRET_KEY" - - "{{ docker_compose_dir }}/environment.sh:/etc/tower/conf.d/environment.sh" - - "{{ docker_compose_dir }}/credentials.py:/etc/tower/conf.d/credentials.py" - - "{{ docker_compose_dir }}/redis_socket:/var/run/redis/:rw" - {% if project_data_dir is defined %} - - "{{ project_data_dir +':/var/lib/awx/projects:rw' }}" - {% endif %} - {% if custom_venv_dir is defined %} - - "{{ custom_venv_dir +':'+ custom_venv_dir +':rw' }}" - {% endif %} - {% if ca_trust_dir is defined %} - - "{{ ca_trust_dir +':/etc/pki/ca-trust/source/anchors:ro' }}" - {% endif %} - {% if ssl_certificate is defined %} - - "{{ ssl_certificate +':/etc/nginx/awxweb.pem:ro' }}" - {% endif %} - {% if (awx_container_search_domains is defined) and (',' in awx_container_search_domains) %} - {% set awx_container_search_domains_list = awx_container_search_domains.split(',') %} - dns_search: - {% for awx_container_search_domain in awx_container_search_domains_list %} - - {{ awx_container_search_domain }} - {% endfor %} - {% elif awx_container_search_domains is defined %} - dns_search: "{{ awx_container_search_domains }}" - {% endif %} - {% if (awx_alternate_dns_servers is defined) and (',' in awx_alternate_dns_servers) %} - {% set awx_alternate_dns_servers_list = awx_alternate_dns_servers.split(',') %} - dns: - {% for awx_alternate_dns_server in awx_alternate_dns_servers_list %} - - {{ awx_alternate_dns_server }} - {% endfor %} - {% elif awx_alternate_dns_servers is defined %} - dns: "{{ awx_alternate_dns_servers }}" - {% endif %} - {% if (docker_compose_extra_hosts is defined) and (':' in docker_compose_extra_hosts) %} - {% set docker_compose_extra_hosts_list = docker_compose_extra_hosts.split(',') %} - extra_hosts: - {% for docker_compose_extra_host in docker_compose_extra_hosts_list %} - - "{{ docker_compose_extra_host }}" - {% endfor %} - {% endif %} - environment: - AWX_SKIP_MIGRATIONS: "1" - http_proxy: {{ http_proxy | default('') }} - https_proxy: {{ https_proxy | default('') }} - no_proxy: {{ no_proxy | default('') }} - SUPERVISOR_WEB_CONFIG_PATH: '/etc/supervisord.conf' - - redis: - image: {{ redis_image }} - container_name: awx_redis - restart: unless-stopped - environment: - http_proxy: {{ http_proxy | default('') }} - https_proxy: {{ https_proxy | default('') }} - no_proxy: {{ no_proxy | default('') }} - command: ["/usr/local/etc/redis/redis.conf"] - volumes: - - "{{ docker_compose_dir }}/redis.conf:/usr/local/etc/redis/redis.conf:ro" - - "{{ docker_compose_dir }}/redis_socket:/var/run/redis/:rw" - {% if docker_logger is defined %} - logging: - driver: {{ docker_logger }} - {% endif %} - - {% if pg_hostname is not defined %} - postgres: - image: {{ postgresql_image }} - container_name: awx_postgres - restart: unless-stopped - volumes: - - "{{ postgres_data_dir }}/12/data/:/var/lib/postgresql/data:Z" - environment: - POSTGRES_USER: {{ pg_username }} - POSTGRES_PASSWORD: {{ pg_password }} - POSTGRES_DB: {{ pg_database }} - http_proxy: {{ http_proxy | default('') }} - https_proxy: {{ https_proxy | default('') }} - no_proxy: {{ no_proxy | default('') }} - {% if docker_logger is defined %} - logging: - driver: {{ docker_logger }} - {% endif %} - {% endif %} - -{% if docker_compose_subnet is defined %} -networks: - default: - driver: bridge - ipam: - driver: default - config: - - subnet: {{ docker_compose_subnet }} -{% endif %} - -volumes: - supervisor-socket: - rsyslog-socket: - rsyslog-config: diff --git a/installer/roles/local_docker/templates/redis.conf.j2 b/installer/roles/local_docker/templates/redis.conf.j2 deleted file mode 100644 index 017bb067d5c0..000000000000 --- a/installer/roles/local_docker/templates/redis.conf.j2 +++ /dev/null @@ -1,4 +0,0 @@ -unixsocket /var/run/redis/redis.sock -unixsocketperm 660 -port 0 -bind 127.0.0.1 diff --git a/setup.cfg b/setup.cfg index beb4521b994d..c48afe266ea4 100755 --- a/setup.cfg +++ b/setup.cfg @@ -13,9 +13,9 @@ # W391 - Blank line at end of file # W293 - Blank line contains whitespace ignore=E201,E203,E221,E225,E231,E241,E251,E261,E265,E303,W291,W391,W293 -exclude=.tox,venv,awx/lib/site-packages,awx/plugins/inventory/ec2.py,awx/plugins/inventory/gce.py,awx/plugins/inventory/vmware.py,awx/plugins/inventory/openstack.py,awx/ui,awx/api/urls.py,awx/main/migrations,awx/main/tests/data,installer/openshift/settings.py +exclude=.tox,venv,awx/lib/site-packages,awx/plugins/inventory/ec2.py,awx/plugins/inventory/gce.py,awx/plugins/inventory/vmware.py,awx/plugins/inventory/openstack.py,awx/ui,awx/api/urls.py,awx/main/migrations,awx/main/tests/data,installer/openshift/settings.py,tools/docker-compose/ansible/roles/sources/files/local_settings.py,tools/ansible/roles/dockerfile/files/settings.py [flake8] max-line-length=160 ignore=E201,E203,E221,E225,E231,E241,E251,E261,E265,E303,W291,W391,W293,E731,W504 -exclude=.tox,venv,awx/lib/site-packages,awx/plugins/inventory,awx/ui,awx/api/urls.py,awx/main/migrations,awx/main/tests/data,node_modules/,awx/projects/,tools/docker,awx/settings/local_*.py,installer/openshift/settings.py,build/,installer/,awxkit/test,awx_collection/ +exclude=.tox,venv,awx/lib/site-packages,awx/plugins/inventory,awx/ui,awx/api/urls.py,awx/main/migrations,awx/main/tests/data,node_modules/,awx/projects/,tools/docker,awx/settings/local_*.py,installer/openshift/settings.py,build/,installer/,awxkit/test,awx_collection/,tools/docker-compose/ansible/roles/sources/files/local_settings.py,tools/ansible/roles/dockerfile/files/settings.py diff --git a/installer/build.yml b/tools/ansible/build.yml similarity index 100% rename from installer/build.yml rename to tools/ansible/build.yml diff --git a/tools/ansible/dockerfile.yml b/tools/ansible/dockerfile.yml new file mode 100644 index 000000000000..9b6bfdf97416 --- /dev/null +++ b/tools/ansible/dockerfile.yml @@ -0,0 +1,6 @@ +--- +- name: Render AWX Dockerfile and sources + hosts: localhost + gather_facts: true + roles: + - {role: dockerfile} diff --git a/installer/roles/dockerfile/defaults/main.yml b/tools/ansible/roles/dockerfile/defaults/main.yml similarity index 78% rename from installer/roles/dockerfile/defaults/main.yml rename to tools/ansible/roles/dockerfile/defaults/main.yml index fd1d93961abd..3d02bb35dab4 100644 --- a/installer/roles/dockerfile/defaults/main.yml +++ b/tools/ansible/roles/dockerfile/defaults/main.yml @@ -1,6 +1,6 @@ --- build_dev: false kube_dev: false -dockerfile_dest: '..' +dockerfile_dest: '../..' dockerfile_name: 'Dockerfile' template_dest: '_build' diff --git a/installer/roles/dockerfile/files/RPM-GPG-KEY-ansible-release b/tools/ansible/roles/dockerfile/files/RPM-GPG-KEY-ansible-release similarity index 100% rename from installer/roles/dockerfile/files/RPM-GPG-KEY-ansible-release rename to tools/ansible/roles/dockerfile/files/RPM-GPG-KEY-ansible-release diff --git a/installer/roles/dockerfile/files/launch_awx.sh b/tools/ansible/roles/dockerfile/files/launch_awx.sh old mode 100755 new mode 100644 similarity index 100% rename from installer/roles/dockerfile/files/launch_awx.sh rename to tools/ansible/roles/dockerfile/files/launch_awx.sh diff --git a/installer/roles/dockerfile/files/launch_awx_task.sh b/tools/ansible/roles/dockerfile/files/launch_awx_task.sh old mode 100755 new mode 100644 similarity index 97% rename from installer/roles/dockerfile/files/launch_awx_task.sh rename to tools/ansible/roles/dockerfile/files/launch_awx_task.sh index 3fe663096394..120cc9e3f838 --- a/installer/roles/dockerfile/files/launch_awx_task.sh +++ b/tools/ansible/roles/dockerfile/files/launch_awx_task.sh @@ -19,6 +19,7 @@ ANSIBLE_REMOTE_TEMP=/tmp ANSIBLE_LOCAL_TEMP=/tmp ansible -i "127.0.0.1," -c loca ANSIBLE_REMOTE_TEMP=/tmp ANSIBLE_LOCAL_TEMP=/tmp ansible -i "127.0.0.1," -c local -v -m postgresql_db --become-user $DATABASE_USER -a "name=$DATABASE_NAME owner=$DATABASE_USER login_user=$DATABASE_USER login_host=$DATABASE_HOST login_password=$DATABASE_PASSWORD port=$DATABASE_PORT" all if [ -z "$AWX_SKIP_MIGRATIONS" ]; then + echo "Running migrations..." awx-manage migrate --noinput fi diff --git a/installer/roles/dockerfile/files/rsyslog.conf b/tools/ansible/roles/dockerfile/files/rsyslog.conf similarity index 100% rename from installer/roles/dockerfile/files/rsyslog.conf rename to tools/ansible/roles/dockerfile/files/rsyslog.conf diff --git a/installer/roles/dockerfile/files/settings.py b/tools/ansible/roles/dockerfile/files/settings.py similarity index 100% rename from installer/roles/dockerfile/files/settings.py rename to tools/ansible/roles/dockerfile/files/settings.py diff --git a/installer/roles/dockerfile/tasks/main.yml b/tools/ansible/roles/dockerfile/tasks/main.yml similarity index 92% rename from installer/roles/dockerfile/tasks/main.yml rename to tools/ansible/roles/dockerfile/tasks/main.yml index 7d315e1856c1..c224fc5dd111 100644 --- a/installer/roles/dockerfile/tasks/main.yml +++ b/tools/ansible/roles/dockerfile/tasks/main.yml @@ -1,6 +1,6 @@ --- -- name: Create .build directory +- name: Create _build directory file: path: "{{ dockerfile_dest }}/{{ template_dest }}" state: directory diff --git a/installer/roles/dockerfile/templates/Dockerfile.j2 b/tools/ansible/roles/dockerfile/templates/Dockerfile.j2 similarity index 94% rename from installer/roles/dockerfile/templates/Dockerfile.j2 rename to tools/ansible/roles/dockerfile/templates/Dockerfile.j2 index edcfbefede62..7acf365c25dc 100644 --- a/installer/roles/dockerfile/templates/Dockerfile.j2 +++ b/tools/ansible/roles/dockerfile/templates/Dockerfile.j2 @@ -1,5 +1,5 @@ ### This file is generated from -### installer/roles/dockerfile/templates/Dockerfile.j2 +### tools/ansible/roles/dockerfile/templates/Dockerfile.j2 ### ### DO NOT EDIT ### @@ -182,7 +182,7 @@ RUN openssl req -nodes -newkey rsa:2048 -keyout /etc/nginx/nginx.key -out /etc/n {% endif %} # Create default awx rsyslog config -ADD installer/roles/dockerfile/files/rsyslog.conf /var/lib/awx/rsyslog/rsyslog.conf +ADD tools/ansible/roles/dockerfile/files/rsyslog.conf /var/lib/awx/rsyslog/rsyslog.conf ## File mappings {% if build_dev|bool %} @@ -193,9 +193,9 @@ ADD tools/docker-compose/start_tests.sh /start_tests.sh ADD tools/docker-compose/bootstrap_development.sh /usr/bin/bootstrap_development.sh ADD tools/docker-compose/entrypoint.sh /entrypoint.sh {% else %} -ADD installer/roles/dockerfile/files/launch_awx.sh /usr/bin/launch_awx.sh -ADD installer/roles/dockerfile/files/launch_awx_task.sh /usr/bin/launch_awx_task.sh -ADD installer/roles/dockerfile/files/settings.py /etc/tower/settings.py +ADD tools/ansible/roles/dockerfile/files/launch_awx.sh /usr/bin/launch_awx.sh +ADD tools/ansible/roles/dockerfile/files/launch_awx_task.sh /usr/bin/launch_awx_task.sh +ADD tools/ansible/roles/dockerfile/files/settings.py /etc/tower/settings.py ADD {{ template_dest }}/supervisor.conf /etc/supervisord.conf ADD {{ template_dest }}/supervisor_task.conf /etc/supervisord_task.conf ADD tools/scripts/config-watcher /usr/bin/config-watcher @@ -219,7 +219,8 @@ RUN for dir in \ /var/lib/nginx ; \ do mkdir -m 0775 -p $dir ; chmod g+rw $dir ; chgrp root $dir ; done && \ for file in \ - /etc/passwd ; \ + /etc/passwd \ + /var/lib/awx/rsyslog/rsyslog.conf ; \ do touch $file ; chmod g+rw $file ; chgrp root $file ; done # Adjust any remaining permissions diff --git a/installer/roles/dockerfile/templates/supervisor.conf.j2 b/tools/ansible/roles/dockerfile/templates/supervisor.conf.j2 similarity index 100% rename from installer/roles/dockerfile/templates/supervisor.conf.j2 rename to tools/ansible/roles/dockerfile/templates/supervisor.conf.j2 diff --git a/installer/roles/dockerfile/templates/supervisor_task.conf.j2 b/tools/ansible/roles/dockerfile/templates/supervisor_task.conf.j2 similarity index 100% rename from installer/roles/dockerfile/templates/supervisor_task.conf.j2 rename to tools/ansible/roles/dockerfile/templates/supervisor_task.conf.j2 diff --git a/installer/roles/image_build/defaults/main.yml b/tools/ansible/roles/image_build/defaults/main.yml similarity index 100% rename from installer/roles/image_build/defaults/main.yml rename to tools/ansible/roles/image_build/defaults/main.yml diff --git a/installer/roles/image_build/tasks/main.yml b/tools/ansible/roles/image_build/tasks/main.yml similarity index 83% rename from installer/roles/image_build/tasks/main.yml rename to tools/ansible/roles/image_build/tasks/main.yml index 065710974cce..ae6e30b7a39a 100644 --- a/installer/roles/image_build/tasks/main.yml +++ b/tools/ansible/roles/image_build/tasks/main.yml @@ -1,7 +1,7 @@ --- - name: Set global version if not provided set_fact: - awx_version: "{{ lookup('file', playbook_dir + '/../VERSION') }}" + awx_version: "{{ lookup('file', playbook_dir + '/../../VERSION') }}" when: awx_version is not defined - name: Verify awx-logos directory exists for official install @@ -23,7 +23,7 @@ # Calling Docker directly because docker-py doesnt support BuildKit - name: Build AWX image - command: docker build -t {{ awx_image }}:{{ awx_version }} .. + command: docker build -t {{ awx_image }}:{{ awx_version }} -f ../../{{ dockerfile_name }} ../.. - name: Tag awx images as latest command: "docker tag {{ item }}:{{ awx_version }} {{ item }}:latest" diff --git a/installer/roles/image_push/tasks/main.yml b/tools/ansible/roles/image_push/tasks/main.yml similarity index 100% rename from installer/roles/image_push/tasks/main.yml rename to tools/ansible/roles/image_push/tasks/main.yml diff --git a/tools/docker-compose/README.md b/tools/docker-compose/README.md index b3fdc1db4f5d..7e564c067561 100644 --- a/tools/docker-compose/README.md +++ b/tools/docker-compose/README.md @@ -1,86 +1,171 @@ -# Docker Compose for Dev container +# Docker Compose for Development -## How to start the Dev container +## Overview -In the root directory of your awx clone, run the following to build your docker image. -This step takes a while, but once your image is built, you don't need to do this again -unless you make changes to the Dockerfile or any of the files used by the Dockerfile. +Here are the main make targets: + +* `docker-compose-build` - used for building the development image, which is used by the `docker-compose` target +* `docker-compose` - make target for development, passes awx_devel image and tag + +Notable files: +* `tools/docker-compose/inventory` file - used to configure the local AWX development deploymen +* `migrate.yml` - playbook for migrating data from Local Docker to the Development Environment + +### Prerequisites + +- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the ``docker`` group, refer to the documentation for details) +- [docker-compose](https://pypi.org/project/docker-compose/) Python module. + + This also installs the `docker` Python module, which is incompatible with [`docker-py`](https://pypi.org/project/docker-py/). If you have previously installed `docker-py`, please uninstall it. +- [Docker Compose](https://docs.docker.com/compose/install/). + +## Configuration + +In the `inventory` file, set your `pg_password`, `broadcast_websocket_secret`, `secret_key`, and any other settings you need for your deployment. + +AWX requires access to a PostgreSQL database, and by default, one will be created and deployed in a container, and data will be persisted to a docker volume. When the container is stopped, the database files will still exist in the docker volume. An external database can be used by setting the `pg_host`, `pg_hostname`, and `pg_username`. + +> If you are coming from a Local Docker installation of AWX, consider migrating your data first, see the [data migration section](#migrating-data-from-local-docker) below. -``` -make docker-compose-build -``` -> Note: By default, this image will be tagged with your branch name. +## Starting the Development Environment -Copy over your local settings +### Build the image +The AWX base container image (defined in the Dockerfile templated from [Dockerfile.j2](./../ansible/roles/dockerfile/templates/Dockerfile.j2)) contains basic OS dependencies and symbolic links into the development environment that make running the services easy. + +Run the following to build the image: + +```bash +$ make docker-compose-build ``` -cp awx/settings/local_settings.py.docker_compose awx/settings/local_settings.py + +> The image will need to be rebuilt if there are any changes to Dockerfile.j2 or any of the files used by the templated Dockerfile. + +Once the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows: + +```bash +(host)$ docker images + +REPOSITORY TAG IMAGE ID CREATED SIZE +ansible/awx_devel latest ba9ec3e8df74 26 minutes ago 1.42GB ``` -Build the UI +> By default, this image will be tagged with your branch name. You can specify a custom tag by setting an environment variable, for example: `DEVEL_IMAGE_NAME=quay.io/your_user/awx_devel:17.0.1` + +### Run AWX + +##### Start the containers +Run the awx, postgres and redis containers. This utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the awx container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process. +```bash +$ make docker-compose ``` -make ui-devel + +> The make target assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG= `in your environment. For example, you might be working in a feature branch, but you want to run the containers using the devel image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose` + +##### Wait for migrations to complete + +The first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following: + +```bash +awx_1 | Operations to perform: +awx_1 | Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic +awx_1 | Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main +awx_1 | Synchronizing apps without migrations: +awx_1 | Creating tables... +awx_1 | Running deferred SQL... +awx_1 | Installing custom SQL... +awx_1 | Running migrations: +awx_1 | Rendering model states... DONE +awx_1 | Applying contenttypes.0001_initial... OK +awx_1 | Applying contenttypes.0002_remove_content_type_name... OK +awx_1 | Applying auth.0001_initial... OK +... ``` -Run the container +##### Clean and Build UI +```bash +$ docker exec tools_awx_1 make clean-ui ui-devel +``` + +See [the ui development documentation](../../awx/ui_next/README.md) for more information on using the frontend development, build, and test tooling. + +Once migrations are completed and the UI is built, you can begin using AWX. The UI can be reached in your browser at `https://localhost:8043/#/home`, and the API can be found at `https://localhost:8043/api/v2`. +##### Create an admin user + +Before you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command: +```bash +$ docker exec -ti tools_awx_1 awx-manage createsuperuser ``` -make docker-compose + +> Remember the username and password, as you will use them to log into the web interface for the first time. + +##### Load Demo Data +Optionally, you may also want to load some demo data. This will create a demo project, inventory, and job template. + +```bash +$ docker exec tools_awx_1 awx-manage create_preload_data ``` -> Note: You may need to add `COMPOSE_TAG=` at the end of this to use the correct image for your branch. +> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate +this container, and thus the database, if the database schema changes in an upstream commit. -The app should now be accessible in your browser at `https://localhost:8043/#/home` +## Migrating Data from Local Docker -## How to use the logstash container +If you are migrating data from a Local Docker installation (17.0.1 and prior), you can +migrate your data to the development environment via the migrate.yml playbook using the steps described [here](./docs/data_migration.md). -#### Modify the docker-compose.yml +## Upgrading the Development Environment -Uncomment the following lines in the `docker-compose.yml` +Upgrading AWX involves checking out the new source code and re-running the make target. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables. -``` -#- logstash -... +After updating the inventory file with any custom values, run the make target from the root of your AWX clone. -#logstash: -# build: -# context: ./docker-compose -# dockerfile: Dockerfile-logstash +```bash +$ make docker-compose ``` -POST the following content to `/api/v2/settings/logging/` (this uses -authentication set up inside of the logstash configuration file). +## Extras +* [Start a shell](#start-a-shell) +* [Start AWX from the container shell](#start-awx-from-the-container-shell) +* [Using Logstash](./docs/logstash.md) +### Start a Shell + +To run `awx-manage` commands and modify things inside the container, you will want to start a shell session on the *awx* container. In a new terminal session, use the `docker exec` command to start the shell session: + +```bash +(host)$ docker exec -it tools_awx_1 bash ``` -{ - "LOG_AGGREGATOR_HOST": "http://logstash", - "LOG_AGGREGATOR_PORT": 8085, - "LOG_AGGREGATOR_TYPE": "logstash", - "LOG_AGGREGATOR_USERNAME": "awx_logger", - "LOG_AGGREGATOR_PASSWORD": "workflows", - "LOG_AGGREGATOR_LOGGERS": [ - "awx", - "activity_stream", - "job_events", - "system_tracking" - ], - "LOG_AGGREGATOR_INDIVIDUAL_FACTS": false, - "LOG_AGGREGATOR_TOWER_UUID": "991ac7e9-6d68-48c8-bbde-7ca1096653c6", - "LOG_AGGREGATOR_ENABLED": true -} -``` -> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. +This creates a session in the *awx* containers, just as if you were using `ssh`, and allows you execute commands within the running container. + +### Start AWX from the container shell -An example of how to view the most recent logs from the container: +Often times you'll want to start the development environment without immediately starting all of the services in the *awx* container, and instead be taken directly to a shell. You can do this with the following: +```bash +(host)$ make docker-compose-test ``` -docker exec -i -t $(docker ps -aqf "name=tools_logstash_1") tail -n 50 /logstash.log + +Using `docker exec`, this will create a session in the running *awx* container, and place you at a command prompt, where you can run shell commands inside the container. + +If you want to start and use the development environment, you'll first need to bootstrap it by running the following command: + +```bash +(container)# /usr/bin/bootstrap_development.sh ``` -#### How to add logstash plugins +The above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it +will drop you back to the shell. + +In order to launch all developer services: + +```bash +(container)# /usr/bin/launch_awx.sh +``` -Add any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container. +`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't +need to call `bootstrap_development.sh` first. diff --git a/tools/docker-compose/ansible/migrate.yml b/tools/docker-compose/ansible/migrate.yml new file mode 100644 index 000000000000..a7320edbde4b --- /dev/null +++ b/tools/docker-compose/ansible/migrate.yml @@ -0,0 +1,7 @@ +--- +- name: Migrate data for upgrades and from Local Docker installs + hosts: localhost + gather_facts: true + roles: + - {role: sources} + - {role: migrate} diff --git a/tools/docker-compose/ansible/roles/migrate/defaults/main.yml b/tools/docker-compose/ansible/roles/migrate/defaults/main.yml new file mode 100644 index 000000000000..3d62cd0f884a --- /dev/null +++ b/tools/docker-compose/ansible/roles/migrate/defaults/main.yml @@ -0,0 +1,5 @@ +--- +postgres_data_dir: "~/.awx/pgdocker" +migrate_local_docker: false +old_docker_compose_dir: "~/.awx/awxcompose" +pg_volume_name: "postgres_data" diff --git a/tools/docker-compose/ansible/roles/migrate/tasks/main.yml b/tools/docker-compose/ansible/roles/migrate/tasks/main.yml new file mode 100644 index 000000000000..29b38e4ee66c --- /dev/null +++ b/tools/docker-compose/ansible/roles/migrate/tasks/main.yml @@ -0,0 +1,5 @@ +--- + +# Migrate data from a bind-mount to a volume +- import_tasks: migrate-from-local-docker.yml + when: migrate_local_docker diff --git a/tools/docker-compose/ansible/roles/migrate/tasks/migrate-from-local-docker.yml b/tools/docker-compose/ansible/roles/migrate/tasks/migrate-from-local-docker.yml new file mode 100644 index 000000000000..e40dac15090e --- /dev/null +++ b/tools/docker-compose/ansible/roles/migrate/tasks/migrate-from-local-docker.yml @@ -0,0 +1,53 @@ +--- + +# Migrate data from old Local Docker to a fresh development environment +- name: Remove awx_postgres to ensure consistent start state + shell: | + docker rm -f awx_postgres + ignore_errors: true + +- name: Start Local Docker database container + docker_compose: + project_src: "{{ old_docker_compose_dir }}" + services: + - postgres + state: present + recreate: always + +- name: Wait for postgres to initialize + wait_for: + timeout: 3 + +- name: Database dump to local filesystem + shell: | + docker-compose -f {{ old_docker_compose_dir }}/docker-compose.yml exec -T postgres pg_dumpall -U {{ pg_username }} > awx_dump.sql + +- name: Stop AWX containers so the old postgres container does not get used + docker_compose: + project_src: "{{ old_docker_compose_dir }}" + state: absent + ignore_errors: true + +- name: Start dev env database container + docker_compose: + project_src: "{{ playbook_dir }}/../_sources" + files: "docker-compose.yml" + services: + - postgres + state: present + recreate: always + environment: + COMPOSE_PROJECT_NAME: "tools" + +- name: Wait for postgres to initialize + wait_for: + timeout: 3 + +- name: Restore to new postgres container + shell: | + COMPOSE_PROJECT_NAME=tools docker-compose -f {{ playbook_dir }}/../_sources/docker-compose.yml exec -T postgres psql -U {{ pg_username }} -d {{ pg_database }} -p {{ pg_port }} < awx_dump.sql + +- name: Clean up temporary awx db dump + file: + path: awx_dump.sql + state: absent diff --git a/tools/docker-compose/ansible/roles/sources/defaults/main.yml b/tools/docker-compose/ansible/roles/sources/defaults/main.yml new file mode 100644 index 000000000000..f867d7ef2a67 --- /dev/null +++ b/tools/docker-compose/ansible/roles/sources/defaults/main.yml @@ -0,0 +1,7 @@ +--- +sources_dest: '../_sources' +compose_name: 'docker-compose.yml' +awx_image: 'quay.io/ansible/awx_devel' +pg_port: 5432 +pg_username: 'awx' +pg_database: 'awx' diff --git a/awx/settings/local_settings.py.docker_compose b/tools/docker-compose/ansible/roles/sources/files/local_settings.py similarity index 88% rename from awx/settings/local_settings.py.docker_compose rename to tools/docker-compose/ansible/roles/sources/files/local_settings.py index 88ef90fd641c..f480f7fdb838 100644 --- a/awx/settings/local_settings.py.docker_compose +++ b/tools/docker-compose/ansible/roles/sources/files/local_settings.py @@ -20,19 +20,6 @@ # INTERNAL_IPS = ('172.19.0.1', '172.18.0.1', '192.168.100.1') # ALLOWED_HOSTS = ['*'] -# Database settings to use PostgreSQL for development. -DATABASES = { - 'default': { - 'ENGINE': 'awx.main.db.profiled_pg', - 'NAME': 'awx-dev', - 'USER': 'awx-dev', - 'PASSWORD': 'AWXsome1', - 'ATOMIC_REQUESTS': True, - 'HOST': 'postgres', - 'PORT': '', - } -} - # Use SQLite for unit tests instead of PostgreSQL. If the lines below are # commented out, Django will create the test_awx-dev database in PostgreSQL to # run unit tests. @@ -76,7 +63,6 @@ #LOGGING['loggers']['django_auth_ldap']['handlers'] = ['console'] #LOGGING['loggers']['django_auth_ldap']['level'] = 'DEBUG' -BROADCAST_WEBSOCKET_SECRET = '🤖starscream🤖' BROADCAST_WEBSOCKET_PORT = 8013 BROADCAST_WEBSOCKET_VERIFY_CERT = False BROADCAST_WEBSOCKET_PROTOCOL = 'http' diff --git a/tools/docker-compose/ansible/roles/sources/tasks/main.yml b/tools/docker-compose/ansible/roles/sources/tasks/main.yml new file mode 100644 index 000000000000..db2edee006b5 --- /dev/null +++ b/tools/docker-compose/ansible/roles/sources/tasks/main.yml @@ -0,0 +1,76 @@ +--- + +- name: Create _sources directory + file: + path: "{{ sources_dest }}/secrets" + state: 'directory' + mode: '0700' + +- name: Detect secrets + stat: + path: "{{ sources_dest }}/secrets/{{ item }}.yml" + register: secrets + when: not lookup('vars', item, default='') + loop: + - pg_password + - secret_key + - broadcast_websocket_secret + +- name: Generate secrets if needed + template: + src: 'secrets.yml.j2' + dest: '{{ sources_dest }}/secrets/{{ item.item }}.yml' + mode: '0600' + when: not lookup('vars', item.item, default='') and not item.stat.exists + loop: "{{ secrets.results }}" + +- name: Include generated secrets unless they are explicitly passed in + include_vars: "{{ sources_dest }}/secrets/{{ item.item }}.yml" + no_log: true + when: not lookup('vars', item.item, default='') + loop: "{{ secrets.results }}" + +- name: Render configuration templates + template: + src: "{{ item }}.j2" + dest: "{{ sources_dest }}/{{ item }}" + mode: '0600' + with_items: + - "database.py" + - "websocket_secret.py" + +- name: Delete old local_settings.py + file: + path: "{{ playbook_dir }}/../../../awx/settings/local_settings.py" + state: absent + +- name: Copy local_settings.py + copy: + src: "local_settings.py" + dest: "{{ sources_dest }}/local_settings.py" + +- name: Get OS info for sdb + shell: | + docker info | grep 'Operating System' + register: os_info + changed_when: false + +- name: Get user UID + shell: id -u + register: current_user + changed_when: false + +- name: Set fact with user UID + set_fact: + user_id: "'{{ current_user.stdout }}'" + +- name: Set global version if not provided + set_fact: + awx_image_tag: "{{ lookup('file', playbook_dir + '/../../../VERSION') }}" + when: awx_image_tag is not defined + +- name: Render Docker-Compose + template: + src: docker-compose.yml.j2 + dest: "{{ sources_dest }}/{{ compose_name }}" + mode: '0600' diff --git a/installer/roles/local_docker/templates/credentials.py.j2 b/tools/docker-compose/ansible/roles/sources/templates/database.py.j2 similarity index 68% rename from installer/roles/local_docker/templates/credentials.py.j2 rename to tools/docker-compose/ansible/roles/sources/templates/database.py.j2 index 9ea7ac277590..c1cd6fd99279 100644 --- a/installer/roles/local_docker/templates/credentials.py.j2 +++ b/tools/docker-compose/ansible/roles/sources/templates/database.py.j2 @@ -1,7 +1,7 @@ DATABASES = { 'default': { 'ATOMIC_REQUESTS': True, - 'ENGINE': 'django.db.backends.postgresql', + 'ENGINE': 'awx.main.db.profiled_pg', 'NAME': "{{ pg_database }}", 'USER': "{{ pg_username }}", 'PASSWORD': "{{ pg_password }}", @@ -9,5 +9,3 @@ DATABASES = { 'PORT': "{{ pg_port }}", } } - -BROADCAST_WEBSOCKET_SECRET = "{{ broadcast_websocket_secret | b64encode }}" diff --git a/tools/docker-compose.yml b/tools/docker-compose/ansible/roles/sources/templates/docker-compose.yml.j2 similarity index 59% rename from tools/docker-compose.yml rename to tools/docker-compose/ansible/roles/sources/templates/docker-compose.yml.j2 index 27261cd1e674..b639dcced8d3 100644 --- a/tools/docker-compose.yml +++ b/tools/docker-compose/ansible/roles/sources/templates/docker-compose.yml.j2 @@ -3,14 +3,13 @@ version: '2' services: # Primary AWX Development Container awx: - user: ${CURRENT_UID} - image: ${DEV_DOCKER_TAG_BASE}/awx_devel:${TAG} + user: "{{ ansible_user_uid }}" + image: "{{ awx_image }}:{{ awx_image_tag }}" container_name: tools_awx_1 hostname: awx command: launch_awx.sh environment: - CURRENT_UID: - OS: + OS: "{{ os_info.stdout }}" SDB_HOST: 0.0.0.0 SDB_PORT: 7899 AWX_GROUP_QUEUES: tower @@ -24,15 +23,14 @@ services: links: - postgres - redis - # - sync - # volumes_from: - # - sync working_dir: "/awx_devel" volumes: - - "../:/awx_devel" - - "../awx/projects/:/var/lib/awx/projects/" - - "./redis/redis_socket_standalone:/var/run/redis/" - - "./docker-compose/supervisor.conf:/etc/supervisord.conf" + - "../../../:/awx_devel" + - "../../docker-compose/supervisor.conf:/etc/supervisord.conf" + - "../../docker-compose/_sources/database.py:/etc/tower/conf.d/database.py" + - "../../docker-compose/_sources/websocket_secret.py:/etc/tower/conf.d/websocket_secret.py" + - "../../docker-compose/_sources/local_settings.py:/etc/tower/conf.d/local_settings.py" + - "redis_socket:/var/run/redis/:rw" privileged: true tty: true # A useful container that simply passes through log messages to the console @@ -48,16 +46,20 @@ services: container_name: tools_postgres_1 environment: POSTGRES_HOST_AUTH_METHOD: trust + POSTGRES_USER: {{ pg_username }} + POSTGRES_DB: {{ pg_database }} + POSTGRES_PASSWORD: {{ pg_password }} volumes: - "awx_db:/var/lib/postgresql/data" redis: image: redis:latest container_name: tools_redis_1 - user: ${CURRENT_UID} volumes: - - "./redis/redis.conf:/usr/local/etc/redis/redis.conf" - - "./redis/redis_socket_standalone:/var/run/redis/" + - "../../redis/redis.conf:/usr/local/etc/redis/redis.conf" + - "redis_socket:/var/run/redis/:rw" + entrypoint: ["redis-server"] command: ["/usr/local/etc/redis/redis.conf"] volumes: awx_db: + redis_socket: diff --git a/installer/roles/local_docker/templates/environment.sh.j2 b/tools/docker-compose/ansible/roles/sources/templates/environment.sh.j2 similarity index 100% rename from installer/roles/local_docker/templates/environment.sh.j2 rename to tools/docker-compose/ansible/roles/sources/templates/environment.sh.j2 diff --git a/installer/roles/local_docker/templates/nginx.conf.j2 b/tools/docker-compose/ansible/roles/sources/templates/nginx.conf.j2 similarity index 100% rename from installer/roles/local_docker/templates/nginx.conf.j2 rename to tools/docker-compose/ansible/roles/sources/templates/nginx.conf.j2 diff --git a/tools/docker-compose/ansible/roles/sources/templates/secrets.yml.j2 b/tools/docker-compose/ansible/roles/sources/templates/secrets.yml.j2 new file mode 100644 index 000000000000..12c64dbfbe6a --- /dev/null +++ b/tools/docker-compose/ansible/roles/sources/templates/secrets.yml.j2 @@ -0,0 +1 @@ +{{ item.item }}: '{{ lookup('vars', item.item, default='') or lookup('password', '/dev/null chars=ascii_letters') }}' diff --git a/tools/docker-compose/ansible/roles/sources/templates/websocket_secret.py.j2 b/tools/docker-compose/ansible/roles/sources/templates/websocket_secret.py.j2 new file mode 100644 index 000000000000..2ba4171f9e1b --- /dev/null +++ b/tools/docker-compose/ansible/roles/sources/templates/websocket_secret.py.j2 @@ -0,0 +1 @@ +BROADCAST_WEBSOCKET_SECRET = "{{ broadcast_websocket_secret | b64encode }}" diff --git a/tools/docker-compose/ansible/sources.yml b/tools/docker-compose/ansible/sources.yml new file mode 100644 index 000000000000..bed7c17fd868 --- /dev/null +++ b/tools/docker-compose/ansible/sources.yml @@ -0,0 +1,6 @@ +--- +- name: Render AWX Dockerfile and sources + hosts: localhost + gather_facts: true + roles: + - {role: sources} diff --git a/tools/docker-compose/bootstrap_development.sh b/tools/docker-compose/bootstrap_development.sh index 68210a8713bc..1d3e399bf7a1 100755 --- a/tools/docker-compose/bootstrap_development.sh +++ b/tools/docker-compose/bootstrap_development.sh @@ -9,9 +9,6 @@ ansible -i "127.0.0.1," -c local -v -m wait_for -a "path=/var/run/redis/redis.so # TODO: FIX #/etc/init.d/ssh start -ansible -i "127.0.0.1," -c local -v -m postgresql_user --become-user postgres -a "name=awx-dev password=AWXsome1 encrypted=yes login_user=postgres login_password=postgrespass login_host=postgres" all -ansible -i "127.0.0.1," -c local -v -m postgresql_db --become-user postgres -a "name=awx-dev owner=awx-dev login_user=postgres login_password=postgrespass login_host=postgres" all - # Move to the source directory so we can bootstrap if [ -f "/awx_devel/manage.py" ]; then cd /awx_devel diff --git a/tools/docker-compose/docs/data_migration.md b/tools/docker-compose/docs/data_migration.md new file mode 100644 index 000000000000..afd107be89a6 --- /dev/null +++ b/tools/docker-compose/docs/data_migration.md @@ -0,0 +1,60 @@ +# Migrating Data from Local Docker + +If you are migrating data from a Local Docker installation (17.0.1 and prior), you can +migrate your data to the development environment via the migrate.yml playbook, or by using the manual steps described below. + +> Note: This will also convert your postgresql bind-mount into a docker volume. + +### Migrate data with migrate.yml + +If you had a custom pgdocker or awxcompose location, you will need to set the `postgres_data_dir` and `old_docker_compose_dir` variables. + +1. Run the [migrate playbook](./ansible/migrate.yml) to migrate your data to the new postgresql container and convert the data directory to a volume mount. +```bash +$ ansible-playbook migrate.yml -e "migrate_local_docker=true" -e "postgres_data_dir=~/.awx/pgdocker" -e "old_docker_compose_dir=~/.awx/awxcompose" +``` + +2. Change directory to the top of your awx checkout and start your containers +```bash +$ make docker-compose +``` + +3. After ensuring your data has been successfully migrated, you may delete your old data directory (typically stored at `~/.awx/pgdocker`). + + +### Migrating data manually + +1. With Local Docker still running, perform a pg_dumpall: +> Note: If Local Docker is no longer running +`docker-compose -f ~/.awx/awxcompose/docker-compose.yml up postgres` + + ```bash + $ docker-compose -f ~/.awx/awxcompose/docker-compose.yml exec postgres pg_dumpall -U awx > awx_dump.sql + ``` + +2. Remove all local docker containers (specifically awx_postgres) +```bash +$ docker -f rm awx_postgres +``` + +3. Template the new docker-compose.yml +```bash +$ ansible-playbook -i tools/ansible/inventory tools/ansible/sources.yml +``` + +4. Start a container with a volume (using the new tools/docker-compose/_sources/docker-compose.yml) +```bash +$ docker-compose -f ../docker-compose/_sources/docker-compose.yml up postgres +``` + +5. Restore to new `awx_postgres` +```bash +$ docker-compose -f ../docker-compose/_sources/docker-compose.yml exec -T postgres psql -U awx -d awx -p 5432 < awx_dump.sql +``` + +6. Run the docker-compose.yml to start the containers +```bash +$ docker-compose -f ../docker-compose/_sources/docker-compose.yml up task +``` + +7. Check to ensure your data migration was successful, then you can delete your the `awx_dump.sql` backup and your old data directory. diff --git a/tools/docker-compose/docs/logstash.md b/tools/docker-compose/docs/logstash.md new file mode 100644 index 000000000000..2e75186b92c5 --- /dev/null +++ b/tools/docker-compose/docs/logstash.md @@ -0,0 +1,49 @@ +# How to use the logstash container + +#### Modify the docker-compose.yml + +Uncomment the following lines in the `docker-compose.yml` + +``` +#- logstash +... + +#logstash: +# build: +# context: ./docker-compose +# dockerfile: Dockerfile-logstash +``` + +POST the following content to `/api/v2/settings/logging/` (this uses +authentication set up inside of the logstash configuration file). + +``` +{ + "LOG_AGGREGATOR_HOST": "http://logstash", + "LOG_AGGREGATOR_PORT": 8085, + "LOG_AGGREGATOR_TYPE": "logstash", + "LOG_AGGREGATOR_USERNAME": "awx_logger", + "LOG_AGGREGATOR_PASSWORD": "workflows", + "LOG_AGGREGATOR_LOGGERS": [ + "awx", + "activity_stream", + "job_events", + "system_tracking" + ], + "LOG_AGGREGATOR_INDIVIDUAL_FACTS": false, + "LOG_AGGREGATOR_TOWER_UUID": "991ac7e9-6d68-48c8-bbde-7ca1096653c6", + "LOG_AGGREGATOR_ENABLED": true +} +``` + +> Note: HTTP must be specified in the `LOG_AGGREGATOR_HOST` if you are using the docker development environment. + +An example of how to view the most recent logs from the container: + +``` +docker exec -i -t $(docker ps -aqf "name=tools_logstash_1") tail -n 50 /logstash.log +``` + +#### How to add logstash plugins + +Add any plugins you need in `tools/elastic/logstash/Dockerfile` before running the container. diff --git a/tools/docker-compose/inventory b/tools/docker-compose/inventory new file mode 100644 index 000000000000..6a55bf6c4688 --- /dev/null +++ b/tools/docker-compose/inventory @@ -0,0 +1,19 @@ +localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python3" + +[all:vars] + +# AWX-Managed Database Settings +# If left blank, these will be generated upon install. +# Values are written out to tools/docker-compose/_sources/secrets/ +# pg_password="" +# broadcast_websocket_secret="" +# secret_key="" + +# External Database Settings +# pg_host="" +# pg_password="" +# pg_username="" +# pg_hostname="" + +# awx_image="ansible/awx" +# migrate_local_docker=false diff --git a/tools/docker-compose/start_tests.sh b/tools/docker-compose/start_tests.sh index f9af1107b751..66f4f8ed1664 100755 --- a/tools/docker-compose/start_tests.sh +++ b/tools/docker-compose/start_tests.sh @@ -5,5 +5,5 @@ cd /awx_devel make clean make awx-link -cp awx/settings/local_settings.py.docker_compose awx/settings/local_settings.py +cp tools/docker-compose/ansible/roles/sources/files/local_settings.py awx/settings/local_settings.py make "${1:-test}" diff --git a/tools/docker-credential-plugins-override.yml b/tools/docker-credential-plugins-override.yml index bec2953d25d5..0f9c44dac56b 100644 --- a/tools/docker-credential-plugins-override.yml +++ b/tools/docker-credential-plugins-override.yml @@ -18,9 +18,10 @@ services: conjur: image: cyberark/conjur + container_name: tools_conjur_1 command: server -p 8300 environment: - DATABASE_URL: postgres://postgres@postgres/postgres + DATABASE_URL: postgres://awx@postgres/postgres CONJUR_DATA_KEY: 'dveUwOI/71x9BPJkIgvQRRBF3SdASc+HP4CUGL7TKvM=' depends_on: - postgres diff --git a/tools/elastic/docker-compose.elastic-override.yml b/tools/elastic/docker-compose.elastic-override.yml index ef99a0cdbc0d..ded304ea502b 100644 --- a/tools/elastic/docker-compose.elastic-override.yml +++ b/tools/elastic/docker-compose.elastic-override.yml @@ -10,7 +10,7 @@ services: # Components of ELK stack for logging elasticsearch: - build: elastic/elasticsearch/ + build: ../../elastic/elasticsearch/ ports: - "9200:9200" - "9300:9300" @@ -18,12 +18,11 @@ services: ES_JAVA_OPTS: "-Xms1g -Xmx1g" # networks: # add back in when a connection to tower_tools is possible # - docker_elk - logstash: - build: elastic/logstash/ + build: ../../elastic/logstash/ command: -f /etc/logstash/conf.d/ volumes: - - ./elastic/logstash/config:/etc/logstash/conf.d + - ../../elastic/logstash/config:/etc/logstash/conf.d ports: - "8085:8085" links: @@ -33,9 +32,9 @@ services: depends_on: - elasticsearch kibana: - build: elastic/kibana/ + build: ../../elastic/kibana/ volumes: - - ./elastic/kibana/config/:/opt/kibana/config/ + - ../../elastic/kibana/config/:/opt/kibana/config/ ports: - "5601:5601" links: diff --git a/tools/redis/redis_socket_standalone/.dir_placeholder b/tools/redis/redis_socket_standalone/.dir_placeholder deleted file mode 100644 index 7660bfed95f5..000000000000 --- a/tools/redis/redis_socket_standalone/.dir_placeholder +++ /dev/null @@ -1 +0,0 @@ -This dir must pre-exist and be owned by the user you are launching awx dev env as. If the dir does not exist before launching the awx dev environment then docker will create the dir and it will be owned by root. Since we start our awx dev environment with user: ${CURRENT_UID} the redis container will be unable to create a socket file in a directory owned by root.