What is the right way to upgrade AWX?

[RogerSik]

What is the right way to upgrade AWX? In some comments it was mention to upgrade only the docker images and everything is fine. When I did this the AWX Webinterface was stuck at ,,Upgrading Please Wait". After researching some one mention there is no Upgrade Way. The only way to do is to export the AWX instance data and create a new instance and then import the AWX data.

Every way is for me fine. Would be only cool that under the README.md would this topic be covered.

[mkempster22]

Generally the first way you mentioned is usually how it is done, but the minor releases in AWX do have a tendency to break the upgrade path causing the stuck at upgrading issue meaning the only way is to wipe the database (and export import data)

Edit: 'Documentation' is here https://github.com/ansible/awx/blob/devel/DATA_MIGRATION.md

See issues #2173 & #138

[boris-42]

@mkempster22 the only problems with this apporach is that you cant export/import secrets which creates a lot of troubles in process...

[mkempster22]

Yeah there is a lot of issues with it. I'm not defending it, it's an awful way of upgrading especially with the AWX team not using SEMVER correctly

[ernst-at-tv2]

@boris-42 @mkempster22 do you know if anyone has been working on a solution to export/import secrets?

[mkempster22]

Beyond what has been developed in the tower-cli and awx-manage projects I haven't seen any other tools for exporting, I'm not sure if exporting more data such as secrets is in the pipeline for them.
Best place to keep up to date with those features would be here: https://github.com/ansible/tower-cli

[boris-42]

@ernst-at-tv2 I know that there is not going to be such functionality presented, basically the problem is that AWX REST api doesn't return secrets back... And it won't, because AWX team thinks that it's hard to make it secure and that AWX is not secrets management system..

The cleanest, but still ugly solution is to hack tower-cli dump command, let it accept postgres connection string, and secret_key and if it is provided it should enrich secrets resource with actual secrets. But I am sure this is not going to be merged in upstream at ever...

[LukeShortCloud]

Ansible Engineering is aware of the upgrade complications and is working towards a solution. See:

• Use semver for AWX version numbers #2203 (comment)
• Upgrade To 1.0.7 Fails #2173 (comment)

[MarkusTeufelberger]

It might be possible to store secrets (along with the rest of the tower/awx config) in a playbook that then runs https://docs.ansible.com/ansible/devel/modules/tower_credential_module.html tasks to add them to tower again.

[wenottingham]

See updates here: https://groups.google.com/forum/#!topic/awx-project/FKEZwrO-96k. Closing.

[t-readyroc]

@wenottingham - that post refers to Ansiblefest session topics? Was upgrading covered?

[wenottingham]

Yes, changes to how migrations are pushed to AWX so that sets of upgrades work was covered there.

[lingfish]

I'm sorry, but I'm still confused here as to the current state; I didn't see/find much in the above link.

Can we now upgrade easily (and it uses migrations?) Can we just bump the docker image version and off we go?

[dminca]

@lingfish I've managed to upgrade from 1.0.1.176 to 2.1.0 but I had the following configuration on a single Node (server):

• running PostgreSQL DB locally
• Kept the old configurations (inventory, role variables et. al.)
• just ran the Ansible playbook that executed the local_docker role and raised all those containers: awx_task, awx_web, memcached and rabbitmq

Now I'm struggling to move it to Kubernetes and although the configuration is the same it's complaining on some DB issues (PK and table relations) after I've imported it in the new PostgreSQL instance from K8s.

LE: managed to migrate it to Kubernetes. Just make sure that you first create a K8s PV which holds your PostgreSQL dump file, start a K8s Job which mounts this PV and imports the PostgreSQL dump into your running PostgreSQL Pod(s). Put these instructions before the Migrate database task and you're all done upgrading to latest version on Kubernetes.

[lingfish]

@wenottingham I realise devs don't comment on closed tickets, but could you make an exception here please and let us know?

[lingfish]

@dminca Interesting info. I'm merely trying to go from 2.1.0 to 2.1.2, and no Kuber. Single node, but external Postgres.

Are you saying you just bumped your versions, and used their playbook to redeploy, migrations happened and all good?

[dminca]

@lingfish yes ofc. The most important bit from the whole picture is your PostgreSQL dump. Have that backed up and you can upgrade it easily.

Destroy the old cluster after you got your DB dump, then raise a new one (latest) and import the PostgreSQL dump before the migrations are being done. It's just a Django app after all...

[sdktr]

@dminca so you didn't reuse the database? merely build out a complete new setup and imported your backup psql dump before AWX started the db migrations?

I'm in the same boat as @lingfish , using k8s already with an external psql database. My upgrade scenario is from 2.0.1 to 2.1.2 however.
Should 'just bumping the container versions' take care of dbmigrations when the new containers start?

Still have to try out the different scenarios myself, but given the amount of complaints about AWX upgrading I'm trying to get as much input beforehand as possible

[dminca]

@sdktr it's almost the same situation. Given that you're already having a running PostgreSQL on another host, destroying everything (except the DB of course), bumping the version to latest, and starting them back up again should do the trick.

[lingfish]

Right, as this is exactly what database migrations are for... I'd just still love a dev to chime in here and confirm the process.

[albeus]

I've managed to upgrade from 2.1.0 to 2.2.1. I use docker compose setup method.

I confirm that creating a new cluster and dumping the old posgres db works.

1. Back up old db using this compose command:
   docker-compose exec postgres pg_dumpall -U postgres --clean > ~/awx_pgdumpall_2018-12-14.sql

2. Stop and clean old cluster.

3. Create a new awx cluster using the installer.

4. Copy the sql dump file to postgres volume (in my case it was
   /usr/local/lib/awx/postgresql/, see postgres_data_dir variable in
   installer/inventory):
   cp ~/awx_pgdumpall_2018-12-14.sql /usr/local/lib/awx/postgresql/

5. Stop cluster and start only postgres
   docker-compose stop docker-compose start postgres

6. Restore the db in new cluster
   docker-compose exec postgres /bin/bash root@ec7356d8ac8c:/# psql -U postgres < /var/lib/postgresql/data/awx_pgdumpall_2018-12-14.sql

This way I can see all my data, credentials, history and so on in upgraded awx instance.

The only unclean thing I noticed is this error in postgres log:

ERROR:  duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(admin) already exists.
STATEMENT:  INSERT INTO "auth_user" ("password", "last_login", "is_superuser", "username", "first_name", "last_name", "email", "is_staff", "is_active", "date_joined") VALUES ('****', NULL, true, 'admin', '', '', 'root@localhost', true, true, '2018-12-14T17:10:26.854629+00:00'::timestamptz) RETURNING "auth_user"."id"

But despite it, all seems working well.

[albeus]

The only unclean thing I noticed is this error in postgres log:

ERROR: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL: Key (username)=(admin) already exists.
STATEMENT: INSERT INTO "auth_user" ("password", "last_login", "is_superuser", "username", "first_name", "last_name", "email", "is_staff", "is_active", "date_joined") VALUES ('****', NULL, true, 'admin', '', '', 'root@localhost', true, true, '2018-12-14T17:10:26.854629+00:00'::timestamptz) RETURNING "auth_user"."id"

I reproduced this problem.

The responsible is the entry point of docker image awx_task. To be precise this is the command that in my case produce the error in postgres: launch_awx_task.sh - line 21.

ansible@awx-server: ~$ docker exec -it awx_task_2.1.2_1_783078bcc559 /bin/bash
[root@awx awx]# echo "from django.contrib.auth.models import User; User.objects.create_superuser('admin', 'root@localhost', 'password')" | awx-manage shell
...
django.db.utils.IntegrityError: duplicate key value violates unique constraint "auth_user_username_key"
DETAIL:  Key (username)=(admin) already exists.

So I confirm that the upgrade procedure was fine. This error is not related to the upgrade but it's probably a bug of "awx task" container.

Coud anyone confirm this?

[dminca]

You got a duplicate key error because you either

• had data in the PosgreSQL DB before importing the dump
• imported the dump with an unprivileged user

I had that error also when importing the dump after doing the migration.

[orgito]

Since 2.1.0 simply running the playbook again have been working for me. No need to export with tower-cli or dump and restoring the database.

[Commifreak]

Since 2.1.0 simply running the playbook again have been working for me. No need to export with tower-cli or dump and restoring the database.

Exact what I thought. Since AWX uses DB-migration it should detect an existing DB and upgrade itself.

I just a few momemts away from updating from 2.1.2 to 3.0.1 using docker method. I hope it works, since the official way is still export and import data via a third party tool afaik.

[Commifreak]

Yep, worked. the mentioned duplicate entry error showed up but I ignored it.

[Commifreak]

Worked again with upgrade to 4.0.0

No negative points so far. I dont know why upgrading is that bad documented (same as the changelogs for awx ;) ). There are several tools that export awx data and import it in a new instance. Never had to use this.

I always prune the system and save the pg_datadir.

#!/bin/bash
## This stops and removes ALL containers/images/volumes!

BUDIR=/opt/ansible_backups/$(date '+%d.%m.%Y_%T')

docker stop $(docker ps -a -q)
mkdir -p $BUDIR
cp -r /opt/awx-* /opt/containerd /opt/pgdocker "$BUDIR"

docker system prune -a --volumes

Download new awx zip, adapt inventory, run the playbook. Done.

[clotted]

Download new awx zip, adapt inventory, run the playbook. Done.

@Commifreak For the uninitiated can you elaborate on "adapt inventory"?

[Commifreak]

I just mean to check the installer/inventory file. Some options may changed. I always have to change the pg_datadir as an example. (Default is still /tmp/).

[Commifreak]

This is the awx release .zip file unzipped with settings made to inventory-file prior installing the awx containers. Just leaving this there in case I habe to re-create the containers with the specific version.

[Commifreak]

Did you switched AWX version?

[Commifreak]

Interesting. I didnt had this problem. Iam not using LDAP, maybe this is the key difference? Gruß, R. Kluth

…

Am 29.04.2019 um 18:44 schrieb Renato Goncalves ***@***.***>: @Commifreak yes, from 3.0.1 to 4.0.0 I was able to log into the UI using my LDAP user, and was able to reset my admin user. I noticed that this proced duplicated my admin user. Not sure why... It is working on 4.0.0 now. Thank you :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nicolaibaralmueller]

1. docker-compose exec postgres pg_dumpall -U postgres --clean > ~/awx_pgdumpall_2018-12-14.sql

I receive below error when running your command.

`
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

`

[albeus]

1. docker-compose exec postgres pg_dumpall -U postgres --clean > ~/awx_pgdumpall_2018-12-14.sql

I receive below error when running your command.

`
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

Docker compose commands must be run in the proper directory (where is your docker-composer configuration file). So ensure that you run the command after you have entered in that directory.

[nicolaibaralmueller]

1. docker-compose exec postgres pg_dumpall -U postgres --clean > ~/awx_pgdumpall_2018-12-14.sql

I receive below error when running your command.
`
ERROR:
Can't find a suitable configuration file in this directory or any
parent. Are you in the right directory?

    Supported filenames: docker-compose.yml, docker-compose.yaml

Docker compose commands must be run in the proper directory (where is your docker-composer configuration file). So ensure that you run the command after you have entered in that directory.

Thanks for the reply. I guess I'm missing some undestanding.

My AWX docker environment is up and running. So I have to go into /tmp/dockercompose where docker-compose.yml is located without editing the file first? FYI: I'm new to docker.

[albeus]

My AWX docker environment is up and running. So I have to go into /tmp/dockercompose where docker-compose.yml is located without editing the file first? FYI: I'm new to docker.

/tmp/ is not a good choice: at the next reboot you will lose your configurations. Docker-compose directory should be placed in a persistent location (eg. /usr/local/lib/) .

Anyway, yes, you should go into /tmp/dockercompose before running the docker-compose command.

[nicolaibaralmueller]

My AWX docker environment is up and running. So I have to go into /tmp/dockercompose where docker-compose.yml is located without editing the file first? FYI: I'm new to docker.

/tmp/ is not a good choice: at the next reboot you will lose your configurations. Docker-compose directory should be placed in a persistent location (eg. /usr/local/lib/) .

Anyway, yes, you should go into /tmp/dockercompose before running the docker-compose command.

Thanks a bunch. I will correct the directory in the future.

I'm trying to find the best awx backup/restore strategy and so far tower-cli receive/send is awefull.
There is git repository awx-migrate which can export credentials with sensitive information (in clear text) but also uses tower-cli.

It would be best to be able to backup the database and restore with a new awx environment BUT what about potential database changes? Will a new awx version installation always work with a different awx version database backup?

[orgito]

Why not using docker instead of docker-compose?

docker exec awx_postgres pg_dumpall -U awx --clean > ~/awx_dump_$(date +%F).sql

assuming the postgres username is awx.

[nicolaibaralmueller]

Why not using docker instead of docker-compose?

docker exec awx_postgres pg_dumpall -U awx --clean > ~/awx_dump_$(date +%F).sql

assuming the postgres username is awx.

Yes why not. Sound easier. What is the advantage of using docker compose with the command?

[nicolaibaralmueller]

My AWX docker environment is up and running. So I have to go into /tmp/dockercompose where docker-compose.yml is located without editing the file first? FYI: I'm new to docker.

/tmp/ is not a good choice: at the next reboot you will lose your configurations. Docker-compose directory should be placed in a persistent location (eg. /usr/local/lib/) .

Anyway, yes, you should go into /tmp/dockercompose before running the docker-compose command.

Sadly following your guide did not work for awx 4.0.0.

Too many errors to count during the psql import.

[Commifreak]

Just upgraded AWX 5.0.0 again with no issues using my script mentioned earlier. Remove docker containers, load new zip, configure inventory, run the playbook - all ok.

I still dont know why everybody says its complicated to upgrade. Had no (really NO) issues since 2.1.2 (from where I started using it).

The only thing whats bothering me: No readable changelogs. But wait, I heard of this nice site: https://groups.google.com/forum/#!topic/awx-project/kRLrbhL4Y5o

[giovannifl]

Just used @Commifreak approach to migrate from AWX 5.0.0 to AWX 6.0.0 and it worked like a charm.
Thanks @Commifreak

[ybizeul]

Upgrade doesn't work for me with @Commifreak method.

Maybe my database has been created in a very early version of AWX and that's causing an issue, but playbook fails to upgrade PostgreSQL database.

database user "awx" is not the install user

Opened #5214

[ybizeul]

Re-reading the comments I just realized no one here seem to have upgraded to 8.0. Could you please let the thread know when you do it ?

[Commifreak]

Re-reading the comments I just realized no one here seem to have upgraded to 8.0. Could you please let the thread know when you do it ?

You are right, me neither. I‘ll try it asap and let you know what happens

[AechLyons]

Maybe I'm just docker stupid or postgres retarded but I fluked getting a upgrade and restore without loosing any information (including ldap) a few months ago.......Now I have burned 3 days trying to get another ansible host with the same info from the DB. In the last few days I have wasted I have also tried to get awx deployment to work with an external PGDB....the closest I got was either the site coming up with a server error or perpetually in a upgrading page and never going anywhere....typically it site wont even come up.

I get this is the bleeding edge version of the tower product but I dont have budget to buy tower. The inability to migrate or restore without having to setup ldap again and have all my users login again or set all my projects and schedules up and set permissions again is beyond frustrating. I can't even dump the PG database and restore it properly.

I though if I could at least save the DB or better yet use a completely different DB that I could have 2 different AWX servers use would be great that way I can take advantage of the development cycle.

All this started because there are a few modules that have been updated for ansible 2.9 I wanted to use. Aggravation is beyond 10. I've given up. Hell if I could get some help just getting the awx deployment to work with a external DB that would be awesome.

@Commifreak I tried your method and failed miserably.....I might as well have just stood up a brand new cluster......recreate my plays and redo LDAP and my user permissions. I've never had such a hard time with something even if it is open source.

[Commifreak]

Which version do you try to update to? There is currently another issue upgrading to AWX 8+ because it requires PostGres10 and the builtin upgrade script for the db fails. I still have no LDAP connected but this is indeed my next step after updating to 10. Gruß, R. Kluth

…

Am 21.11.2019 um 09:59 schrieb Aech ***@***.***>:  Maybe I'm just docker stupid or postgres retarded but I fluked getting a upgrade and restore without loosing any information (including ldap) a few months ago.......Now I have burned 3 days trying to get another ansible host with the same info from the DB. In the last few days I have wasted I have also tried to get awx deployment to work with an external PGDB....the closest I got was either the site coming up with a server error or perpetually in a upgrading page and never going anywhere....typically it site wont even come up. I get this is the bleeding edge version of the tower product but I dont have budget to buy tower. The inability to migrate or restore without having to setup ldap again and have all my users login again or set all my projects and schedules up and set permissions again is beyond frustrating. I can't even dump the PG database and restore it properly. I though if I could at least save the DB or better yet use a completely different DB that I could have 2 different AWX servers use would be great that way I can take advantage of the development cycle. All this started because there are a few modules that have been updated for ansible 2.9 I wanted to use. Aggravation is beyond 10. I've given up. Hell if I could get some help just getting the awx deployment to work with a external DB that would be awesome. @Commifreak I tried your method and failed miserably.....I might as well have just stood up a brand new cluster......recreate my plays and redo LDAP and my user permissions. I've never had such a hard time with something even if it is open source. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[orgito]

I recently upgrade from 4 to 7 and from 7 to 9.0.1 without any issues.

[orgito]

probably skipping the 8 upgrade avoided the DB script migration issue.

[shanemcd]

This won't help the issue you're experiencing, but for others who find this issue via a search in the future, I recently added a note to the install docs about upgrading.

[AechLyons]

@Commifreak I'm on 8 trying to upgrade to 9.0.1

[AechLyons]

and i can confirm that the docker container that houses postgres is PG10

[sigismund]

I recently upgrade from 4 to 7 and from 7 to 9.0.1 without any issues.

@orgito and how you managed to do this? Did you use this method?

[orgito]

@orgito and how you managed to do this? Did you use this method?

No, I just followed the upgrade procedure. Adjusted the variables in the inventory file and ran the install playbook. I also made a backup, just in case. But it was not necessary. The install playbook took care of the migrations.

I went directly to from 4 to 7 and from 7 to 9.0.1.

[stegr04]

I just wanted to express my gratitude to @Commifreak, @dminca, @albeus (and others) for all of your help while also trying to give back a little bit of what I'm learning. I'm using awx v9.0.1 and was trying to find a way to backup and restore my awx environment. I'm still not certain what is meant when referring to installer/installing. I'm using geerlingguy's playbook with (on centos7.6 with docker and selinux pip_install_packages - not docker-py and then using the tasks outlined by butler54 in geerlingguy/ansible-role-awx#30). This sets up my awx environment with docker containers.

While trying to find a backup/restore solution that worked for me I wasn't clear on whether Commifreak was using Albeus's steps or if Commifreak was purely using the backup of the pg_data_dir folder - though I understand its being used for a slightly different purpose (upgrade vs backup/restore). Nonetheless I found errors while trying to copy my pg_data_dir backup back to its expected location. I didn't dig into this too much.

I have several things being backed up now (pg_dumpall by albeus, containerd and pg_data_dir folders from Commifreak, and tower-cli export). But the pg_dumpall was giving me errors while I was trying to restore it. A lot more than the duplicate key message mentioned by many here. I'm new to postgres. So when I received an error about the postgres role not being available I assumed it was because I needed to use the db user created/used in the awx specific container being used in this version. The user is awx in the version I'm on. But I still had problems. This seems to stem from using the --clear during the pg_dumpall command which, if I understand its impact correctly, creates an instruction to drop the database before it recreates everything. This couldn't be done successfully without using: psql -U awx -d postgres < /path/to/pg_dumpall.sql

My instruction to restore it from outside of the docker container is:
docker-compose exec postgres sh -c "psql -U awx -d postgres < /var/lib/postgresql/data/pgdata/awx_pgdumpall.sql

Now on to trying to upgrade to 9.1.0. I suspect I'll learn more about the installer. :) I hope this helps someone.

[mnhan3]

anyone successfully upgrade from 9.3 to 10.0 or 13.0 using this method. I was successful in going from 6 -> 9.3 but have had no luck upgrading to 10 or higher. Attempting to so do fails in the awx-manage migrate step and it throws a python trc running

File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/migrations/0109_v370_job_template_organization_field.py", line 16, in rebuild_jt_parents
rebuild_role_parentage(apps, schema_editor, models=('jobtemplate',))

On a fresh installs of 6.0 -> 13.0 it works fine as there is no data to migrate. But I need to migrate my existing db. attempting to export and import the db into 13.0 fails miserably as the schema have changed. So looking for a way to manually do the migration as the installer just pukes on awx-manage migrate plays.

[G3ph4z]

@mnhan3 I didn't succeed either, I tried to upgrade in so many ways, but no one of them works. I just can't believe this.

[fferraro87]

I've also try to update from 13 to 15 in many ways but nothing doesn't works

[diegorebolledo]

Hey guys!

I've tried to upgrade since 11.2.0 to 16 but doesn't work!! I tried in many ways, backup and restore postgres and tower-cli configuration, just setting the inventory (without preload and with preload) and vars.yml (like the official info says), stopping and deleting the dockers, just running the install and always have the same issue, when tried to launch a job, alerts an ssh_key_unlock error:

POST /api/v2/job_templates/7/launch/ 400
ssh_key_unlock

Have been someone success upgrading at the last version of awx?

PD: when I tried in the official way (just executing the install and setting the inventory file) the container on awx_postgres cann't to start, it got restarting forever.