Internal Server Error, data corruption

[maksimluzik]

This is

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Docker / Postgres volume mount

SUMMARY

This relates to the bug reported earlier, but got closed? #796
Issue is still valid and it seems to be caused by the mounting postgres volume to the host's /tmp/ folder. Better solution would be to mount postgres data to /home/username/.pgdocker or /root/pgdocker (or some other non temporary location)

ENVIRONMENT

• AWX version: AWX 1.0.2.372
• AWX install method: docker on linux
• Ansible version: Ansible 2.4.2.0
• Operating System: CentOS (happens also on Debian and probably any other Linux distro/Unix as well)
• Web Browser: Does not depend on the browser

STEPS TO REPRODUCE

Install AWX setup, let it be and run for ~1 month or more. See 502 bad gateway error, when accessing AWX from the browser. Restart instance, see Internal Server Error.

EXPECTED RESULTS

To be able to access AWX, not getting any data corrupted.

ACTUAL RESULTS

Postgres docker instance logs: "initdb: directory "/var/lib/postgresql/data" exists but is not empty" and FATAL: could not open file "base/16384/2601": No such file or directory. Checkin the path confirms, that the data is gone.

ADDITIONAL INFORMATION

[matburt]

If you are still using the same pg container from before the original change was made then you'll continue to have that problem. You need to move to the new image layout. You are free to move the default pg folder location somewhere other than /tmp or, ideally, manage an external postgres database.

[maksimluzik]

But the problem is not in the container, as it is in AWX repo, here is the issue:

https://github.com/ansible/awx/blob/devel/installer/inventory

Docker common parameter is set to
postgres_data_dir=/tmp/pgdocker

I think this should be something else than /tmp/ by default?

[matburt]

You can change that to whatever you want when you run the installer. The idea is to provide a normalized place that will be generally available and accessible for most users.

[maksimluzik]

I see that many will stumble upon this issue, as for example many are using this AWX configuration as default (including vagrant setups etc.). When you spin up instance you have no idea about this postgres path (i.e. for example projects like these: https://github.com/geerlingguy/awx-container). At some point your setup will crash, because /tmp/ directory is emptied and your data is corrupted. I myself for example have used a lot of time to troubleshoot this and lost data because of this. In my opinion this is design flaw, which should be addressed. I am a bit surprised that this is not seen as issue. If the problem is about accesibility, then for example /var/pgdocker would be even better. Or as docker is run as sudo, then I do not see myself any issues putting it in /root/pgdocker

[maksimluzik]

It seems also to be painful to replace if you do not want to fork the repo. As using variable through many roles seems to be problematic with Ansible. At least I couldn't make it work, when specified in roles declaration:

- { role: geerlingguy.awx, postgres_data_dir: '/root/pgdocker' }

or adding into vars declaration:

vars:
    postgres_data_dir: "/root/pgdocker"

does not seem to take effect.

[matburt]

Last thing before I walk away from this issue, we don't maintain @geerlingguy's repository. We cover our officially supported installation methods in exhaustive detail, here's the relevant section that lists the tunables and how to apply them: https://github.com/ansible/awx/blob/devel/INSTALL.md#inventory-variables

If you have a problem with that other role then you need to take it up with that maintainer.

[geerlingguy]

Yes, please open an issue/file a PR with that role, and note that I have not had much time to make any improvements of late. The role is currently mostly a proof of concept meant to be forked and used how you see fit. The primary reason I created the role was because there was no other simple way of getting AWX set up on a machine at that time.

[maksimluzik]

I am a bit puzzled @matburt even though it is geerlingguy.awx repo same thing applies also to the official ansible/awx repo. I did not manage to override the variable in any case, the link you provided to manage inventory-variable means I would need to fork this repo, but that means I would need to fork 2 other repos as well just to make this work. And I still can not understand how this is indifferent, the code out of the box is broken. Why it should be so that anyone who wants to use AWX need to fix it by himself every time by specifying correct postgres path? This is just absurd, what is the benefit of having it in /tmp/ folder? I do not see any. I was not the first one with this issue, nor I will be the last one.

[Lupi619]

I am also experiencing this issue. I am new to Ansible AWX but have installed just the defaults to play around with it and get more familiar with it and try to see if I can find proper use for it at my job, which it would be a huge help. However every time I have to work on another project and come back I get the Internal: Server Error. @matburt is this going to happen to everyone using the defaults? It would be helpful to have a step by step with the install and not just as a tunable. If this is the default wouldn't it be considered a bug? Still unsure how to fix my issue as I am still fairly new to this.

[steaksauce-]

Seems like the issue is ~ "using /tmp/ for pgdata breaks after a while", and the response is ~ "/tmp is readily available to anyone".

While I don't know if it's well documented anywhere, you can easily modify the awx/installer/inventory file:

# line postgres_data_dir=/tmp/pgdocker
postgres_data_dir=/whatever/you/want

I've tested this working.

While Tower has been around a while, AWX is still young and will mature in both documentation and overall product over time. As a non-developer (I am operations), most of my pull-requests (which are few in numbers) are documentation fixes. If something comes up that is not expected as per the documentation, we as the community can submit a PR to update the documentation.

I guess in this case, would it be okay to add a warning about using defaults for postgres (ie, the default password is a security risk, and the default postgres_data_dir can bork your data) and just a quick snippet of information on how to change these values?

[arjunnambiartc]

@maksimluzik ..i have encountered the same error..While i know the fix is to change the mount path, is there any way to restore the old data without having to nuke everything and start afresh ?

[maksimluzik]

@arjunnambiartc you might be able to restore some data, but it might be incomplete and broken (a lot of manual work). In my case I saw it was easier to re-enter the data (through UI) that I lost into a new fresh setup than start playing with data recovery tools. Depends of course how much data did you loose? You could try some approaches like this one: https://unix.stackexchange.com/questions/80270/unix-linux-undelete-recover-deleted-files. Good luck!

[abedwardsw]

also agree that it would be nice to see the default change to /var or something other than /tmp since old /tmp data files are removed automatically in a lot of cases by the OS

[flavored-soda]

So, this was actually a major headache for me today and now I get to go out how to uninstall these docker containers due to this wonderful impediment to my workflow.. was hoping to be productive today -- would have done this much different if the auto-break "feature" wasn't here. Whenever you guys fix this, you need to write a fix and migrate script, because being new enough to NOT know this would happen (and dump a bunch of work in something now I have to decide IF I'll stick with) I have to decide : 1) if this is even worth my time, and 2) how to try and salvage any configuration I did before the auto-breaking configuration. Really, who would install a database in the OS maintained tmp directory?

[ssaammoo]

If I understand correctly .. then postgres will break any time if I reboot the host where AWX is installed on ?

[steaksauce-]

If I understand correctly .. then postgres will break any time if I reboot the host where AWX is installed on ?

Unless you change the directory that it lies in.

[inhumantsar]

You can change that to whatever you want when you run the installer

@matburt This is not reasonable for a variable that's essentially undocumented. No one is going to know/think to change it until they've already been burned by this problem.

It's not OK to use unsafe defaults. Especially considering that there are plenty of safe alternatives.