Unit king-phisher.service entered failed state.

[Deensk]

Issue Description

I'm having issue starting king-phisher server after matching PostgreSQL DB password with King-Phisher server. I just upgraded to the latest version 1.13.1 and service wont start.
found out it was complaining about "CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM pg_user' via su and psql" and I ALTER USER king_phisher WITH PASSWORD 'new password'; but still wont start.

Reproduction Step

systemctl start king-phisher.service
result output
Job for king-phisher.service failed because a configured resource limit was exceeded. See "systemctl status king-phisher.service" and "journalctl -xe" for details.

checked journalctl -xe with this command "journalctl -u king-phisher"
and result output is
[root@d1terll.com king-phisher]# journalctl -u king-phisher
-- Logs begin at Mon 2019-07-15 19:10:57 EDT, end at Tue 2019-07-16 10:17:15 EDT. --
Jul 15 19:11:11 d1terll.com systemd[1]: Starting King Phisher Server...
Jul 15 19:11:13 d1terll.com python3[1410]: Loading .env environment variables…
Jul 15 19:11:17 d1terll.com systemd[1]: PID file /var/run/king-phisher.pid not readable (yet?) after start.
Jul 15 19:11:17 d1terll.com runuser[2653]: pam_unix(runuser:session): session opened for user postgres by (uid=0)
Jul 15 19:11:17 d1terll.com su[2680]: (to postgres) root on none
Jul 15 19:11:18 d1terll.com python3[1410]: CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM pg_u
Jul 15 19:11:18 d1terll.com systemd[1]: king-phisher.service never wrote its PID file. Failing.
Jul 15 19:11:18 d1terll.com systemd[1]: Failed to start King Phisher Server.
Jul 15 19:11:18 d1terll.com systemd[1]: Unit king-phisher.service entered failed state.
Jul 15 19:11:18 d1terll.com systemd[1]: king-phisher.service failed.

when i check status, i get below output as well.

[root@d1terll.com king-phisher]# systemctl status king-phisher.service
● king-phisher.service - King Phisher Server
Loaded: loaded (/usr/lib/systemd/system/king-phisher.service; enabled; vendor preset: disabled)
Active: failed (Result: resources) since Tue 2019-07-16 10:17:15 EDT; 11min ago
Process: 24916 ExecStart=/usr/bin/python3 /opt/king-phisher/KingPhisherServer /opt/king-phisher/server_config.yml (code=exited, status=0/SUCCESS)

Jul 16 10:17:14 d1terll.com systemd[1]: Starting King Phisher Server...
Jul 16 10:17:14 d1terll.com python3[24916]: Loading .env environment variables…
Jul 16 10:17:15 d1terll.com runuser[24929]: pam_unix(runuser:session): session opened for user postgres by (uid=0)
Jul 16 10:17:15 d1terll.com su[24952]: (to postgres) root on none
Jul 16 10:17:15 d1terll.com python3[24916]: CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM ... and psql
Jul 16 10:17:15 d1terll.com systemd[1]: PID file /var/run/king-phisher.pid not readable (yet?) after start.
Jul 16 10:17:15 d1terll.com systemd[1]: king-phisher.service never wrote its PID file. Failing.
Jul 16 10:17:15 d1terll.com systemd[1]: Failed to start King Phisher Server.
Jul 16 10:17:15 d1terll.com systemd[1]: Unit king-phisher.service entered failed state.
Jul 16 10:17:15 d1terll.com systemd[1]: king-phisher.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Environment Details

Host OS: NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
King Phisher Version:1.13.1

Error Details / Stack Trace

CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM pg_u

[zeroSteiner]

Please completely fill out the issue template. We need all of the details that it requests to better identify the issue and provide the help that you are requesting. Please do not open a new issue but instead just edit this one and fill out all of the details.

[Deensk]

Thanks for the info zeroSteiner, I updated the ticket with more info.
Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[zeroSteiner]

@wolfthefallen can you take a look at this when you have a chance?

[wolfthefallen]

Sure I should have sometime later this week to look into this.

[Deensk]

I also wondering if we need to increase any ulimit values such us open files limit?

[zeroSteiner]

If you're still getting the same error, it looks like it has something to do with the database connection. I'm basing that on the fact that it's a query which is failing.

Can you run the server with debug output and in the foreground, then post the logs here?

The full arguments would be sudo ./KingPhisherServer -L DEBUG -f server_config.yml.

[Deensk]

Below is the output i got after running the debug.
[root@d1terll king-phisher]# sudo ./KingPhisherServer -L DEBUG -f server_config.yml
DEBUG target directory: /opt/king-phisher
INFO checking for the pipenv environment
DEBUG pipenv path: '/usr/bin/pipenv'
DEBUG pipenv Pipfile: /opt/king-phisher/Pipfile
Loading .env environment variables…
DEBUG king phisher version: 1.13.2 (rev: 63507ae) python version: 3.5.6
DEBUG plugin dependency path: /root/.local/lib/king-phisher/python3.5/site-packages
INFO listening on 0.0.0.0:80
INFO serving files has been enabled
INFO initializing database connection with driver postgresql
DEBUG using postgresql-setup to ensure that the database is initialized
DEBUG postgresql service is already running via systemctl
CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM pg_user' via su and psql
INFO unloading 0 plugins
[root@d1terll king-phisher]#

[zeroSteiner]

So the issue appears to be during the initialization of the database. The command su postgres -c psql -At -c "SELECT username FROM pg_user" is failing for some reason. Could you drop into a root shell and run that command then post the output?

I haven't seen this error before and am not quite sure what would cause that to fail. Maybe if the postgres user isn't available, or if they don't have the permissions to query the database? Just a guess here, I'm thinking maybe the postgres user isn't available or the username is different on RHEL.

[Deensk]

No Luck
[root@d1terll ~]# su postgres -c psql -At -c "SELECT username FROM pg_user"
su: invalid option -- 'A'
postgres service is running and i can log into su -postgress and then run psql to see all the database users.

[zeroSteiner]

You might need to do su postgres -c 'psql -At -c "SELECT usename FROM pg_user"' instead. Notice the single ticks around the psql sub command. It looks like the way I originally wrote it caused the psql arguments to be incorrectly interpreted as su arguments. I also misspelled usename as username.

For reals though, this command should give us a meaningful error: su postgres -c 'psql -At -c "SELECT usename FROM pg_user"'

[Deensk]

sure, make sense. So when I ran the command it asked me for password and I enter the postgres PW and got the result below

[root@d1terll ~]# su postgres -c 'psql -At -c "SELECT usename FROM pg_user"'
could not change directory to "/root"
Password:
kpost
test1user
postgres
koopal
king_phisher
[root@d1terll ~]#

[zeroSteiner]

It prompted you for a password? That's probably the issue. I don't know why it would do that since you're running as root.

A quick google search shows that maybe if your /etc/pam.d/su file is missing the line auth sufficient pam_rootok.so that you might be prompted for the password even while running as root.

[Deensk]

Right, i got the prompt running it again. I will have to into my pam file to try and change that.

[Deensk]

The PAM file seems to be there already but still same asking for a password. No luck.

[root@d1terll ~]# grep pam_root /etc/pam.d/su
auth sufficient pam_rootok.so

[zeroSteiner]

The issue is definitely that your system is, for some reason prompting you for a password when you use su from the root account. I have no idea why that would be but it would definitely cause issues with the King Phisher startup routine.

Since the service is already running, I'm not sure we could get around this startup check without a code change to effectively bypass it.

[wolfthefallen]

@Deensk check your pam su file. On most linux based systems this is found at /etc/pam.d/su
make sure you have
auth sufficient pam_rootok.so
in the file and not commented out.

This line in the pam su configuration file allows root to su to another user with out being prompted for a password to that users account.

[zeroSteiner]

@wolfthefallen I think Deensk already said that was the case in the commend here.

[Deensk]

Yes, the auth sufficient pam_rootok.so seems to be ok.
Even recreating the king-phisher database still result in the same error.
python3[31668]: CRITICAL server failed to build with error: failed to execute postgresql query 'SELECT usename FROM pg_user' via su and psql
Do I need to modify anything else in PAM or Postgres file to bypass the su command?

[zeroSteiner]

It's not going to have anything to do with the database. The issue is entirely with the fact that su is prompting your root user for a password which since it's not that PAM configuration, I have no idea why that is.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Deensk]

Doing more research on this and still haven't found anything yet.

[Script-Nomad]

I'm curious if you could get around this by running pkexec command during the configuration steps in order to get around the sudo password prompt. This would cause the prompt to display via the local graphical interface to the user when needed. If sudo privileges are required either way, better to use a method that is guaranteed to be visible to the user.

[zeroSteiner]

Yeah but most of the time the server is installed on a headless system with no GUI.

[Deensk]

Yes, the install is in headless system with no GUI.
I want to try reinstalling everything and see.

[Deensk]

I tried to reinstall but getting the following error below. I also try clearing command "pipenv lock --clear" but still showing the same error.

[root@d1terll king-phisher]# sudo ./KingPhisherServer --verify-config server_config.yml
WARNING no pre-existing pipenv environment was found, installing it now
pipenv encountered the following error:
Installing dependencies from Pipfile.lock (d316a9)…
Installing initially failed dependencies…
ERROR failed to install the pipenv environment

[wolfthefallen]

navigate to your King-phisher folder and run:
export PIPENV_VENV_IN_PROJECT=True && pipenv --site-packages install
if you are having permission issues use:
export PIPENV_VENV_IN_PROJECT=True && sudo -E pipenv --site-packages install

[Deensk]

when i try "export PIPENV_VENV_IN_PROJECT=True && sudo -E pipenv --site-packages install"
it failed couple of times before exporting pipenv. server settings seems to passed but failed to start postgres service even though i have postgres 9.6 up and running.

[root@d1terll king-phisher]# sudo ./KingPhisherServer --verify-config server_config.yml
Loading .env environment variables…
[+] configuration verification passed
[+] all required settings are present
[root@d1terll king-phisher]# sudo ./KingPhisherServer -L DEBUG server_config.yml
DEBUG target directory: /opt/king-phisher
INFO checking for the pipenv environment
DEBUG pipenv path: '/usr/bin/pipenv'
DEBUG pipenv Pipfile: /opt/king-phisher/Pipfile
Loading .env environment variables…
DEBUG king phisher version: 1.14.0 (rev: 4e8de56) python version: 3.5.6
DEBUG using plugin-specific library path: /var/king-phisher/lib/python3.5/site-packages
INFO listening on 0.0.0.0:80
INFO serving files has been enabled
INFO initializing database connection with driver postgresql
DEBUG using postgresql-setup to ensure that the database is initialized
DEBUG starting external process: /usr/bin/postgresql-setup --initdb
DEBUG starting external process: /usr/bin/systemctl status postgresql.service
INFO postgresql service is not running, starting it now via systemctl
DEBUG starting external process: /usr/bin/systemctl start postgresql
[root@d1terll king-phisher]# ERROR failed to start the postgresql service via systemctl
CRITICAL server failed to build with error: postgresql service failed to start via systemctl
INFO unloading 0 plugins

[wolfthefallen]

From the debug log you provided King Phisher is not seeing the postgresql service running, and fails to start it through systemctl.

Please trouble shoot your postgresql instance and make sure it is running through systemctl you can view postgresql logs with journalctl -u postgresql.service to view the errors of starting the service.

[Deensk]

Sure, got further on su -PostgreSQL command on startup. "su postgres -c psql -At -c "SELECT usename FROM pg_user"

[root@d1terll king-phisher]# sudo ./KingPhisherServer -L DEBUG server_config.yml
DEBUG target directory: /opt/king-phisher
INFO checking for the pipenv environment
DEBUG pipenv path: '/usr/bin/pipenv'
DEBUG pipenv Pipfile: /opt/king-phisher/Pipfile
Loading .env environment variables…
DEBUG king phisher version: 1.14.0 (rev: 4e8de56) python version: 3.5.6
DEBUG using plugin-specific library path: /var/king-phisher/lib/python3.5/site-packages
INFO listening on 0.0.0.0:80
INFO serving files has been enabled
INFO initializing database connection with driver postgresql
DEBUG using postgresql-setup to ensure that the database is initialized
DEBUG starting external process: /usr/bin/postgresql-setup --initdb
DEBUG starting external process: /usr/bin/systemctl status postgresql.service
DEBUG postgresql service is already running via systemctl
DEBUG starting external process: su postgres -c psql -At -c "SELECT usename FROM pg_user"

[zeroSteiner]

It looks like you're running into the same issue here where it's hanging because su is probably prompting for a password.

[Deensk]

right, definitely don't know why.

[Deensk]

Not sure if this will be secure enough, can we set postgres database not to prompt user password at all?
using MD5

[zeroSteiner]

Okay, I looked into this again and I'm thinking I might have been wrong about it being su that was prompting for the password. I was able to reproduce this on my system by adjusting the pg_hba.conf file to use md5 as the auth method for local connections.

Could you share the contents of your pg_hba.conf file? Mine is in /var/lib/pgsql/data/pg_hba.conf and you'll probably need root access to read it. From my understanding, the entries are read top to bottom, so the first that matches will be used as the configuration when a user goes to authenticate. The first non-commented line I have is

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer

You probably have one that is either higher that is matching, or a configuration using something other than peer. You don't necessarily need to use peer authentication for all local connections for all users on all databases. To fix this issue, I think you need to at least set the authentication to use peer for the postgres user with the syntax:

local   all             postgres                                peer

After changing that, restart the postgresql service and then see if King Phisher will start.

[Deensk]

Ok, this is how my config looks

postgres=# SHOW hba_file;
hba_file

/var/lib/pgsql/data/pg_hba.conf

"local" is for Unix domain socket connections only

local all all md5

IPv4 local connections:

host all all 127.0.0.1/32 md5
host all all 192.168.1.0/24 md5

IPv6 local connections:

host all all ::1/128 md5

Allow replication connections from localhost, by a user with the

replication privilege.

#local replication postgres trust
#host replication postgres 127.0.0.1/32 trust
#host replication postgres ::1/128 trust

permit local connections from the king_phisher user for the king_phisher database

host king_phisher king_phisher 127.0.0.1/32 md5

[zeroSteiner]

Yeah it's that local all all md5 line there towards the top that's causing the password to be prompted. Insert local all postgres peer right before it and then restart postgresql.

[Deensk]

Ok, that worked with su and DEBUG running periodic maintenance tasks continue to run
"executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all"
is that normal?

[root@d1terll king-phisher]# sudo ./KingPhisherServer -L DEBUG server_config.yml
DEBUG target directory: /opt/king-phisher
INFO checking for the pipenv environment
DEBUG pipenv path: '/usr/bin/pipenv'
DEBUG pipenv Pipfile: /opt/king-phisher/Pipfile
Loading .env environment variables…
DEBUG king phisher version: 1.14.0 (rev: 4e8de56) python version: 3.5.6
DEBUG using plugin-specific library path: /var/king-phisher/lib/python3.5/site-packages
INFO listening on 0.0.0.0:80
INFO serving files has been enabled
INFO initializing database connection with driver postgresql
DEBUG using postgresql-setup to ensure that the database is initialized
DEBUG starting external process: /usr/bin/postgresql-setup --initdb
DEBUG starting external process: /usr/bin/systemctl status postgresql.service
DEBUG postgresql service is already running via systemctl
DEBUG starting external process: su postgres -c psql -At -c "SELECT usename FROM pg_user"
DEBUG starting external process: su postgres -c psql -At -c "SELECT datname FROM pg_database"
[root@d01piroutil01 king-phisher]# DEBUG campaigns table not found, creating all new tables
DEBUG current database schema version: 9 (latest)
DEBUG alembic version table not found, attempting to create and set version
INFO alembic_version table initialized to c9a8d520a26
DEBUG connected to postgresql database: king_phisher
INFO restored 0 valid sessions and skipped 0 expired sessions from the database
DEBUG use pam service 'sshd' for authentication
DEBUG forked an authenticating process with pid: 9637
INFO the job manager has been started
INFO adding new job with id: 3941f0a7-309c-4968-a4e9-d26bf1b22fe5 and callback function: _maintenance
INFO adding new job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all
DEBUG web socket manager worker running in tid: 0x7f43a5be0700
DEBUG initialized the table api dataset (schema version: 8)
INFO including 4 custom http headers
INFO server running in process: 9603 main tid: 0x7f43ba888740
INFO dropped privileges to the nobody account (uid: 99, gid: 99)
DEBUG executing job with id: 3941f0a7-309c-4968-a4e9-d26bf1b22fe5 and callback function: _maintenance
DEBUG running periodic maintenance tasks
DEBUG executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all
DEBUG executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all
DEBUG executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all
DEBUG executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all
DEBUG executing job with id: 777a1e03-b1ba-4c19-a197-310af5d02cb8 and callback function: ping_all

[zeroSteiner]

Yes that is 100% intended behavior. You won't see those messages if logging is set above DEBUG, I think the default is WARNING but when you have it set to DEBUG you get all kinds of internal diagnostic messages that I use to figure out what's going on. That message in particular is the periodic ping that's sent over the websocket to any connected clients. The event fires on a timer regardless if there are any clients connected or not.

It looks like you're all set now so I'll close this ticket. I'm glad we were able to figure it out.

[Deensk]

Sure, Thank you guys for the help.