MHN Troubleshooting Guide

Brady Sullivan edited this page Oct 24, 2017 · 14 revisions

This is a Work in Progress troubleshooting guide to MHN, compiled from responses to questions sent to the MHN Google Group (modern-honey-network@googlegroups.com).

Password reset through the web app is not working and/or retrieving http://YOUR-SITE/static/mhn.rules causes a 404.

I believe this is because your celery worker is not working. Can you try this?

sudo supervisorctl status

Is the status of your "mhn-celery-worker" FATAL or something other than RUNNING? If so, then try this:

sudo supervisorctl restart

If the mhn-celery-worker process still does not start cleanly. Then try this:

cd /var/log/mhn/
sudo chown www-data mhn.log
sudo supervisorctl start mhn-celery-worker

If this fails (i.e. "sudo supervisorctl status" shows the process is not RUNNING after about 10 sec), then review these logs?

/var/log/mhn/worker.err
/var/log/mhn/worker.log
/var/log/mhn/mhn.log

This should steer you in the direction of the problem.

I can't login to the MHN server web app

Can you describe the error you're encountering when logging in? Password failure? Connection failure?

If password failure, can you try this (you should see similar output to mine below, if not, it indicates a problem). The select statement is useful to verify that the user you expect to exist does in fact exist.

    # sudo su -
# cd $MHN_HOME/server/

# sqlite3 mhn.db
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> select * from user;
1|USERNAME@SITE.com|sklfdjhkasdlfhklsadhfklasdhfkldsahklsd|1|

sqlite> .quit

# cd $MHN_HOME/
# . env/bin/activate
# cd server
# python manual_password_reset.py

Walk through the password reset and make sure you see user found, updating password at the end.

Troubleshooting the Honeypot side

  • I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor

Can you run these commands from your honeypot system and provide the output?

Before running these commands start a script to wget against your honeypot.

$ sudo netstat -luntp
$ sudo supervisorctl status
$ sudo iptables -L
$ sudo tcpdump -nnNN tcp port 10000

I am expecting output like this:

$ sudo netstat -luntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      561/sshd        
tcp6       0      0 :::21                   :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::22                   :::*                    LISTEN      561/sshd        
tcp6       0      0 :::1433                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::443                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::445                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::5060                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::5061                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::135                  :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::3306                 :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::42                   :::*                    LISTEN      3763/dionaea    
tcp6       0      0 :::80                   :::*                    LISTEN      3763/dionaea    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           464/dhclient3   
udp        0      0 0.0.0.0:40077           0.0.0.0:*                           3763/dionaea    
udp6       0      0 :::5060                 :::*                                3763/dionaea    
udp6       0      0 :::69                   :::*                                3763/dionaea    

 
$ sudo supervisorctl status
dionaea                          RUNNING    pid 3763, uptime 3 days, 22:31:19


$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

$ sudo tcpdump -nnNN tcp port 10000
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:46:14.009646 IP 1.2.3.4.42873 > 5.6.6.8.10000: Flags [P.], seq 1180349317:1180349611, ack 2474834734, win 913, options [nop,nop,TS val 85084174 ecr 169636000], length 294
16:46:14.012967 IP 5.6.6.8.10000 > 1.2.3.4.42873: Flags [.], ack 294, win 162, options [nop,nop,TS val 169661444 ecr 85084174], length 0

Dionaea Honeypot troubleshooting (same symptoms)

Can you log into your dionaea box and look at its config (/etc/dionaea/dionaea.conf)? Scroll to the hpfeeds section.
Your should see something like this:

hpfeeds = {
        hp1 = {
                server = "1.2.3.4"
                port = "10000"
                ident = "XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX"
                secret = "XXXXXXXXXXXXXXXX"
                // dynip_resolve: enable to lookup the sensor ip through a webservice
                dynip_resolve = "http://queryip.net/ip/"
        }

Is your "server" variable correct? It should be the IP of your MHN server. If so, can you try this (from your dionaea box)?

nc -vvv <YOUR-SERVER-IP> 10000

Do you see something like this? Or does it just hang?

$ nc -vvv <YOUR-SERVER-IP> 10000
Connection to <YOUR-SERVER-IP> 10000 port [tcp/webmin] succeeded!
@hp2?Z

If it hangs either your hpfeeds service isn't running on your server or there may be a firewall blocking it or there may be another network connectivity issue.

Troubleshooting from the MHN Server

  • I don't see events flowing back to my MHN server even though I am scanning/wget'ing my sensor

On your MHN server box, as root run these commands. You should get similar out to mine. If your output differs, this could be the problem.

# netstat -luntp | grep 10000
tcp        0      0 0.0.0.0:10000           0.0.0.0:*
LISTEN      980/python

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# cd /var/log/mhn/
# tail -f hpfeeds-broker.err
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 49187)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 39176)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 42873)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)
DEBUG:root:broker publish to dionaea.connections by ('XXX.XXX.XXX.XXX', 46710)
DEBUG:root:broker publish to geoloc.events by ('127.0.0.1', 42282)

# mongo hpfeeds
MongoDB shell version: 2.6.2
connecting to: hpfeeds
> db.auth_key.find()
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "mnemosyne",
"subscribe" : [ "conpot.events", "thug.events", "beeswarm.hive",
"dionaea.capture", "dionaea.connections", "thug.files",
"beeswarn.feeder", "cuckoo.analysis", "kippo.sessions",
"glastopf.events", "glastopf.files", "mwbinary.dionaea.sensorunique",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "honeymap",
"subscribe" : [ "geoloc.events" ], "secret" : "XXXXXXXXXXXXXX",
"publish" : [ ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "identifier" : "geoloc",
"subscribe" : [ "dionaea.connections", "dionaea.capture",
"glastopf.events", "beeswarm.hive", "kippo.sessions", "conpot.events",
"snort.alerts" ], "secret" : "XXXXXXXXXXXXXX", "publish" : [
"geoloc.events" ] }
{ "_id" : ObjectId("YYYYYYYYYYYYYYYYYY"), "subscribe" : [ ], "secret"
: "XXXXXXXXXXXXXX", "identifier" :
"XXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX-XXXXXXXX", "publish" : [
"mwbinary.dionaea.sensorunique", "dionaea.capture",
"dionaea.capture.anon", "dionaea.caputres", "dionaea.connections" ] }
Type "it" for more
>

The last entry is the entry for one of our honeypots. The identifier and secret need to match what is in your honeypot's /etc/dionaea/dionaea.conf config under the hpfeeds section. If these don't match your honeypot will not be able to talk with the server.

Running the honeypot on a private network

Because your honeypot is on a private network, you will almost certainly not see any events show up on the Honeymap since it only renders events that can be IP Geo located and private IPs do not get geolocated by Maxmind.

Can you confirm that you see no events on http://YOUR-SERVER/ui/attacks/ and no counts on the base page? If so, can you run this command? You should have output similar to mine:

# mongo mnemosyne
MongoDB shell version: 2.6.2
connecting to: mnemosyne
> db.getCollectionNames()
[
"daily_stats",
"dork",
"file",
"hpfeed",
"session",
"system.indexes",
"url"
]
> db.session.find()
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipCall", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-05-20T20:12:14.028Z"), "source_ip" :
"192.198.107.186", "source_port" : 5070, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T12:25:29.493Z"), "source_ip" :
"209.126.99.96", "source_port" : 5086, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T12:25:32.782Z"), "source_ip" : "107.22.35.136",
"source_port" : 55261, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:48.040Z"), "source_ip" : "107.22.35.136",
"source_port" : 52915, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:47.137Z"), "source_ip" : "178.252.183.100",
"source_port" : 3955, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:51.478Z"), "source_ip" : "115.79.59.61",
"source_port" : 3453, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "httpd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:50.476Z"), "source_ip" : "115.79.59.61",
"source_port" : 3454, "destination_port" : 80, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:53.105Z"), "source_ip" : "107.22.35.136",
"source_port" : 52917, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:52.274Z"), "source_ip" : "93.174.93.218",
"source_port" : 52569, "destination_port" : 21320, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.602Z"), "source_ip" : "178.252.183.100",
"source_port" : 4053, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:59.507Z"), "source_ip" : "181.193.66.61",
"source_port" : 43670, "destination_port" : 23, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:05:58.327Z"), "source_ip" :
"209.126.99.96", "source_port" : 5068, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:05:58.169Z"), "source_ip" : "107.22.35.136",
"source_port" : 52919, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.196Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.233Z"), "source_ip" : "107.22.35.136",
"source_port" : 52921, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "smbd",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:03.061Z"), "source_ip" : "178.252.183.100",
"source_port" : 4077, "destination_port" : 445, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" :
"SipSession", "hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"),
"timestamp" : ISODate("2014-06-21T13:06:01.699Z"), "source_ip" :
"176.58.113.94", "source_port" : 6072, "destination_port" : 5060,
"identifier" : "XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" :
"dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:05.730Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:04.967Z"), "source_ip" : "27.192.16.128",
"source_port" : 48078, "destination_port" : 4899, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }
{ "_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "protocol" : "pcap",
"hpfeed_id" : ObjectId("XXXXXXXXXXXXXXXXXXXXXXXXX"), "timestamp" :
ISODate("2014-06-21T13:06:08.296Z"), "source_ip" : "107.22.35.136",
"source_port" : 52923, "destination_port" : 8080, "identifier" :
"XXXXXX-XXXXXX-XXXXXXX-XXXXXXXX", "honeypot" : "dionaea" }

I have an internally deployed honeypot and even when I scan it, I see not events in MHN.

The issue is, out of the box, mnemosyne filters out attacks coming from rfc1918 addresses so, you need to configure it not to do so. Here are the steps:

As root, run these commands.

cd /opt/mnemosyne/
git fetch origin 
git stash 
git merge origin/master 
git stash pop

Now, edit mnemosyne.cfg and append this to the end of the file.

[normalizer]
ignore_rfc1918 = False

Lastly, run this command:

supervisorctl restart mnemosyne

Then run.

supervisorctl status

You should see something like this:

geoloc                           RUNNING    pid 16719, uptime 1:25:31
honeymap                         RUNNING    pid 17223, uptime 0:30:29
hpfeeds-broker                   RUNNING    pid 980, uptime 10 days, 5:19:51
mhn-celery-beat                  RUNNING    pid 961, uptime 10 days, 5:19:51
mhn-celery-worker                RUNNING    pid 14878, uptime 3:44:43
mhn-uwsgi                        RUNNING    pid 9580, uptime 9 days, 3:46:32
mnemosyne                        RUNNING    pid 17749, uptime 0:05:26

I can't reset my password because I fat fingered the email during the install

Try this:

cd /opt/mhn/server
sudo sqlite3 mhn.db 
sqlite> select * from user;   

Find the ID associated with the fat fingered email address. It will probably be 1.

sqlite> update user set email='your.real.email@gmail.com' where id=1;
sqlite> .quit

Now run the password reset script

sudo su - 
cd /opt/mhn
source env/bin/activate
python server/manual_password_reset.py

I input the wrong URL for my honeymap server during the installation

edit this file: /opt/mhn/server/config.py

Change this variable to the correct URL:

HONEYMAP_URL = 'http://10.0.0.1:3000'

Then restart mhn-uwsgi:

sudo supervisorctl restart mhn-uwsgi

MHN Server fails to start after reboot

  • Ubuntu 16.04 It's likely mongodb isn't running after reboot.
sudo systemctl start mongodb
sudo supervisorctl restart all
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.