No Data in Kibana or Moloch after upload

[chrswtsn]

I followed the installation guide for Ubuntu 18.04 LTS, I used the git method for grabbing the install files. Everything installed and populated exactly as stated in the guide. When I attempt to upload a pcap through https://localhost:8443 the pcap is accepted. I have validated this by checking the pcap/upload directory and I see it move over to the pcap/processed directory. The mime type for my pcap shows "application/vnd.tcpdump.pcap". I also ran "docker-compose ps" and everything is "up" with elastalert, kibana, elasticsearch, and logstash showing "(healthy)". I have tried running the wipe.sh script and doing reboots with no change. The pcap does not show in the history or sessions tab of moloch even when specifying "all" as a time range. I'm not really sure what to do at this point.

[mmguero]

A couple things we could due to debug:

1. Enable PCAP_PIPELINE_DEBUG by setting it to true in docker-compose.yml
2. stop/wipe/restart Malcolm
3. wait till everything is started up
4. upload your PCAP file
5. after a while, run docker-compose logs > debug.txt and then paste the contents of the docker-compose logs here

[chrswtsn]

debug.txt
I looked over the output of the debug.txt but I don't see any thing that I can tie back to PCAP not loading.

[mmguero]

There is some some weird stuff going on right off the bat, not quite sure what to think of it:

moloch_1         | Elasticsearch ERROR: 2020-03-26T18:04:57Z
moloch_1         |   Error: Request error, retrying
moloch_1         |   POST http://elasticsearch:9200/hunts/hunt/_search?rest_total_hits_as_int=true => read ECONNRESET
moloch_1         |       at Log.error (/data/moloch/node_modules/elasticsearch/src/lib/log.js:226:56)
moloch_1         |       at checkRespForFailure (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:259:18)
moloch_1         |       at HttpConnector.<anonymous> (/data/moloch/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
moloch_1         |       at ClientRequest.wrapper (/data/moloch/node_modules/lodash/lodash.js:4929:19)
moloch_1         |       at ClientRequest.emit (events.js:198:13)
moloch_1         |       at Socket.socketErrorListener (_http_client.js:392:9)

...

moloch_1         |       at Socket.emit (events.js:198:13)
moloch_1         |       at emitErrorNT (internal/streams/destroy.js:91:8)
moloch_1         |       at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
moloch_1         |       at process._tickCallback (internal/process/next_tick.js:63:19)
moloch_1         |
moloch_1         | Elasticsearch WARNING: 2020-03-26T18:04:57Z
moloch_1         |   Unable to revive connection: http://elasticsearch:9200/
moloch_1         |
moloch_1         | Elasticsearch WARNING: 2020-03-26T18:04:57Z
moloch_1         |   No living connections
moloch_1         |
moloch_1         | Error fetching hunt jobs { Error: No Living connections
moloch_1         |     at sendReqWithConnection (/data/moloch/node_modules/elasticsearch/src/lib/transport.js:226:15)

...

pcap-monitor_1   | pcap_watcher.py:	waiting for Elasticsearch to be healthy
pcap-monitor_1   | Traceback (most recent call last):
pcap-monitor_1   |   File "/usr/local/lib/python3.7/dist-packages/urllib3/connection.py", line 157, in _new_conn
pcap-monitor_1   |     (self._dns_host, self.port), self.timeout, **extra_kw
pcap-monitor_1   |   File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 84, in create_connection
pcap-monitor_1   |     raise err
pcap-monitor_1   |   File "/usr/local/lib/python3.7/dist-packages/urllib3/util/connection.py", line 74, in create_connection
pcap-monitor_1   |     sock.connect(sa)
pcap-monitor_1   | ConnectionRefusedError: [Errno 111] Connection refused

and all sorts of other errors about "no living connections" soon thereafter.

But here's the other thing that's weird, a little bit later:

zeek_1           | Traceback (most recent call last):
zeek_1           |   File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
zeek_1           |     self.run()
zeek_1           |   File "/usr/lib/python3.7/threading.py", line 865, in run
zeek_1           |     self._target(*self._args, **self._kwargs)
zeek_1           |   File "/usr/lib/python3.7/multiprocessing/pool.py", line 105, in worker
zeek_1           |     initializer(*initargs)
zeek_1           |   File "/usr/local/bin/pcap_zeek_processor.py", line 213, in zeekFileWorker
zeek_1           |     shutil.copy(tgzFileName, uploadDir)
zeek_1           |   File "/usr/lib/python3.7/shutil.py", line 245, in copy
zeek_1           |     copyfile(src, dst, follow_symlinks=follow_symlinks)
zeek_1           |   File "/usr/lib/python3.7/shutil.py", line 121, in copyfile
zeek_1           |     with open(dst, 'wb') as fdst:
zeek_1           | PermissionError: [Errno 13] Permission denied: '/zeek/upload/AUTOZEEK,AUTOCARVEall,test.pcap-test-1585246210516659.tar.gz'

So I see a couple of things that I haven't seen before:

1. Elasticsearch stops accepting connections pretty quick after starting. Not sure why this would be. Perhaps we're running into either memory limitations (how much memory have you allocated to elasticsearch?) or into problems with available file/socket handles (run ulimit -a, what do you have for open files?).

2. Directory permissions? what are the ownership and permissions settings for zeek and zeek/upload?

Anyway, these seem to be both related to the issues you're having. Something specific to your system configuration or platform, as I've never seen these particular errors in the distros I've tested Malcolm on.

[chrswtsn]

The VM is running 16GB RAM.

output of ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63755
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63755
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

zeek

drwxrwxr-x  3 root root 4096 Jan 10 10:33 .
drwxr-xr-x 27 root root 4096 Mar 17 00:24 ..
drwxrwxr-x  2 root root 4096 Jan 10 10:33 config
-rw-rw-r--  1 root root 1131 Jan 10 10:33 supervisord.conf

zeek-logs/upload

drwxrwxr-x  6 root root 4096 Jan 10 10:33 .
drwxr-xr-x 27 root root 4096 Mar 17 00:24 ..
drwxrwxr-x  2 root root 4096 Jan 10 10:33 current
drwxrwxr-x  4 root root 4096 Jan 10 10:33 extract_files
drwxrwxr-x  2 root root 4096 Jan 10 10:33 processed
drwxrwxr-x  2 root root 4096 Jan 10 10:33 upload

[mmguero]

the open files (-n) 1024 is going to be an issue problem. You can look at the documentation in the section Operating system configuration. It talks about some operating system tweaks you should have made prior to running Malcolm.

If you run install.py as the instructions outline, it should have prompted you to make those systems settings changes.

The other thing that's weird is that everything's owned by root. Normally everything can run as a regular user just fine.

If it were me, I would to this:

1. completely blow away your entire malcolm directory
2. do the walkthrough again, this time don't do it as root but as your regular user.
3. pay specific attention to the questions in the install.py part of that, including those for increasing file handle limits, etc.
4. reboot and try again

[chrswtsn]

I went back through my OS configurations all were present as specified in the Operating system configuration. I was able to resolve the problem after a rebuild. I did not create a non root user account prior to running 'install.py'. The malcolm directory was built under the non-existing users path i.e. /home/user/Malcolm. I am now seeing data, thank you so much for all the help.

[mmguero]

Super, I'm glad. Have a good day, stay safe.