[blocker]splunk_eventgen processes crash under multiprocess mode

[YifengMao]

Describe the bug
most of processes crashed after ~3 hrs running in eventgen multiprcess mode(20 processes)

env:
8 indexers, es bundle, 2 egx nodes, 750G perdayvolume per egx node.

2019-05-20 07:40:03 eventgen ERROR MainProcess
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/eventgen_core.py", line 335, in _worker_do_work
item = work_queue.get(timeout=10)
File "", line 2, in get
File "/usr/lib/python2.7/multiprocessing/managers.py", line 759, in _callmethod
kind, result = conn.recv()
EOFError
Note: we are not using SA-Eventgen, SA-Eventgen/lib is just a dir location for checkout eventgen 6.3.4 code.

From the image, the 2 egx instances crashed one after one at nearly the same time. After manually restarted, they crashed again after similar time period.

root 55539 16260 0 07:33 pts/1 00:02:14 python /opt/splunk/etc/apps/backend/eventgen_server.py --eventgencmd /opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen
root 55561 55539 35 07:33 pts/1 02:02:41 python /opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/main.py -v generate /opt/splunk/etc/apps/datamix/app/defa
root 55566 55561 99 07:33 pts/1 06:20:00 [python] <defunct>
root 55681 55561 27 07:33 pts/1 01:35:10 [python] <defunct>
root 55682 55561 27 07:33 pts/1 01:36:06 [python] <defunct>
root 55687 55561 27 07:33 pts/1 01:35:11 [python] <defunct>
root 55692 55561 27 07:33 pts/1 01:35:31 [python] <defunct>
root 55697 55561 27 07:33 pts/1 01:35:28 [python] <defunct>
root 55701 55561 27 07:33 pts/1 01:36:08 [python] <defunct>
root 55705 55561 27 07:33 pts/1 01:36:39 [python] <defunct>
root 55709 55561 27 07:33 pts/1 01:35:20 [python] <defunct>
root 55714 55561 27 07:33 pts/1 01:36:19 [python] <defunct>
root 55719 55561 27 07:33 pts/1 01:36:21 [python] <defunct>
root 55724 55561 27 07:33 pts/1 01:35:54 [python] <defunct>
root 55730 55561 27 07:33 pts/1 01:35:38 [python] <defunct>
root 55734 55561 27 07:33 pts/1 01:36:20 [python] <defunct>
root 55739 55561 27 07:33 pts/1 01:36:19 [python] <defunct>
root 55744 55561 27 07:33 pts/1 01:36:40 [python] <defunct>
root 55749 55561 27 07:33 pts/1 01:36:09 [python] <defunct>
root 55754 55561 27 07:33 pts/1 01:35:29 [python] <defunct>
root 55762 55561 27 07:33 pts/1 01:35:49 [python] <defunct>
root 55767 55561 27 07:33 pts/1 01:36:24 [python] <defunct>
root 55771 55561 27 07:33 pts/1 01:36:29 [python] <defunct>

[YifengMao]

1 idx, index=main, 100G per node(actual 200), 1 egx node, not crash in 5 hr
1 idx, index=main, 750G per node(actual 1.5T), 1 egx node, crash in 14 hr **
8 idx, index=8*, 750G per node(actual 1.5T), 2 egx node, both crashed in 2.5hr, reproducible
8 idx, index=8*, 750G per node(actual 1.5T), 1 egx node, crashed in 3.5 hr, reproducible

it issue need more investigation to figure out what are the main factors that cause the crash
looks like the crash is related to output module, as logger issue and multiprocess issue are fixed in develop branch, i'll try develop branch to see why the output worker crashes

[li-wu]

1 idx, index=main, 100G per node(actual 200), 1 egx node, not crash in 5 hr
1 idx, index=main, 750G per node(actual 1.5T), 1 egx node, not crash in 12 hr
8 idx, index=8*, 750G per node(actual 1.5T), 2 egx node, both crashed in 2.5hr, reproducible
8 idx, index=8*, 750G per node(actual 1.5T), 1 egx node, crashed in 3.5 hr, reproducible

it issue need more investigation to figure out what are the main factors that cause the crash
looks like the crash is related to output module, as logger issue and multiprocess issue are fixed in develop branch, i'll try develop branch to see why the output worker crashes

Thank you Yifeng for the reporting and debugging.

[YifengMao]

this crash can be reproduced in 6.3.6 latest and no additional logs found

[YifengMao]

error.log
attached logs found from verbose mode stdout. All the crashes might be related to https://stackoverflow.com/questions/3649458/broken-pipe-when-using-python-multiprocessing-managers-basemanager-syncmanager

[YifengMao]

I disabled logging queue and run CLI with multiprocess, it kept running 7 hrs and didn't crash. Keep monitoring

            self.loggingQueue = None #self.manager.Queue()
            #self.logging_pool = Thread(target=self.logger_thread, args=(self.loggingQueue, ), name="LoggerThread")
            #self.logging_pool.start()

    @staticmethod
    def _proc_worker_do_work(work_queue, logging_queue, config):
        genconfig = config
        stopping = genconfig['stopping']
        #qh = logutils.queue.QueueHandler(logging_queue)
        root = logging.getLogger()
        root.setLevel(logging.DEBUG)
        #root.addHandler(qh)

[YifengMao]

not crashed in 1 day, i'll file a commit to disable logging queue(we wont loss any useful logs) in multiprocess mode as it is possibly a python bug

[li-wu]

Here is the result of one day running