Pyshark fails when run with Flask - tshark_process = existing_process or self.eventloop.run_until_complete(self._get_tshark_process())

[stadelmannj]

Hi,
running Pyshark under Flask on Linux (with Python 3.6.6) always fails, even with the most simple capture reading and for loop.
it only works if cap = pyshark.FileCapture(file) is run under the main program (before flask; which does not work in my case since I need to re-read the file while being in Flask)
the same code works fine on Python 3.7 for Windows.

error msg:

root@ubuntu01:/opt/callmon# python3 test.py
 * Serving Flask app "test" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://192.168.2.157:5000/ (Press CTRL+C to quit)
[2018-10-20 05:56:07,129] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "test.py", line 10, in mainpage
    for i in cap:
  File "/usr/local/lib/python3.6/dist-packages/pyshark/capture/capture.py", line 207, in _packets_from_tshark_sync
    tshark_process = existing_process or self.eventloop.run_until_complete(self._get_tshark_process())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/dist-packages/pyshark/capture/capture.py", line 369, in _get_tshark_process
    stdin=stdin)
  File "/usr/lib/python3.6/asyncio/subprocess.py", line 225, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1194, in subprocess_exec
    bufsize, **kwargs)
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 203, in _make_subprocess_transport
    self._child_watcher_callback, transp)
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 867, in add_child_handler
    "Cannot add child handler, "
RuntimeError: Cannot add child handler, the child watcher does not have a loop attached
192.168.2.128 - - [20/Oct/2018 05:56:07] "GET / HTTP/1.1" 500 -
^CTask was destroyed but it is pending!
task: <Task pending coro=<BaseSubprocessTransport._connect_pipes() running at /usr/lib/python3.6/asyncio/base_subprocess.py:177> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f23acf25678>()]>>
Future exception was never retrieved
future: <Future finished exception=RuntimeError('Event loop is closed',)>
Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1102, in connect_read_pipe
    yield from waiter
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/asyncio/base_subprocess.py", line 177, in _connect_pipes
    proc.stdout)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 1104, in connect_read_pipe
    transport.close()
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 423, in close
    self._close(None)
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 452, in _close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/usr/lib/python3.6/asyncio/base_events.py", line 575, in call_soon
    self._check_closed()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 358, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
root@ubuntu01:/opt/callmon#

i reduced the code to the minimum to prove it always fails, even w/o complex display-filter for extensive for-loops.
Code :

from flask import Flask, render_template, redirect, url_for, request
import pyshark, datetime, os, sys, time, subprocess

app = Flask(__name__, static_url_path='/static')


@app.route('/')
def mainpage():
  cap = pyshark.FileCapture(filedate)
  for i in cap:
     print (i.sip.method)
  return ('Hello World')


# Main
if __name__ == '__main__':
  filedate = ('/opt/callmon/pcap/2018-10-17.pcap')
  app.run(host="192.168.2.157",port=5000)

any help would be appreciated since we really need to run this on Linux and not Windows

[h3nr1-g]

Hey stadelmannj,
I ran into the same issue a few days ago. I "solved" it by downgrading to pyshark 0.3.8.

[ssdemajia]

It is because python 3.7 only main thread can have EvenLoop, but flask request is solved in worker thread, so if you want to use pyshark in flask, you must work in main thread. I solved this by using python 2.7 and pyshark 0.3.3

[mgoffin]

Happened to run across this myself in another project that uses threads. Tried Python 3.6, 3.7, downgrading to pyshark 0.3.8, none of that helped.

[solidossnakos]

I was having random errors related to cap = pyshark.FileCapture(file).
What worked for me is using:
cap.close()
PS: I'm using cap = pyshark.FileCapture(file) in multiple threads(not main)

[KimiNewt]

@mgoffin @solidossnakos - Can you post code where that happens?
Can you also try adding (after you initialised a cap object) this line: asyncio.get_child_watcher().attach_loop(cap.eventloop)

[mgoffin]

I can't share the code short of the instantiation of the object as it's a private code base at the moment. But to describe it...we have a core process that spawns multiple long-running threads and in one of those threads is where we instantiate the FileCapture object like so:

try:
    cap = FileCapture(
        data,
        display_filter=display_filter,
        decryption_key=decryption_key,
        encryption_type=encryption_type,
    )
    asyncio.get_child_watcher().attach_loop(cap.eventloop)
except Exception as e:
    print(e)

This triggers the exception and prints set_wakeup_fd only works in main thread

[conor-f]

I get the same issue as @mgoffin when using a LiveCapture, including the set_wakeup_fd only works in main thread. Any solutions?

[nikitawootten]

@conor-f

I'm still getting this issue now, but I found a workaround if your setup allows it.
I managed to get live capture working by running it in another process (from multiprocessing.Process) and sending the results back through a multiprocessing queue.

[marten-seemann]

Is there any update on this issue?

[SaiMun92]

I am having the same error. Are there any fixes?

[yiboliu31]

Hi, @SadeghKrmi
I met the same issue in Django project when I want to parse a pcap file and render it to the html.
But your solution didn't work for me.
May I know your python/Django/pyshark version?

[muthvar1]

I too am facing this issue in my code after moving to python3. The same code works in python2.7. None of the suggested workarounds or fixes seem to work for me. Looks like pyshark really dropped the ball on this one. How can you not have you module built to be thread safe?