Workers do not pick items from worklist #527

Closed
viyatb opened this Issue Dec 8, 2015 · 7 comments

Comments

Projects
None yet
3 participants
@viyatb
Member

viyatb commented Dec 8, 2015

@7a : The workers do not pick up the work in the work list on first run. I reported this before, still happening, this is pretty bad because you could be creating more work in the middle of a scan and the workers should be able to pick up new work automatically

I think Control + C on the terminal will work, trying <-- that works, but still, users should not need to do that, we should pick up new work from the worklist as soon as it's added, this is a workaround, not a fix.

Update: Another odd behaviour: the workers take the worklist items if you try to remove the "stuck" worker and add/remove a new worker

@tunnelshade ?

@tunnelshade

This comment has been minimized.

Show comment
Hide comment
@tunnelshade

tunnelshade Dec 8, 2015

Member

The process that happens here

  • When a new work is added it is stored in the worklist table
  • The FileServer actually polls for free workers and if there are any then get the work and give it

The problem might be that some worker might be struct on some work. Then this is difficult to solve. Can you reproduce and try @delta24

Member

tunnelshade commented Dec 8, 2015

The process that happens here

  • When a new work is added it is stored in the worklist table
  • The FileServer actually polls for free workers and if there are any then get the work and give it

The problem might be that some worker might be struct on some work. Then this is difficult to solve. Can you reproduce and try @delta24

@viyatb

This comment has been minimized.

Show comment
Hide comment
@viyatb

viyatb Dec 8, 2015

Member

@tunnelshade, to reproduce: Add a target and select say 2 groups of plugins: external, passive. I observed no direct activity on the console, and the Worker-1 was idle.

Also I think that this was the error showed on the log file when this happened (In fact everytime the plugins get stuck or workers do not take their assigned work, this error happens):

[ERROR] [2015-12-08 10:29:34,011] [File 'pool.py', line 572, in _finalize_fairy] - Exception during reset or similar
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 565, in _finalize_fairy
    fairy._reset(pool, echo)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 699, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 406, in do_rollback
    dbapi_connection.rollback()
DatabaseError: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

Probably similar error, dropbox/nsot#40
Log file file_server.log

Member

viyatb commented Dec 8, 2015

@tunnelshade, to reproduce: Add a target and select say 2 groups of plugins: external, passive. I observed no direct activity on the console, and the Worker-1 was idle.

Also I think that this was the error showed on the log file when this happened (In fact everytime the plugins get stuck or workers do not take their assigned work, this error happens):

[ERROR] [2015-12-08 10:29:34,011] [File 'pool.py', line 572, in _finalize_fairy] - Exception during reset or similar
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 565, in _finalize_fairy
    fairy._reset(pool, echo)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/pool.py", line 699, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 406, in do_rollback
    dbapi_connection.rollback()
DatabaseError: server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

Probably similar error, dropbox/nsot#40
Log file file_server.log

@tunnelshade

This comment has been minimized.

Show comment
Hide comment
@tunnelshade

tunnelshade Dec 8, 2015

Member

Please do the following things for debugging

  • Check if the periodic callback is happening (this actually polls the workers)
  • Check if the workers are visible as idle to this callback (Workers are different process and this callback happens in a different process)
  • Check if worklist is returning work

The error here is actually due to a failed db commit. Might be while saving work?

Member

tunnelshade commented Dec 8, 2015

Please do the following things for debugging

  • Check if the periodic callback is happening (this actually polls the workers)
  • Check if the workers are visible as idle to this callback (Workers are different process and this callback happens in a different process)
  • Check if worklist is returning work

The error here is actually due to a failed db commit. Might be while saving work?

@prvnkumark

This comment has been minimized.

Show comment
Hide comment
@prvnkumark

prvnkumark Jan 1, 2016

Hi delta24,

I am using lattest version, still facing same issue. I have to press ctr-c to get worklist loaded into workers.

Hi delta24,

I am using lattest version, still facing same issue. I have to press ctr-c to get worklist loaded into workers.

@viyatb

This comment has been minimized.

Show comment
Hide comment
@viyatb

viyatb Jan 1, 2016

Member

Yes, I am still looking into it. I have a couple of options, and I'll fix this in a week. :)

Member

viyatb commented Jan 1, 2016

Yes, I am still looking into it. I have a couple of options, and I'll fix this in a week. :)

@prvnkumark

This comment has been minimized.

Show comment
Hide comment
@prvnkumark

prvnkumark Jan 1, 2016

Thanks bro :)

Thanks bro :)

@viyatb viyatb added this to the OWTF Quality Release milestone Jan 18, 2016

@viyatb viyatb self-assigned this Jan 18, 2016

@viyatb

This comment has been minimized.

Show comment
Hide comment
@viyatb

viyatb Jan 20, 2016

Member

@tunnelshade I turned on the debug=True flag in the tornado servers and I got:

Process InterfaceServer-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/viyat/workspace/owtf/framework/lib/owtf_process.py", line 53, in run
    self.pseudo_run()
  File "/home/viyat/workspace/owtf/framework/interface/server.py", line 38, in pseudo_run
    self.server.start(0)
  File "/usr/lib/python2.7/site-packages/tornado/tcpserver.py", line 199, in start
    process.fork_processes(num_processes)
  File "/usr/lib/python2.7/site-packages/tornado/process.py", line 121, in fork_processes
    raise RuntimeError("Cannot run in multiple processes: IOLoop instance "
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()

Could this be causing the issue?

Member

viyatb commented Jan 20, 2016

@tunnelshade I turned on the debug=True flag in the tornado servers and I got:

Process InterfaceServer-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/viyat/workspace/owtf/framework/lib/owtf_process.py", line 53, in run
    self.pseudo_run()
  File "/home/viyat/workspace/owtf/framework/interface/server.py", line 38, in pseudo_run
    self.server.start(0)
  File "/usr/lib/python2.7/site-packages/tornado/tcpserver.py", line 199, in start
    process.fork_processes(num_processes)
  File "/usr/lib/python2.7/site-packages/tornado/process.py", line 121, in fork_processes
    raise RuntimeError("Cannot run in multiple processes: IOLoop instance "
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()

Could this be causing the issue?

@viyatb viyatb closed this in 243a1e7 Jan 30, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment