-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parser multiprocessing pool is not repopulated #9713
Comments
The pool seems to be re-populated without any code changes:
But still it's not working. |
Working on this in this branch |
The process pool seems to be maintained without any major issues, but:
First run#export W3AF_CPU_PROFILING=1
export W3AF_MEMORY_PROFILING=1
export W3AF_CORE_PROFILING=1
export W3AF_THREAD_ACTIVITY=1
export W3AF_PROCESSES=1
export W3AF_PSUTILS=1
export W3AF_PYTRACEMALLOC=1 Instances which run this test:
Second run (memory profiling disabled)#export W3AF_CPU_PROFILING=1
#export W3AF_MEMORY_PROFILING=1
export W3AF_CORE_PROFILING=1
export W3AF_THREAD_ACTIVITY=1
export W3AF_PROCESSES=1
export W3AF_PSUTILS=1
#export W3AF_PYTRACEMALLOC=1 Instances which run this test:
Expected results
|
AnalysisMemory profiling enabled
Memory profiling disabled
ResultMemory profiling breaks multiprocessing+parser_cache |
* Added message to remind me of #9713
This is still an issue since I can't run scans to profile very important information such as:
This is how we count the number of parser cache timeouts (10 seconds): I believe I should run tests with the following configurations:
|
Increasing |
Potential reason documented in https://docs.python.org/2/library/multiprocessing.html:
And another one:
|
https://github.com/noxdafox/pebble is a potential replacement for mp_document_parser implementation , these guys might be handling some of the errors explained above correctly with a lock: https://github.com/noxdafox/pebble/blob/master/pebble/pool/process.py#L360-L366 |
… multiprocessing, queue usage without lock, etc. as described in #9713
Note on 772f769: The test does break the multiprocessing parser once every 5 runs, on a rainy day, on my computer, and while I'm playing hard-core music 🗡️ . I found no way to reproduce this in 100% of the cases, since it depends on the parent process killing the worker in a specific point in time when the worker is writing to the queue. |
re: performance, the new implementation from 27c6e25 seems to run the unittests faster: 35sec vs. 45sec. |
Running a lot of tests to make sure this change works, but it looks promising! Fixing the build https://circleci.com/gh/andresriancho/w3af/tree/feature%2Fpebble-mp and making sure there are no performance issues |
This was fixed with pebble |
TODO
Issue
After one 60 second timeout, all I see is a repeating series of:
With different URLs.
The text was updated successfully, but these errors were encountered: