Passive Engine Re-write for Paralellism

[Mzack9999]

• Passive engine rewrite: Ok
• General code refactor: Ok
• some missing code: Fixed (Need to be tested)

@codingo @Ice3man543

Changelog

• Introduced general pool backbone (applied to passive and resolver engines)
• Passive engine now works in parallel

[codingo]

I think it would be good to make these kinds of changes in multiple commits. There's some additional refactoring I can see here but given that it's all in one commit it's a bit tricky for me to test and track the changes. I'll start that work once this is merged though (I'd already started some of it already when adding more verbose error handling, but I'll come back to it in this branch).

The code itself looks good, is there something further you're working on before merging with this?

[Ice3man543]

@Mzack9999 as codingo noted, there is some scope for more refactoring here, changing the naming scheme, making some small changes to make it compatible with the latest commits i made. But, you have done an amazing work. Once you are finished, notify us and we will tidy up things a little bit.
👍

[Mzack9999]

I think I'm done with the big changes. Imho next PRs should be about:

• Better handling output engine (with parallelism results tend to overlap and be confusing)
• Refactor single passive sources to drop code redundancy
• Rework the helper.State structure, since it keeps state of the current elaboration for one thread only, but the program is now multi-thread

[Ice3man543]

@codingo Probably you can review it. You also wanted to do some more refactor, right?

[codingo]

I do, but I think it would be best in a separate branch so we can put a release tag on this to keep everything clean.

I'll set this up in the test bench shortly!

[codingo]

@Ice3man543 looks to satisfy testing on my end. Since it's a rather significant change suggest you do the same and then we can merge.

[Ice3man543]

Sure @codingo

[Ice3man543]

@codingo This doesn't complies with the new output style we created in ui rewrite branch. Also, there are a few things that I don't understand. Probably need to refactor some more and make this a bit more easier to understand. All in all, it's great work by @Mzack9999 👍
Though i am not in the mood to code, so won't review it yet. Probably tomorrow ...

[Mzack9999]

@Ice3man543 let me know if you need any help

[Ice3man543]

@Mzack9999 So i finally took a look at the code and ran it. The tool has became a lot more faster 😄 thanks to you. So great work 👍 Keep it up. But there are some issues with the pull request that stop me from merging it right now. It's a lot more faster though :).

• Aquatone Style JSON output is not working
• Output doesn't matches at some places with new standard.
• Resolver logic has been messed up. Resolver should return *helper.Job structures as output so that we can output them to file. But in passive-engine rewrite branch it returns []string.

If all these mistakes can be fixed by you, then awesome. Otherwise, i am going out of town for a few days. After i return, i can fix these and merge the branch.
@codingo What do you say?

[Mzack9999]

I replaced helper.Job with a more generic helper.Domain. I'll try to correct in next days the various outputs

[Mzack9999]

@Ice3man543 @codingo hopefully I merged correctly the output styles from master, I just left a TODO related to output to directory which I've no time to address right now, but should be a fast fix

[Ice3man543]

@Mzack9999 I am working on fixing some things. Will fix everything i can see and then merge :)

[Ice3man543]

@Mzack9999 The aquatone style output is still messed up. I am working now on cleaning it. Will be done by morning i guess if i leave everything and work on this! Do you think merging it to the master is correct? Also, have you done any benchmarks? I will do them :)

[Ice3man543]

Oh god, i merged it by mistake :(
@codingo Can you test the new build ?

[Ice3man543]

@Mzack9999 The aquatone -nW flag output doesn't works as expected. Check it out.

[Ice3man543]

@Mzack9999 @codingo No worries now. I made some chages so that it works fine. Now, everything is working as supposed and even faster due to Worker pools and create a more modular architecture :)
Thanks a lot @Mzack9999 for this. It is really appreciated. Tomorrow morning, i'll add worker pools to bruteforcer and fix some bugs, making our 1.2 release. Also, we got 200 stars. Congrats on that 👍