A better multi threaded framework!

[Ali-Razmjoo]

Hello everyone.

Right now OWASP Nettacker has two kinds of multi-threading/processing. the first one is -M switch which is the number of hosts to scan together. It's using multiprocessing module. another one is the number of threads to a host -t switch. It's using the threading module.

the problem is when the threading number is too high, and we want to count the active threads! for example, check the core/parse.py.

    for target in targets:
        for sm in scan_method:
            trying += 1
            p = multiprocessing.Process(target=start_attack, args=(
                str(target).rsplit()[0], trying, total_targets, sm, users, passwds, timeout_sec, thread_number,
                ports, log_in_file, time_sleep, language, verbose_level, show_version, check_update, socks_proxy,
                retries, ping_flag, methods_args))
            p.start()
            while 1:
                n = 0
                processes = multiprocessing.active_children()
                for process in processes:
                    if process.is_alive() is True:
                        n += 1
                    else:
                        processes.remove(process)
                if n >= thread_number_host:
                    time.sleep(0.01)
                else:
                    break

there is a for to check if the process is still alive and count it, otherwise remove the process from the array and if array length would be smaller than thread number, then add a new process.

or you can check one of the modules to see how the count with threading works.

        for port in ports:
            port = int(port)
            t = threading.Thread(target=connect,
                                 args=(target, int(port), timeout_sec, log_in_file, language, time_sleep,
                                       thread_tmp_filename, socks_proxy))
            threads.append(t)
            t.start()
            trying += 1
            if verbose_level is not 0:
                info(messages(language, 72).format(trying, total_req, num, total, target, port))
            while 1:
                try:
                    n = 0
                    for thread in threads:
                        if thread.isAlive() is True:
                            n += 1
                        else:
                            threads.remove(thread)
                    if n >= max:
                        time.sleep(0.01)
                    else:
                        break
                except KeyboardInterrupt:
                    break
                    break

it's the same!

if set the thread number in tcp_connect_port_scan as 1000. it takes more than "1 sec" to check all the dead threads and add new threads. and if we remove the time.sleep the CPU usage will go high! So we need a better counting algorithm and thread control!

Let me know if there is any question!

Regards.

[Ali-Razmjoo]

Hello,

It's better now! (almost fixed)

Regards.

[Ali-Razmjoo]

hello,

working on new upgrades...

regards.

[Ali-Razmjoo]

Hello,

there is a fundamental problem in multithreading in this framework. last night I push some codes to enable an option if the user wants to use multi-threading instead of multiprocessing.

this is how the framework was working:

in the first view, it's fast! because it's using multi-core of CPU for scanning each target with each module. in the real test, I scanned 20 IPs 192.168.1.1-192.168.1.20 in 8.458 seconds with 1000 threads and 20 processes with 1000 default ports in no ping and TCP connect mode with 2 seconds timeout. that means 20* 1000 = 20,000 requests sent in 8.458 seconds. that's powerful!! but let's do it with 1 single port. I used the same IP range and it takes 5.997 seconds (almost 6 seconds) and it's just sent 20 requests. it's because of "multiprocessing". threads are faster than opening a new process.

new codes working like this:

I've tried this 20 IPs with the single port with the code I contributed last night it took only 3.335 seconds! (used one core of CPU (2% maybe))

I've also retest with 1000 default ports it took 14.553 seconds (used 100% of one core of CPU). it's using threads instead of process. it's faster then we'd open more main threads for short times.

but there were some issues that made me revert the codes I pushed.

• framework must know when we've to use threading or multiprocessing.
• when the framework uses multiprocessing, it's using max thread number in each module, that means we used -t 1000 that means use 1000 threads for connecting to a host, but if you use port_scan and ssh_brute, it would let them create 1000 threads each!

after I pushed my codes in the master branch I released that

• codes are not working well on python 3
• codes will not be compatible if future to creating blocks in defined size, (I was using a class to keep thread numbers for each host in a dict, and then after we create new process we could not access the shared dict)

so the whole master branch was WC and I revert the codes pushed to unfuck the situation. I created a dev branch for testing fundamentals commits.

what's best we need to do?
somehow we must estimate the time and usage of targets depend on selected modules and the requests they will make, then separate them to blocks (with defined size), open each block in a new process, and each process opens the threads.

if anyone would like to help on this one, I appreciate your ideas and contributions.

Best Regards.

[Rohan-Salwan]

Hii Ali,
I would love to do work on this issue but for that I need your clear point of view you about what you are really expecting from contributor in this issue so please ellaborate it. I am member of slack nettacker group so we can continue our conversation on that platform too.

[Ali-Razmjoo]

done in #440