bug: Directories scan never ends on v2.0.2

[DrorDvash]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I've updated reNgine to version 2.0.2 after I saw a closed issue where ffuf results were fixed and are now shown in the dashboard UI. This issue seems to be resolved, and the results are presented. However, the scan running never ends or keeps running over and over again.

Scan type: Subdomain Discovery, Port Scan, Directory and Files Search

Current scan running for 23 hours (in v1.3.6 the same scan finished in 30-50 minutes)

When looking at the results, I noticed that each subdomain was scanned multiple times for directory fuzzing (ffuf) instead of just once.

Expected Behavior

Each domain should be scanned 1 time only with ffuf, and the scan should end correctly.

Steps To Reproduce

1. Create Custom (quick) scan engine:

subdomain_discovery: {
  'uses_tools': ['subfinder', 'ctfr', 'sublist3r', 'tlsx', 'oneforall', 'netlas'],
  'enable_http_crawl': true,
  'threads': 30,
  'timeout': 5,
}
http_crawl: {}
port_scan: {
  'enable_http_crawl': true,
  'timeout': 5,
  # 'exclude_ports': [],
  # 'exclude_subdomains': [],
  'ports': ['top-100'],
  'rate_limit': 150,
  'threads': 30,
  'passive': false,
  # 'use_naabu_config': false,
  # 'enable_nmap': true,
  # 'nmap_cmd': '',
  # 'nmap_script': '',
  # 'nmap_script_args': ''
}
dir_file_fuzz: {
  'auto_calibration': true,
  'enable_http_crawl': true,
  'rate_limit': 150,
  'extensions': ['html', 'php','git','yaml','conf','cnf','config','gz','env','log','db','mysql','bak','asp','aspx','txt','conf','sql','json','yml','pdf'],
  'follow_redirect': false,
  'max_time': 0,
  'match_http_status': [200, 204],
  'recursive_level': 2,
  'stop_on_error': false,
  'timeout': 5,
  'threads': 30,
  'wordlist_name': 'dicc'
}
screenshot: {
  'enable_http_crawl': true,
  'intensity': 'normal',
  'timeout': 10,
  'threads': 40
}

# custom_header: "Cookie: Test"

2. Start a new scan using the above engine.

Environment

- reNgine: v2.0.2
- OS: Ubuntu 22.04
- Python: Python 3.10.12
- Docker Engine: 24.0.7
- Docker Compose: v2.21.0

Anything else?

If any logs are needed, please specify which ones and provide instructions on how to extract them for you. (I used make logs, but there are numerous lines.)

Thank you.

[github-actions]

👋 Hi @DrorDvash,
Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki
For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[AnonymousWP]

Did you update httpx via tools arsenal?

[DrorDvash]

Yes i did, already updated all tools links in the Dockerfile. (e.g. amass to v4)

so everything is up-to-date.

[AnonymousWP]

Thanks, I'm trying to reproduce the issue. If you want to post some detailed logs, check out #994.

Are you running Docker Desktop on Windows perhaps? In that case you can also check logs directly in the container:

[DrorDvash]

Well, i did export DEBUG=1 and then docker-compose restart web, know i'm getting 502 Bad Gateway nginx/1.25.3..i cannot see the dashboard anymore.

In addition, i think there is something causing error in the make logs command, every time i'm running it:
error from daemon in stream: Error grabbing logs: invalid character 'l' after object key:value pair

but that's another issue not related.

[psyray]

I also noticed this problem, FFUF is relaunched again and again, but only on first task.
If I kill the parent celery process and relaunch task it runs only once
Really strange problem.

[psyray]

Well, i did export DEBUG=1 and then docker-compose restart web, know i'm getting 502 Bad Gateway nginx/1.25.3..i cannot see the dashboard anymore.

In addition, i think there is something causing error in the make logs command, every time i'm running it: error from daemon in stream: Error grabbing logs: invalid character 'l' after object key:value pair but that's another issue not related.

Weird, I'll do this a lot of time and no problem

[DrorDvash]

i have removed the export DEBUG=1 and then make down && make up, now i can see the dashboard.
i'll try again

[AnonymousWP]

Can reproduce the issue (late reply cause was busy with other things earlier), and it seems to be related to #1095 (comment). I.e. related to FFUF, cause I have similar errors in the log of the web container (see dashboard logs):

:: Progress: [211982/212036] :: Job [2/19] :: 96 req/sec :: Duration: [0:39:58] :: Errors: 24508 ::
:: Progress: [211995/212036] :: Job [2/19] :: 88 req/sec :: Duration: [0:39:58] :: Errors: 24508 ::
:: Progress: [212005/212036] :: Job [2/19] :: 88 req/sec :: Duration: [0:39:58] :: Errors: 24508 ::
:: Progress: [212018/212036] :: Job [2/19] :: 88 req/sec :: Duration: [0:39:59] :: Errors: 24508 ::
:: Progress: [212027/212036] :: Job [2/19] :: 89 req/sec :: Duration: [0:39:59] :: Errors: 24508 ::
:: Progress: [212036/212036] :: Job [2/19] :: 86 req/sec :: Duration: [0:39:59] :: Errors: 24508 ::
:: Progress: [212036/212036] :: Job [2/19] :: 82 req/sec :: Duration: [0:39:59] :: Errors: 24508 ::
[INFO] Starting queued job on target: https://web.test.com/blog/FUZZ


:: Progress: [30/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [30/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [30/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [30/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [38/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [50/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::
:: Progress: [63/212036] :: Job [3/19] :: 0 req/sec :: Duration: [0:00:00] :: Errors: 24508 ::

[psyray]

Can reproduce the issue (late reply cause was busy with other things earlier), and it seems to be related to #1095 (comment). I.e. related to FFUF, cause I have similar errors in the log of the web container (see dashboard logs):

How do you reproduce it ?

[psyray]

OK I think I've understood the problem.
FFUF command launch is inside a loop of retrieved URLs,

rengine/web/reNgine/tasks.py

Lines 1631 to 1648 in fd5a5e5

	for url in urls:
	'''
	Above while fetching urls, we are not ignoring files, because some
	default urls may redirect to https://example.com/login.php
	so, ignore_files is set to False
	but, during fuzzing, we will only need part of the path, in above example
	it is still a good idea to ffuf base url https://example.com
	so files from base url
	'''
	url_parse = urlparse(url)
	url = url_parse.scheme + '://' + url_parse.netloc
	url += '/FUZZ' # TODO: fuzz not only URL but also POST / PUT / headers
	proxy = get_random_proxy()
	# Build final cmd
	fcmd = cmd
	fcmd += f' -x {proxy}' if proxy else ''
	fcmd += f' -u {url} -json'

So I think there's a problem somewhere in the URL retrieval

rengine/web/reNgine/tasks.py

Lines 1620 to 1626 in fd5a5e5

	urls = get_http_urls(
	is_alive=True,
	ignore_files=False,
	write_filepath=input_path,
	get_only_default_urls=True,
	ctx=ctx
	)

I will try to debug.

[psyray]

Got it, problem come from here

rengine/web/reNgine/tasks.py

Lines 1683 to 1684 in fd5a5e5

	if created:
	urls.append(endpoint.http_url)

Newly created endpoint are appended to the urls var.
As urls var is the loop var, at each newly created endpoint, script add another entry in the loop.
Recursive launch of ffuf

Don't know why this is here...
@AnonymousWP @yogeshojha
Any idea ?
I think I could delete it

[AnonymousWP]

@psyray Nicely spotted, I was also thinking that there should be some infinite loop somewhere in the code due to a for-loop. Has this code always been present (I didn't bother checking)? Maybe with ocervell's PR. Anyway, I think you could delete and test locally, then see whether any errors arise and whether it fixes the issue or not.

[psyray]

@psyray Nicely spotted, I was also thinking that there should be some infinite loop somewhere in the code due to a for-loop. Has this code always been present (I didn't bother checking)? Maybe with ocervell's PR. Anyway, I think you could delete and test locally, then see whether any errors arise and whether it fixes the issue or not.

It fixes, for sure.
I have also fixed other bugs while debugging this one.
Currently testing

[DrorDvash]

I'm glad to see that you fixed the issue, and I would like to get the newest code releases + the issue fix, but I'm a little bit confused which branch should i stick with for now?
I have checked the master -> web/reNgine/tasks.py and i can see the issue you pointed to (urls.append(endpoint.http_url)) it is still here, not integrated in the master.

i have checked the 2.1.0 -> web/reNgine/tasks.py - the same.

So, which branch has the latest commits + ffuf fix? @psyray

[psyray]

#1120

Mine
https://github.com/yogeshojha/rengine/tree/fix-recursive-ffuf-launch

Do a git pull and a checkout

git pull
git checkout fix-recursive-ffuf-launch

[DrorDvash]

#1120

Mine https://github.com/yogeshojha/rengine/tree/fix-recursive-ffuf-launch

Do a git pull and a checkout

git pull
git checkout fix-recursive-ffuf-launch

yes I saw that branch but I've also seen more new commits from the very last days in the master / 2.1.0 branches, so I wanted to have the newest features / bug fixes in addition to the ffuf fix.

So there is no such branch currently?

[AnonymousWP]

You can switch to the 2.1.0 branch, as that contains all 2.1.0-related fixes.

[psyray]

You can switch to the 2.1.0 branch, as that contains all 2.1.0-related fixes.

But not this one 😁

[DrorDvash]

You can switch to the 2.1.0 branch, as that contains all 2.1.0-related fixes.

But not this one 😁

you haven't merged ffuf fix (fix-recursive-ffuf-launch) to any other branch with the latest commits?

[psyray]

You can switch to the 2.1.0 branch, as that contains all 2.1.0-related fixes.

But not this one 😁

you haven't merged ffuf fix (fix-recursive-ffuf-launch) to any other branch with the latest commits?

Nope, fix target master directly.
we can't wait release 2.1.0 to merge this one.