bug fix in url path and adds integration tests

[tarunKoyalwar]

Proposed changes

• fix extra slash in unsafe request
• add integration test for raw request edgecases
• closes Raw Request Bug [Slash is added with UnSafe=true] #3326
• closes Changed Encoding Handling since v2.8.8 Breaks Nuclei-Templates #3329
  • fixed with remove .Prepare() method  retryablehttp-go#57

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[tarunKoyalwar]

template used in integration test

nuclei/integration_tests/http/raw-unsafe-path.yaml

Lines 19 to 50 in 35dee8b

	# Test all unsafe URL Handling Edgecases
	requests:
	- raw:
	# relative path without leading slash
	- |+
	GET 1337 HTTP/1.1
	Host: scanme.sh
	# same but with param
	- |+
	GET 1337?with=param HTTP/1.1
	Host: scanme.sh
	# url encoded characters in path
	- |+
	GET /some%0A/%0D HTTP/1.1
	Host: scanme.sh
	# test unsupported chars in path
	- |+
	GET /%20test%0a HTTP/1.1
	Host: scanme.sh
	# test payload integrity params
	- |+
	GET /text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d HTTP/1.1
	Host: scanme.sh
	# test for missing trailing slash
	- |+
	GET /test/..;/..;/ HTTP/1.1
	Host: scanme.sh
	Origin: {{BaseURL}}
	unsafe: true
	matchers:

[ehsandeep]

./nuclei -t test.yaml -u http://google.com -debug-req

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.9

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.9 (latest)
[INF] Using Nuclei Templates 9.3.7 (latest)
[INF] Templates added in last update: 58
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] [C] Dumped HTTP request for http://google.com/1337

GET 1337 HTTP/1.1
Host: google.com

[INF] No results found. Better luck next time!

[tarunKoyalwar]

@ehsandeep @Mzack9999 , all url encoding edgecases supported and their behaviour are documented in integration test yaml file along with some explaination at

testcases for unsafe raw requests

nuclei/integration_tests/http/raw-unsafe-path.yaml

Lines 1 to 58 in 9905447

	id: raw-unsafe-path
	info:
	name: Test RAW Unsafe Paths
	author: pd-team
	severity: info
	description: >
	- https://github.com/projectdiscovery/nuclei/pull/3211
	- https://github.com/projectdiscovery/nuclei/pull/3127
	reference:
	# adding expected results here for context and debugging
	- "1337"
	- "1337?with=param"
	- "/some%0A/%0D"
	- "/%20test%0a"
	- "/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d"
	- "/test/..;/..;/"
	- "/xyz/%u2s/%invalid"
	# Test all unsafe URL Handling Edgecases
	requests:
	- raw:
	# relative path without leading slash
	- |+
	GET 1337 HTTP/1.1
	Host: scanme.sh
	# same but with param
	- |+
	GET 1337?with=param HTTP/1.1
	Host: scanme.sh
	# url encoded characters in path
	- |+
	GET /some%0A/%0D HTTP/1.1
	Host: scanme.sh
	# test unsupported chars in path
	- |+
	GET /%20test%0a HTTP/1.1
	Host: scanme.sh
	# test payload integrity params
	- |+
	GET /text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d HTTP/1.1
	Host: scanme.sh
	# test for missing trailing slash
	- |+
	GET /test/..;/..;/ HTTP/1.1
	Host: scanme.sh
	Origin: {{BaseURL}}
	# test relative path with invalid/corrupted characters
	- |+
	GET /xyz/%u2s/%invalid HTTP/1.1
	Host: scanme.sh
	unsafe: true
	matchers:
	- type: status
	status:
	- 200

testcases for other/regular requests

nuclei/integration_tests/http/http-paths.yaml

Lines 1 to 84 in 9905447

	id: raw-unsafe-path
	info:
	name: Test RAW Unsafe Paths
	author: pd-team
	severity: info
	description: >
	- https://github.com/projectdiscovery/nuclei/pull/3211
	- https://github.com/projectdiscovery/nuclei/pull/3127
	reference:
	# adding expected results here for context and debugging
	- "/1337?with=param"
	- "/some%0A/%0D"
	- "/%73%6f%6d%65%0A/%0D"
	- "/%00test%20"
	- "/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d"
	- "/test/..;/..;/"
	- "/xyz/%25u2s/%25invalid"
	# duplicating here because same results are expected even if http request is written in different format
	- "/1337?with=param"
	- "/some%0A/%0D"
	- "/%73%6f%6d%65%0A/%0D"
	- "/%00test%20"
	- "/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d"
	- "/test/..;/..;/"
	- "/xyz/%25u2s/%25invalid"
	# Test all templates with FullURLs
	requests:
	- raw:
	# relative path without leading slash with param
	# If relative path does not have `/` prefix it is autocorrected
	- |+
	GET 1337?with=param HTTP/1.1
	Host: scanme.sh
	# url encoded characters in path
	- |+
	GET /some%0A/%0D HTTP/1.1
	Host: scanme.sh
	# percent encoded characters in path
	# In URL encoding only key characters are encoded
	# while in percent encoding all characters are url encoded (similar to burp decoder)
	- |+
	GET /%73%6f%6d%65%0A/%0D HTTP/1.1
	Host: scanme.sh
	# test null and % chars in path
	- |+
	GET /%00test%20 HTTP/1.1
	Host: scanme.sh
	# test payload integrity in parameter
	- |+
	GET /text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d HTTP/1.1
	Host: scanme.sh
	# test for missing trailing slash
	- |+
	GET /test/..;/..;/ HTTP/1.1
	Host: scanme.sh
	Origin: {{BaseURL}}
	# test relative path with invalid/corrupted characters
	# In such case instead of error or panic nuclei escaped unsupported character (i.e /xyz/%25u2s/%25invalid)
	# if template requires this condition to not escape unsupported characters. It can only be done in unsafe raw requests
	- |+
	GET /xyz/%u2s/%invalid HTTP/1.1
	Host: scanme.sh
	matchers:
	- type: status
	status:
	- 200
	# Same testcases as mentioned above but in path based request format
	- method: GET
	path:
	- "{{BaseURL}}/1337?with=param"
	- "{{BaseURL}}/some%0A/%0D"
	- "{{BaseURL}}/%73%6f%6d%65%0A/%0D"
	- "{{BaseURL}}/%00test%20"
	- "{{BaseURL}}/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{}.getparam')%7d"
	- "{{BaseURL}}/test/..;/..;/"
	- "{{BaseURL}}/xyz/%u2s/%invalid"
	matchers:
	- type: status
	status:
	- 200