fix make http request inconsistencies

[tarunKoyalwar]

Proposed changes

• Most of the Make HTTP protocol request logic was centred towards {{BaseURL}} (automerge params etc) which caused inconsistencies when templates used {{RootURL}} or any other variable in Path section of the template.
  • This was also cause behind Path handling issues with {{RootURL}} variables #3242
  • New logic implemented in this PR does not depend on variables (for automerge etc) and is implemented after variables are evaluated
• Minor improvements to debug.sh for debugging integration tests
• Other Code refactor and Improvements in Make (http protocol)
  • Improved trailing slash logic (uses a proper regex to match) + added unit test
  • removes unnecessary url parsing

closes #3242

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[tarunKoyalwar]

Example Run

 $ go run . -id CVE-2022-35405 -v -u scanme.sh      

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.8

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.8 (latest)
[INF] Using Nuclei Templates 9.3.6 (latest)
[INF] Templates added in last update: 30
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[VER] [CVE-2022-35405] Sent HTTP request to https://scanme.sh/xmlrpc
[WRN] [CVE-2022-35405] Could not execute request for scanme.sh: POST https://scanme.sh:8282/xmlrpc giving up after 2 attempts: Post "https://scanme.sh:8282/xmlrpc": dial tcp [2400:6180:0:d0::91:1001]:8282: connect: connection refused
[WRN] [CVE-2022-35405] Could not execute request for scanme.sh: POST https://scanme.sh:9292/xmlrpc giving up after 2 attempts: Post "https://scanme.sh:9292/xmlrpc": dial tcp [2400:6180:0:d0::91:1001]:9292: connect: connection refused
[WRN] [CVE-2022-35405] Could not execute request for scanme.sh: POST https://scanme.sh:7272/xmlrpc giving up after 2 attempts: Post "https://scanme.sh:7272/xmlrpc": dial tcp [2400:6180:0:d0::91:1001]:7272: connect: connection refused
[INF] No results found. Better luck next time!

[tarunKoyalwar]

Things to Note from current state

• If (target/ given input) has parameters they will be appended to all http requests with corresponding order(first target params then template params)

For Raw Requests

• If given input has path ex: scanme.sh/wordpress then this path /wordpress will be automerged in raw requests ex: if raw request had path GET /xyz HTTP/1.1 then it will be converted to GET /wordpress/xyz HTTP/1.1 (along with any parameters if any)
• Using {{BaseURL}} {{RootURL}} in path of raw request Ex: GET /{{BaseURL}}/wp is discouraged/not supported due to above point

For Non-Raw Requests (Specifically path portion of template)

• target url port is only updated if path matches regex^{{(BaseURL|RootURL)}}:(\d+))
• During expression evaluation variable map are created from various sources ( payload in template(ex fuzz) , target url , vars passed from CLI ) . when conflict occurs preference order is (target url < vars passed from CLI < payloads in template (ex: fuzz))

[tarunKoyalwar]

echo "scanme.sh/?admin=true" | go run . -id "CVE-2018-16670" -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.8

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.8 (latest)
[INF] Using Nuclei Templates 9.3.6 (latest)
[INF] Templates added in last update: 30
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[VER] [CVE-2018-16670] Sent HTTP request to https://scanme.sh/services/user/values.xml?admin=true&var=STATUS
[INF] No results found. Better luck next time!

[Mzack9999]

some minor changes

[Mzack9999]

• Implementation: lgtm
• Flagging potential issue related to process exit code to verify