HTTP Requests Annotation Prototype

[Mzack9999]

Proposed changes

This PR implements a proposal for http requests to override settings via annotations. Raw requests are specified as a multiline string. Protocol settings are applied at the template level for all the subsequent requests, without the possibility to override per requests setting. Here it's proposed a mechanism inspired by a few programming languages based on annotations. If specified, it allows overriding specific request settings. An annotation looks like the following snippet:

@Setting:Value

For example, to override the real target of the request (the connection will be performed to login.microsoftonline.com:443

id: azure-domain-tenant-id

info:
  name: Microsoft Azure - Domain Tenant ID
  author: V0idC0de
  severity: info
  description: Checks if the domain is part of an Azure tenant and finds the ID using Azure's OpenID discovery page.
  tags: azure,microsoft,cloud


requests:
  - raw:
      - |
        @Host: https://login.microsoftonline.com:443
        GET /{{Host}}/v2.0/.well-known/openid-configuration HTTP/1.1
        Host: login.microsoftonline.com

    matchers:
      - type: status
        status:
          - 200

    extractors:
      - type: regex
        part: body
        group: 1
        regex:
          - "\"https:\\/\\/login\\.microsoftonline\\.com\\/([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

[ehsandeep]

@Mzack9999 we can extend this support to capture other cases, for example:

• hostname input
• input with scheme

currently, scheme is set based on the input target input url scheme, which will be ignored if it is defined in the template as shown below.

- |
  @Host: 93.184.216.34
  POST /  HTTP/1.1
  Host: {{Hostname}}

- |
  @Host: example.com
  POST /  HTTP/1.1
  Host: {{Hostname}}

- |
  @Host: https://example.com
  POST /  HTTP/1.1
  Host: {{Hostname}}

[ehsandeep]

We can also consider adding the relevant information about this feature in the guide document for HTTP protocol.

Done in projectdiscovery/nuclei-docs#32