Extending YAML to support include preprocessing

[Mzack9999]

Proposed changes

This PR implements the YAML include directive (# !include:path/to/file.yaml). The should be .yaml (static blocks)

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Examples

requests.yaml

- |
  GET /1 HTTP/1.1
  Host: {{Hostname}}
  Accept: */*

- |
  GET /2 HTTP/1.1
  Host: {{Hostname}}
  Accept: */*

template.yaml

id: CVE-2014-3206

info:
  name: test
  author: test
  severity: info
  description: test
  tags: info

requests:
  - raw:
    # !include:requests.yaml
    unsafe: true

Run with:

echo http://192.168.1.1 | go run . -t template.yaml -v -vv -debug

[ehsandeep]

@Mzack9999 not sure if I'm doing something wrong, but I don't see any outgoing requests using shared example.

$ bat requests.yaml 
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: requests.yaml
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ - |
   2   │   GET /1 HTTP/1.1
   3   │   Host: {{Hostname}}
   4   │   Accept: */*
   5   │ 
   6   │ - |
   7   │   GET /2 HTTP/1.1
   8   │   Host: {{Hostname}}
   9   │   Accept: */*
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
$ bat template.yaml 
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: template.yaml
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ id: CVE-2014-3206
   2   │ 
   3   │ info:
   4   │   name: test
   5   │   author: test
   6   │   severity: info
   7   │   description: test
   8   │   tags: info
   9   │ 
  10   │ requests:
  11   │   - raw:
  12   │     # !include:requests.yaml
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

$ echo https://example.com | go run . -t template.yaml -v

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.8.1

		projectdiscovery.io

[INF] Using Nuclei Engine 2.8.1 (outdated)
[INF] Using Nuclei Templates 9.3.1 (latest)
[INF] Templates added in last update: 2
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] No results found. Better luck next time!

updated code with latest changes