HybridTechDetection with wappalyzer and tech templates.

[xxcdd]

Proposed changes

implement hybrid tech detection

This commit introduces the implementation of hybrid technology detection in the automaticscan package. The hybrid detection utilizes both Wappalyzer and Nuclei templates to identify technologies used by a target.

Keep Wappalyzer-based technology detection using fingerprints.
Add tech tag templates for technology detection.

Closes #4520
Based on #4562

Test

nuclei  -u http://testphp.vulnweb.com/ -duc -as



                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.5

                projectdiscovery.io


[WRN] Found 12 template[s] loaded with deprecated paths, update before v3 for continued support.
[WRN] Found 1 templates with runtime error (use -validate flag for further examination)
[INF] Current nuclei version: v3.1.5 (outdated)
[INF] Current nuclei-templates version: v9.7.2 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 7380
[INF] Executing 7398 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[INF] Loaded 410 cluster templates from the tech tag.
[INF] Executing wappalyzer based tech detection on input urls
[waf-detect:nginxgeneric] [http] [info] http://testphp.vulnweb.com/
[INF] Executing Wappalyzer based tech detection get (php, dreamweaver, nginx, ubuntu) for host &{http://testphp.vulnweb.com/  }
[INF] Executing Template based tech detection get (tech, php, dreamweaver, nginx, waf, misc, nginxgeneric) for host &{http://testphp.vulnweb.com/  }
[INF] Executing tags (php, dreamweaver, nginx, ubuntu, misc, nginxgeneric) for host &{http://testphp.vulnweb.com/  } (85 templates)
[nginx-version] [http] [info] http://testphp.vulnweb.com/ ["nginx/1.19.0"]
[clientaccesspolicy] [http] [info] http://testphp.vulnweb.com/clientaccesspolicy.xml
[INF] Using Interactsh Server: oast.live

[olearycrew]

Thanks for this contribution @xxcdd !

[Mzack9999]

supersedes #4562

[tarunKoyalwar]

lgtm !

 ./nuclei -u https://www.hackerone.com -as  -c 1000  

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.1.5

		projectdiscovery.io

[INF] Current nuclei version: v3.1.5 (outdated)
[INF] Current nuclei-templates version: v9.7.5 (latest)
[WRN] Scan results upload to cloud is disabled.
[INF] New templates added in latest release: 106
[INF] Templates loaded for current scan: 7463
[INF] Executing 7481 signed templates from projectdiscovery/nuclei-templates
[INF] Targets loaded for current scan: 1
[INF] Automatic scan tech-detect: Templates clustered: 250 (Reduced 237 Requests)
[INF] Executing Automatic scan on 1 target[s]
[drupal-detect] [http] [info] https://www.hackerone.com
[waf-detect:ats] [http] [info] https://www.hackerone.com/
[waf-detect:cloudflare] [http] [info] https://www.hackerone.com/
[INF] Found 13 tags and 2 matches on detection templates on https://www.hackerone.com [wappalyzer: 12, detection: 4]
[INF] Executing 112 templates on https://www.hackerone.com
[drupal-detect] [http] [info] https://www.hackerone.com
[drupal-login] [http] [info] https://www.hackerone.com
[INF] Using Interactsh Server: oast.pro