Check severity att while validating #3540

dogancanbakir · 2023-04-13T12:15:43Z

Proposed changes

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

…abot/go_modules/v2/google.golang.org/protobuf-1.29.1 chore(deps): bump google.golang.org/protobuf from 1.29.0 to 1.29.1 in /v2

nuclei v2.9.1

ShubhamRasal

lgtm

Notes:

template validate workflow is failing due to this change.

Thanks for the PR @dogancanbakir 💯

ShubhamRasal · 2023-04-13T13:50:21Z

❯ ./nuclei -t templates/test.yaml -u https://example.com -silent
[FTL] Could not run nuclei: no valid templates were found

❯ ./nuclei -w workflow/basic.yaml -u https://example.com -silent

jimen0 · 2023-04-13T15:47:53Z

@ehsandeep - is this backwards compatible? Won't this break people's templates if these don't contain a severity attribute?

dogancanbakir · 2023-04-14T08:07:11Z

@ShubhamRasal Fixed failing template: projectdiscovery/nuclei-templates#7060

ehsandeep · 2023-04-14T19:00:14Z

@ehsandeep - is this backwards compatible? Won't this break people's templates if these don't contain a severity attribute?

Initially, severity was the mandatory field, at some point for workflows, it was updated to make it optional but it looks like a change was made long back, so it makes sense to make this change backward compatible.

ehsandeep

@dogancanbakir @ShubhamRasal We can mark this condition as SyntaxErrorStats with -validate option to ensure we are not adding new templates in the template project and as SyntaxWarningStats for the default run not to break/stop loading existing templates.

- workflow loader that contains tags load all the template and parse it - i.e it iw printing warning recursively, ignore as the templates already getting valiated

Mzack9999

proposing function split

v2/pkg/parsers/parser.go

- split the function into two diff

Mzack9999

lgtm!

# runtime execution => ok with warning
$ go run . -t .\test.yaml -u 8x8exch02.8x8.com -v                                                                                          

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.1

                projectdiscovery.io

[WRN] Loaded template test.yaml: with syntax warning : field 'severity' is missing
[WRN] Found 1 templates with syntax warning (use -validate flag for further examination)
[INF] Using Nuclei Engine 2.9.1 (outdated)
[INF] Using Nuclei Templates v9.4.3 (outdated)
[INF] Templates added in last update: 0
[INF] Templates loaded for scan: 1
[INF] Targets loaded for scan: 1
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[VER] [basic-example] Sent HTTP request to https://8x8exch02.8x8.com
[INF] No results found. Better luck next time!

# validation => hard fail
$ go run . -t .\test.yaml -u 8x8exch02.8x8.com -v -validate

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.1

                projectdiscovery.io

[ERR] Error occurred loading template test.yaml: Loaded template test.yaml: with syntax warning : field 'severity' is missing
[FTL] Could not validate templates: errors occured during template validation
exit status 1

Note: just like nested variable support with delimiters ({{ a+ {{var}} }}) without a reliable recursive expression engine, I believe this inconsistent behavior will be a potential source of future problems. If severity is optional then we should simply default to undefined or a standard value

Mzack9999 · 2023-04-26T11:06:03Z

@ShubhamRasal could you please complete the following task:

Mention in the docs that optional severity policy is deprecated (optional during runtime execution but mandatory presence during template validation)

Thanks!

ShubhamRasal · 2023-04-27T09:45:59Z

@ShubhamRasal could you please complete the following task:

Mention in the docs that optional severity policy is deprecated (optional during runtime execution but mandatory presence during template validation)

Thanks!

projectdiscovery/nuclei-docs#146

* Make severity attribute required * Update test err msg * minor * Do not strict check serverity * Fix failing test * Don't print warning in workflow loader - workflow loader that contains tags load all the template and parse it - i.e it iw printing warning recursively, ignore as the templates already getting valiated * Fix error typo * Resolve comments - split the function into two diff --------- Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>

* Add utility to write max-requests to templates * fix lint error * fix max-request update edge case * fix convert max-request: 1 => max-request: 1 * WIP, most of the code is commented * Refactor the find and replace logic * Skip if template has the max-requests, do not overwrite - return errors - add warnings * Fix the wrong index calculation - Refactor the getInfoBlock function to not compile regex everytime * Update -tc flag to filter fields within the classification section (#3606) * Add fields from Classification section in a template to the -tc flag expression evaluation Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Add tests for filtering Classification section using -tc flag Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Fix hyphenated Metadata keys beings added to parameters Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Add tests to the fix for hyphenated fields encountered in Metadata section Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> --------- Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Check severity att while validating (#3540) * Make severity attribute required * Update test err msg * minor * Do not strict check serverity * Fix failing test * Don't print warning in workflow loader - workflow loader that contains tags load all the template and parse it - i.e it iw printing warning recursively, ignore as the templates already getting valiated * Fix error typo * Resolve comments - split the function into two diff --------- Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com> * tlsx dep update (#3620) * updated interactsh version (#3621) * updated interactsh version * workflow update * aws signer: fix missing x-content-sha256 header (#3601) * fix missing x-content-sha256 header * fix variable priority in self-contained templates * remove debug statement * adds generic raw request parser for self-contained req * more integration tests * bug fix: 10x faster race requests * fix failing integration test * chore(deps): bump github.com/xanzy/go-gitlab in /v2 (#3624) Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.82.0 to 0.83.0. - [Release notes](https://github.com/xanzy/go-gitlab/releases) - [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go) - [Commits](xanzy/go-gitlab@v0.82.0...v0.83.0) --- updated-dependencies: - dependency-name: github.com/xanzy/go-gitlab dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/miekg/dns from 1.1.53 to 1.1.54 in /v2 (#3625) Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.53 to 1.1.54. - [Release notes](https://github.com/miekg/dns/releases) - [Changelog](https://github.com/miekg/dns/blob/master/Makefile.release) - [Commits](miekg/dns@v1.1.53...v1.1.54) --- updated-dependencies: - dependency-name: github.com/miekg/dns dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /v2 (#3626) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.61 to 1.11.64. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.61...feature/s3/manager/v1.11.64) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix check for OS made in MustDisableSandbox() (#3631) Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Fix wrong template loading in dev branch (#3629) * Templates wrong loading * Add tests to cover following scenarios - check optional fields only if template loaded - it should return warning only if template is loaded * enable color in windows (#3634) * enable color in windows * fixed win workflow * typo update * tlsx dep update (#3633) * tlsx dep update * upgrde httpx => 1.3.0 * Fix check for OS made in MustDisableSandbox() (#3631) Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> * Fix wrong template loading in dev branch (#3629) * Templates wrong loading * Add tests to cover following scenarios - check optional fields only if template loaded - it should return warning only if template is loaded * enable color in windows (#3634) * enable color in windows * fixed win workflow * typo update --------- Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> Co-authored-by: Ramana Reddy <ramanaredy.manda@gmail.com> Co-authored-by: Suraj Kamath <kamathsuraj95@gmail.com> Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io> * Expose DNS fields for matchers and extractors (#3613) * Extend dns extractor to dns answer records * add test template * Ignore error for dns variables are not found * Add all the records of answer section * Fixed the wrong typecasting * Issue 3564 var override (#3599) * Check if the variables are override by other means - you can override the template variable value using command line flags * Update lazy eval logic - previously, we were checking any function/expression in variable - now, update the logic, lazy eval only if variable contains any protocol variable(global) * add integration tests * Add test to check the dsl function working in variable * gather all generate variables logic in utils * go mod update * Refactor the generate variables function * go mod update+ fix typo --------- Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> * update rod to v0.112.9 #3552 (#3637) * update rod to v0.112.9 * removed unused reflection --------- Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com> * change max-requests label to max-request --------- Signed-off-by: iamargus95 <kamathsuraj95@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com> Co-authored-by: Suraj Kamath <kamathsuraj95@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io> Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io> Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com> Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io> Co-authored-by: lu4nx <lx@shellcodes.org>

Mzack9999 and others added 3 commits March 17, 2023 17:51

Merge pull request projectdiscovery#3433 from projectdiscovery/depend…

f4b0625

…abot/go_modules/v2/google.golang.org/protobuf-1.29.1 chore(deps): bump google.golang.org/protobuf from 1.29.0 to 1.29.1 in /v2

Merge pull request projectdiscovery#3491 from projectdiscovery/dev

5b22ca8

nuclei v2.9.1

Make severity attribute required

ef243b7

dogancanbakir requested a review from ShubhamRasal April 13, 2023 12:15

dogancanbakir added 2 commits April 13, 2023 15:28

Update test err msg

d2eb408

minor

09b32a4

dogancanbakir requested a review from tarunKoyalwar April 13, 2023 12:45

ehsandeep linked an issue Apr 13, 2023 that may be closed by this pull request

severity not marked as required attribute #3528

Closed

ShubhamRasal assigned dogancanbakir Apr 13, 2023

ShubhamRasal approved these changes Apr 13, 2023

View reviewed changes

tarunKoyalwar requested review from ehsandeep and removed request for tarunKoyalwar April 14, 2023 13:27

ehsandeep requested changes Apr 14, 2023

View reviewed changes

ShubhamRasal added 2 commits April 19, 2023 16:15

Do not strict check serverity

910ff1f

Fix failing test

9428995

ShubhamRasal self-assigned this Apr 19, 2023

ShubhamRasal added 2 commits April 19, 2023 22:08

Don't print warning in workflow loader

5c8b373

- workflow loader that contains tags load all the template and parse it - i.e it iw printing warning recursively, ignore as the templates already getting valiated

Fix error typo

3b782e9

ShubhamRasal changed the title ~~Make severity att required~~ Check severity att while validating Apr 19, 2023

ShubhamRasal requested review from Mzack9999 and ShubhamRasal and removed request for ShubhamRasal April 19, 2023 17:12

Mzack9999 requested a review from ehsandeep April 23, 2023 19:20

Mzack9999 requested changes Apr 25, 2023

View reviewed changes

v2/pkg/parsers/parser.go Show resolved Hide resolved

ehsandeep added the Status: Revision Needed Submitter of PR needs to revise the PR related to the issue. label Apr 25, 2023

ShubhamRasal added 2 commits April 26, 2023 12:25

Merge branch 'dev' into make-severity-att-required

4e4cb51

Resolve comments

8a09241

- split the function into two diff

ehsandeep requested a review from Mzack9999 April 26, 2023 10:04

Mzack9999 approved these changes Apr 26, 2023

View reviewed changes

ehsandeep approved these changes Apr 27, 2023

View reviewed changes

ehsandeep merged commit 4e0ccb3 into projectdiscovery:dev Apr 27, 2023
9 of 11 checks passed

ehsandeep removed the request for review from ShubhamRasal April 27, 2023 09:57

ehsandeep removed the Status: Revision Needed Submitter of PR needs to revise the PR related to the issue. label Apr 27, 2023

ShubhamRasal mentioned this pull request May 1, 2023

Fix template loading and error warning #3628

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check severity att while validating #3540

Check severity att while validating #3540

dogancanbakir commented Apr 13, 2023 •

edited

Loading

ShubhamRasal left a comment

ShubhamRasal commented Apr 13, 2023

jimen0 commented Apr 13, 2023

dogancanbakir commented Apr 14, 2023

ehsandeep commented Apr 14, 2023

ehsandeep left a comment

Mzack9999 left a comment

Mzack9999 left a comment

Mzack9999 commented Apr 26, 2023

ShubhamRasal commented Apr 27, 2023

Check severity att while validating #3540

Check severity att while validating #3540

Conversation

dogancanbakir commented Apr 13, 2023 • edited Loading

Proposed changes

Checklist

ShubhamRasal left a comment

Choose a reason for hiding this comment

ShubhamRasal commented Apr 13, 2023

jimen0 commented Apr 13, 2023

dogancanbakir commented Apr 14, 2023

ehsandeep commented Apr 14, 2023

ehsandeep left a comment

Choose a reason for hiding this comment

Mzack9999 left a comment

Choose a reason for hiding this comment

Mzack9999 left a comment

Choose a reason for hiding this comment

Mzack9999 commented Apr 26, 2023

ShubhamRasal commented Apr 27, 2023

dogancanbakir commented Apr 13, 2023 •

edited

Loading