introduce exclude target flag #4214

dogancanbakir · 2023-10-03T09:49:07Z

Closes #3877. Based on #3946.

$ cat list.txt 
1.1.1.1
status.example.com
1.1.1.1/24
*.gov
admin*.gov

$ go run . -l list.txt -eh 1.1.1.1/24

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.15

                projectdiscovery.io

[INF] Number of hosts excluded from input: 256
[INF] Supplied input was automatically deduplicated (1 removed).
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6892
[INF] Targets loaded for current scan: 3
[INF] Running httpx on input host
^C[INF] CTRL+C pressed: Exiting
[INF] Creating resume file: /home/vscode/.config/nuclei/resume-cken09o54n9so99pq4fg.cfg
exit status 1

mnafees · 2023-10-04T07:18:31Z

Hi @ehsandeep @tarunKoyalwar, we would like to use this feature and I can volunteer to review this PR, if that is something that's possible.

ehsandeep

Display counter in single line instead of list of each

[INF] Number of hosts excluded from input list: 110

Notes to add in docs
- Exclude target process input per line as it is.
- Exclude target does not resolve hostname to exclude ip.

ehsandeep

echo 1.1.1.1/24 | ./nuclei -id ptr-fingerprint -eh exclude.txt 

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.15

		projectdiscovery.io

[INF] Number of hosts excluded from input: 10
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 1
[INF] Targets loaded for current scan: 246
[INF] No results found. Better luck next time!

tarunKoyalwar

current implementation of CIDR / IP lgtm ! proposing following changes instead of doing a exact match on target domains/urls
ex:

this works

$ echo "hackerone.com" | ./nuclei -eh hackerone.com                                             1 ↵

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.15

		projectdiscovery.io

[INF] Number of hosts excluded from input: 1
[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] No results found. Better luck next time!

this does not

$ echo "https://hackerone.com" | ./nuclei -eh hackerone.com

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.15

		projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] Targets loaded for current scan: 2
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 1195 (Reduced 2268 Requests)
[dnssec-detection] [dns] [info] hackerone.com
[mx-service-detector:Google Apps] [dns] [info] hackerone.com

$ echo "hackerone.com" | ./nuclei -eh https://hackerone.com

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.15

		projectdiscovery.io

[WRN] Found 1 templates loaded with deprecated protocol syntax, update before v3 for continued support.
[INF] Current nuclei version: v2.9.15 (latest)
[INF] Current nuclei-templates version: v9.6.4 (latest)
[INF] New templates added in latest release: 121
[INF] Templates loaded for current scan: 6895
[INF] Targets loaded for current scan: 2
[INF] Running httpx on input host
[INF] Found 1 URL from httpx
[INF] Templates clustered: 1195 (Reduced 2268 Requests)
[nameserver-fingerprint] [dns] [info] hackerone.com [a.ns.hackerone.com.,b.ns.hackerone.com.]

proposed changes / discussion

perform exclusion of hosts after uncover output ( i think this will be needed in near future)
since the flag is -eh exclude hosts . exclusion of targets should be done based on Host value i.e (hostname:port) and not urls . current implementation only supports exact match
i think we should have different methods for remove and add instead of having a common processValue since we might get a enhancement issue soon asking for wildcard exclusion support . if we have different methods there won't be any need of refactor and wildcard functionality would be easy to implement in future
Incorrect target count during exclusion, check this does not section , it counts exclusion host as a target

[INF] Targets loaded for current scan: 2

what do you think ? cc: @dogancanbakir @ehsandeep

ehsandeep

Merge conflict + comments from @tarunKoyalwar

dogancanbakir · 2023-11-08T07:36:39Z

FYI, will review, maybe refactor the code and then proceed with adding the new feature.

tarunKoyalwar

implementation lgtm ! suggested a change regarding potential duplicated code

$ echo "hackerone.com" | ./nuclei -eh hackerone.com             

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.3

		projectdiscovery.io

[INF] Number of hosts excluded from input: 1
[INF] Current nuclei version: v3.0.3 (outdated)
[INF] Current nuclei-templates version: v9.6.9 (latest)
[INF] New templates added in latest release: 73
[INF] Templates loaded for current scan: 7278
[INF] Executing 5264 signed templates from projectdiscovery/nuclei-templates
[WRN] Executing 2028 unsigned templates. Use with caution.
[INF] No results found. Better luck next time!

More Changes

we have a exported/public method in Input struct i.Set(item) which when used bypasses exclution hosts list hence Suggesting addition of new method

func (i*Input) SetWithExclusions(target string)

^ alternative method to i.Set(target) exported method which does not add target to store if given target is already blacklisted/excluded

we use i.Set(item) in runner and we can replace it with i.SetWithExclusions(item)

pkg/core/inputs/hybrid/hmap.go

ehsandeep

echo "https://hackerone.com" | ./nuclei -eh hackerone.com

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v3.0.3

		projectdiscovery.io

panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).delItem.func1({0x1400005c360, 0x22, 0x8?}, {0x14000d694d7?, 0x1?, 0x1?})
	/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:441 +0xec
github.com/projectdiscovery/hmap/store/disk.(*LevelDB).Scan(0x14000a2cea8?, {{0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0, 0x14000ea64f8})
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.24/store/disk/leveldb.go:224 +0x1e0
github.com/projectdiscovery/hmap/store/hybrid.(*HybridMap).Scan(0x14000b2bed0?, 0xd?)
	/Users/geekboy/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.24/store/hybrid/hybrid.go:258 +0xe4
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).delItem(0x1400068f500, 0xd?)
	/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:425 +0xe4
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).Del(0x1400068f500, {0x14000b2bed0, 0xd})
	/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:393 +0x5dc
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.(*Input).initializeInputSources(0x0?, 0x14000d69cf8)
	/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:181 +0x410
github.com/projectdiscovery/nuclei/v3/pkg/core/inputs/hybrid.New(0x14000d69cf8)
	/Users/geekboy/Github/nuclei/pkg/core/inputs/hybrid/hmap.go:91 +0x33c
github.com/projectdiscovery/nuclei/v3/internal/runner.New(0x104936d80)
	/Users/geekboy/Github/nuclei/internal/runner/runner.go:218 +0x820
main.main()
	/Users/geekboy/Github/nuclei/cmd/nuclei/main.go:119 +0x340

ehsandeep

@dogancanbakir we can create follow up ticket for regex support to cover cases like wildcard based exclusion.

dogancanbakir · 2023-11-23T11:02:07Z

For regex support, #4412

tarunKoyalwar

LGTM !

introduce exclude target flag

eb2fd50

dogancanbakir requested review from ehsandeep and tarunKoyalwar October 3, 2023 09:49

dogancanbakir linked an issue Oct 3, 2023 that may be closed by this pull request

option to exclude hosts to scan from input list #3877

Closed

ehsandeep requested changes Oct 4, 2023

View reviewed changes

dogancanbakir added 2 commits October 4, 2023 11:40

change logging

32922f5

update docs

c0c0772

dogancanbakir requested a review from ehsandeep October 4, 2023 11:59

ehsandeep and others added 2 commits October 4, 2023 18:44

misc option update

a3b79e1

update input count

d15eee0

ehsandeep approved these changes Oct 4, 2023

View reviewed changes

dogancanbakir requested a review from Mzack9999 October 9, 2023 11:38

tarunKoyalwar requested changes Oct 9, 2023

View reviewed changes

tarunKoyalwar assigned dogancanbakir Oct 9, 2023

ehsandeep mentioned this pull request Oct 11, 2023

Exclude Hosts From Input Lists #3946

Closed

4 tasks

ehsandeep requested changes Nov 5, 2023

View reviewed changes

Merge branch 'dev' into introduce_exclude_target_flag

907b674

dogancanbakir added 2 commits November 20, 2023 19:15

separate funcs

aef3e44

exclude with host-based comparison

439da26

dogancanbakir requested review from ehsandeep and tarunKoyalwar November 21, 2023 10:27

dogancanbakir added 2 commits November 21, 2023 10:32

fix test

4f6aa5a

fix lint

7eafd04

tarunKoyalwar reviewed Nov 21, 2023

View reviewed changes

pkg/core/inputs/hybrid/hmap.go Outdated Show resolved Hide resolved

dogancanbakir added 2 commits November 22, 2023 12:22

remove duplicate

a5d9b6f

introduce SetWithExclusions

8225a0b

dogancanbakir requested a review from tarunKoyalwar November 22, 2023 12:49

ehsandeep reviewed Nov 23, 2023

View reviewed changes

init map /shrug

f87bea5

dogancanbakir requested a review from ehsandeep November 23, 2023 10:25

ehsandeep approved these changes Nov 23, 2023

View reviewed changes

tarunKoyalwar approved these changes Nov 23, 2023

View reviewed changes

ehsandeep merged commit 7cb03f2 into dev Nov 23, 2023
12 checks passed

ehsandeep deleted the introduce_exclude_target_flag branch November 23, 2023 18:35

BrewTestBot mentioned this pull request Nov 30, 2023

nuclei 3.1.0 Homebrew/homebrew-core#156031

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce exclude target flag #4214

introduce exclude target flag #4214

dogancanbakir commented Oct 3, 2023 •

edited

mnafees commented Oct 4, 2023

ehsandeep left a comment •

edited by dogancanbakir

ehsandeep left a comment

tarunKoyalwar left a comment •

edited by dogancanbakir

ehsandeep left a comment •

edited

dogancanbakir commented Nov 8, 2023

tarunKoyalwar left a comment

ehsandeep left a comment

ehsandeep left a comment

dogancanbakir commented Nov 23, 2023

tarunKoyalwar left a comment

introduce exclude target flag #4214

introduce exclude target flag #4214

Conversation

dogancanbakir commented Oct 3, 2023 • edited

mnafees commented Oct 4, 2023

ehsandeep left a comment • edited by dogancanbakir

Choose a reason for hiding this comment

ehsandeep left a comment

Choose a reason for hiding this comment

tarunKoyalwar left a comment • edited by dogancanbakir

Choose a reason for hiding this comment

this works

this does not

proposed changes / discussion

ehsandeep left a comment • edited

Choose a reason for hiding this comment

dogancanbakir commented Nov 8, 2023

tarunKoyalwar left a comment

Choose a reason for hiding this comment

More Changes

ehsandeep left a comment

Choose a reason for hiding this comment

ehsandeep left a comment

Choose a reason for hiding this comment

dogancanbakir commented Nov 23, 2023

tarunKoyalwar left a comment

Choose a reason for hiding this comment

dogancanbakir commented Oct 3, 2023 •

edited

ehsandeep left a comment •

edited by dogancanbakir

tarunKoyalwar left a comment •

edited by dogancanbakir

ehsandeep left a comment •

edited