Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug] An error occurs when running nuclei with multiple instances #2301

Closed
hktalent opened this issue Jul 18, 2022 · 15 comments
Closed

[bug] An error occurs when running nuclei with multiple instances #2301

hktalent opened this issue Jul 18, 2022 · 15 comments
Assignees
@Mzack9999
Labels
Investigation Something to Investigate Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. Type: Question A query or seeking clarification on parts of the spec. Probably doesn't need the attention of all.

Comments

@hktalent
Copy link
Contributor

hktalent commented Jul 18, 2022 •
edited

fatal error: concurrent map iteration and map write

test code:

package main

import (
	"bytes"
	"fmt"
	"github.com/hktalent/scan4all/nuclei_Yaml"
	"net/http"
	_ "net/http/pprof"
	"os"
	"sync"
)


func DoNuclei(buf *bytes.Buffer, wg *sync.WaitGroup, oOpts *map[string]interface{}) {
	defer wg.Done()
	xx := make(chan bool)
	go nuclei_Yaml.RunNuclei(buf, xx, nil)
	<-xx
}

func main() {
	if true {
		go func() {
			http.ListenAndServe(":6060", nil)
		}()
		buf := bytes.Buffer{}
		var wg sync.WaitGroup
		wg.Add(1)
		buf.WriteString("http://192.168.10.31:8888\n")
		pwd, _ := os.Getwd()
		m1 := map[string]interface{}{"UpdateTemplates": false, "Templates": []string{pwd + "/config/nuclei-templates"}, "TemplatesDirectory": pwd + "/config/nuclei-templates", "NoUpdateTemplates": true}
		go DoNuclei(&buf, &wg, &m1)
		wg.Add(1)
		go DoNuclei(&buf, &wg, &m1)
		wg.Add(1)
		go DoNuclei(&buf, &wg, &m1)
		wg.Wait()
	}
}

error:

[0:00:25] | Templates: 3661 | Hosts: 1 | RPS: 102 | Matched: 19 | Errors: 92 | Requests: 2574/5043 (51%)
fatal error: concurrent map iteration and map write

goroutine 36114 [running]:
runtime.throw({0x52c27f3?, 0x1?})
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/panic.go:992 +0x71 fp=0xc010f3f288 sp=0xc010f3f258 pc=0x4038b51
runtime.mapiternext(0x5031840?)
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/map.go:871 +0x4eb fp=0xc010f3f2f8 sp=0xc010f3f288 pc=0x4013aeb
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeRequest(0xc00ebe1680, {0xc0007cac60, 0x19}, 0xc01385f540, 0xc008d9d680?, 0x0, 0xc008d9d6b0, 0x0?)
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:573 +0x28fb fp=0xc010f3ff38 sp=0xc010f3f2f8 pc=0x4d0f59b
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP.func1(0xc008412b40?)
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:210 +0x9f fp=0xc010f3ffc8 sp=0xc010f3ff38 pc=0x4d0bbdf
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP.func2()
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:216 +0x2a fp=0xc010f3ffe0 sp=0xc010f3ffc8 pc=0x4d0bb0a
runtime.goexit()
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc010f3ffe8 sp=0xc010f3ffe0 pc=0x406b861
created by github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:207 +0x377

goroutine 1 [chan receive]:
main.DoNuclei(0xc00093bd40, 0xc000869e30?, 0x5280fd1?)
        /Users/51pwn/MyWork/scan4all/test/nuclei/testNuclei.go:29 +0xca
main.main()
        /Users/51pwn/MyWork/scan4all/test/nuclei/testNuclei.go:45 +0x2c5

log.txt

options

{
  "Tags": [],
  "ExcludeTags": [],
  "Workflows": [],
  "WorkflowURLs": [],
  "Templates": [
    "/Users/51pwn/MyWork/scan4all/config/nuclei-templates"
  ],
  "TemplateURLs": [],
  "RemoteTemplateDomainList": [
    "api.nuclei.sh"
  ],
  "ExcludedTemplates": [],
  "ExcludeMatchers": null,
  "CustomHeaders": [],
  "Vars": {},
  "Severities": null,
  "ExcludeSeverities": null,
  "Authors": [],
  "Protocols": [
    1,
    2,
    3,
    4,
    5,
    6,
    7,
    8,
    9
  ],
  "ExcludeProtocols": null,
  "IncludeTags": [],
  "IncludeTemplates": [],
  "IncludeIds": [],
  "ExcludeIds": [],
  "InternalResolversList": null,
  "ProjectPath": "/var/folders/_l/pnb2t_9s0f192bqlz1348vpr0000gn/T/",
  "InteractshURL": "",
  "InteractshToken": "",
  "Targets": [
    "http://192.168.10.31:8888"
  ],
  "TargetsFilePath": "",
  "Resume": "",
  "Output": "",
  "ProxyInternal": false,
  "Proxy": [],
  "TemplatesDirectory": "/Users/51pwn/MyWork/scan4all/config/nuclei-templates",
  "TraceLogFile": "",
  "ErrorLogFile": "",
  "ReportingDB": "",
  "ReportingConfig": "",
  "MarkdownExportDirectory": "",
  "SarifExport": "",
  "ResolversFile": "",
  "StatsInterval": 5,
  "MetricsPort": 9092,
  "MaxHostError": 30,
  "BulkSize": 64,
  "TemplateThreads": 64,
  "HeadlessBulkSize": 10,
  "HeadlessTemplateThreads": 10,
  "Timeout": 5,
  "Retries": 1,
  "RateLimit": 150,
  "RateLimitMinute": 0,
  "PageTimeout": 20,
  "InteractionsCacheSize": 5000,
  "InteractionsPollDuration": 5,
  "InteractionsEviction": 60,
  "InteractionsCoolDownPeriod": 5,
  "MaxRedirects": 10,
  "FollowRedirects": false,
  "OfflineHTTP": false,
  "StatsJSON": false,
  "Headless": false,
  "ShowBrowser": false,
  "UseInstalledChrome": false,
  "SystemResolvers": false,
  "Metrics": false,
  "Debug": false,
  "DebugRequests": false,
  "DebugResponse": false,
  "LeaveDefaultPorts": false,
  "AutomaticScan": false,
  "Silent": false,
  "Version": false,
  "Validate": false,
  "NoStrictSyntax": false,
  "Verbose": false,
  "VerboseVerbose": false,
  "NoColor": false,
  "UpdateTemplates": false,
  "JSON": false,
  "JSONRequests": false,
  "EnableProgressBar": true,
  "TemplatesVersion": false,
  "TemplateList": false,
  "HangMonitor": false,
  "Stdin": false,
  "StopAtFirstMatch": false,
  "Stream": false,
  "NoMeta": false,
  "NoTimestamp": false,
  "Project": false,
  "NewTemplates": false,
  "NewTemplatesWithVersion": null,
  "NoInteractsh": false,
  "UpdateNuclei": false,
  "NoUpdateTemplates": true,
  "EnvironmentVariables": false,
  "MatcherStatus": false,
  "ClientCertFile": "",
  "ClientKeyFile": "",
  "ClientCAFile": "",
  "ZTLS": false,
  "ShowMatchLine": false,
  "EnablePprof": false,
  "StoreResponse": false,
  "StoreResponseDir": "output",
  "DisableRedirects": true,
  "SNI": "",
  "HealthCheck": false,
  "InputReadTimeout": 0,
  "DisableStdin": false
}
@hktalent hktalent added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Jul 18, 2022
@hktalent
Copy link
Contributor Author

image

@hktalent
Copy link
Contributor Author

image

@hktalent
Copy link
Contributor Author

hktalent commented Jul 18, 2022 •
edited

@ehsandeep
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go
The analysis found:
previousEvent
finalEvent
There are thread safety bugs
Finally tracked here:
type InternalEvent map[string]interface{}

The same bug exists in v2.7.4

fatal error: concurrent map writes

goroutine 21282 [running]:
runtime.throw({0x529fe07?, 0x40f2905?})
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/panic.go:992 +0x71 fp=0xc00776d290 sp=0xc00776d260 pc=0x403a471
runtime.mapassign_faststr(0x5261772?, 0x5?, {0xc00f19c510, 0xe})
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/map_faststr.go:212 +0x39c fp=0xc00776d2f8 sp=0xc00776d290 pc=0x40187fc
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeRequest(0xc0095b2b60, {0xc0087c07a0, 0x19}, 0xc01253ef50, 0xc0088ba1b0?, 0x0, 0xc0088ba1e0, 0x0?)
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:592 +0x2aef fp=0xc00776df38 sp=0xc00776d2f8 pc=0x4d1b94f
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP.func1(0xc0096665a0?)
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:210 +0x9f fp=0xc00776dfc8 sp=0xc00776df38 pc=0x4d1791f
github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP.func2()
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:216 +0x2a fp=0xc00776dfe0 sp=0xc00776dfc8 pc=0x4d1784a
runtime.goexit()
        /usr/local/Cellar/go/1.18.4/libexec/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00776dfe8 sp=0xc00776dfe0 pc=0x406d181
created by github.com/projectdiscovery/nuclei/v2/pkg/protocols/http.(*Request).executeTurboHTTP
        /Users/51pwn/MyWork/scan4all/vendor/github.com/projectdiscovery/nuclei/v2/pkg/protocols/http/request.go:207 +0x377

goroutine 1 [chan receive]:
main.DoNuclei(0xc005cfc9c0, 0xc005cfc9f0?, 0xc000010f00)
        /Users/51pwn/MyWork/scan4all/test/nuclei/testNuclei.go:29 +0xe5
main.main()
        /Users/51pwn/MyWork/scan4all/test/nuclei/testNuclei.go:45 +0x277

@hktalent
Copy link
Contributor Author

@ehsandeep @forgedhallpass

Can you provide an http interface to submit asynchronous tasks?
1. Start the http receive task
2. The task contains url + template tag + option parameter, for example, you can specify url template data (yaml, zip)
post http://127.0.0.1/nuclei/v1/api/addTask

Parameter example:
url=http://127.0.0.1/test
templatesIds=[]strings{}
tags=[]strings{}
templates=[]strings{"./config/xx1.yaml","http://127.0.0.1/pocs/xx1.yaml","http://127.0.0.1/pocs/all.zip"}

@rijalrojan
Copy link

same err,how fix, thanks?

@l0n3rs
Copy link

l0n3rs commented Jul 20, 2022

Oh，I also encountered the same problem. Can you solve it as soon as possible.

@hktalent
Copy link
Contributor Author

@ehsandeep
bug2

go DoNuclei(&buf, &wg, &m1)
		wg.Add(1)
		go DoNuclei(&buf, &wg, &m1)
		wg.Add(1)
		go DoNuclei(&buf, &wg, &m1)

console log, you can see > 100%:

[0:01:00] | Templates: 3680 | Hosts: 1 | RPS: 89 | Matched: 26 | Errors: 186 | Requests: 5380/5094 (105%)
[0:01:05] | Templates: 3680 | Hosts: 1 | RPS: 58 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:05] | Templates: 3680 | Hosts: 1 | RPS: 75 | Matched: 23 | Errors: 136 | Requests: 4905/5094 (96%)
[0:01:05] | Templates: 3680 | Hosts: 1 | RPS: 82 | Matched: 26 | Errors: 189 | Requests: 5383/5094 (105%)
[0:01:10] | Templates: 3680 | Hosts: 1 | RPS: 54 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:10] | Templates: 3680 | Hosts: 1 | RPS: 70 | Matched: 23 | Errors: 136 | Requests: 4905/5094 (96%)
[0:01:10] | Templates: 3680 | Hosts: 1 | RPS: 76 | Matched: 26 | Errors: 192 | Requests: 5386/5094 (105%)
[0:01:12] | Templates: 3680 | Hosts: 1 | RPS: 67 | Matched: 23 | Errors: 136 | Requests: 4905/5094 (96%)
[0:01:15] | Templates: 3680 | Hosts: 1 | RPS: 50 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:15] | Templates: 3680 | Hosts: 1 | RPS: 71 | Matched: 26 | Errors: 194 | Requests: 5388/5094 (105%)
[0:01:20] | Templates: 3680 | Hosts: 1 | RPS: 47 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:20] | Templates: 3680 | Hosts: 1 | RPS: 67 | Matched: 26 | Errors: 196 | Requests: 5390/5094 (105%)
[0:01:25] | Templates: 3680 | Hosts: 1 | RPS: 44 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:25] | Templates: 3680 | Hosts: 1 | RPS: 63 | Matched: 26 | Errors: 196 | Requests: 5390/5094 (105%)
[0:01:25] | Templates: 3680 | Hosts: 1 | RPS: 44 | Matched: 17 | Errors: 174 | Requests: 3784/5094 (74%)
[0:01:25] | Templates: 3680 | Hosts: 1 | RPS: 63 | Matched: 26 | Errors: 196 | Requests: 5390/5094 (105%)

@hktalent
Copy link
Contributor Author

@ehsandeep
request.go

+ var someMapMutex = sync.RWMutex{}
// executeRequest executes the actual generated request and returns error if occurred
func (request *Request) executeRequest(reqURL string, generatedRequest *generatedRequest, previousEvent output.InternalEvent, hasInteractMatchers bool, callback protocols.OutputEventCallback, requestCount int) error {
	request.setCustomHeaders(generatedRequest)
.....
+  someMapMutex.Lock()
		for k, v := range previousEvent {
			finalEvent[k] = v
		}
		for k, v := range outputEvent {
			finalEvent[k] = v
		}

		// Add to history the current request number metadata if asked by the user.
		if request.ReqCondition {
			for k, v := range outputEvent {
				key := fmt.Sprintf("%s_%d", k, requestCount)
				previousEvent[key] = v
				finalEvent[key] = v
			}
		}
+ someMapMutex.Unlock()

now fix request.go:573

@hktalent hktalent mentioned this issue Jul 20, 2022
Closed
@ehsandeep
Copy link
Member

@hktalent running multiple instances of nuclei at the same from the same system is not expected in general, but we will look into the crash.

@hktalent
Copy link
Contributor Author

@ehsandeep
image
I would like to be able to support concurrent nuclei multi-instance, which is very valuable, in the case of multi-tasking, in the case of dynamically joining the target

thanks

@hktalent
Copy link
Contributor Author

@ehsandeep
It is recommended not to use internal. At present, I have encountered such a situation. I need to call nuclei runner.Close() externally to terminate the current task.
Then I found that because of the use of internal, it violated the rules of golang
code example
(x1.(*runner2.Runner)).Close()

@hktalent
Copy link
Contributor Author

@ehsandeep After each update package, it needs to be repaired manually, otherwise memory errors and abnormal exits will occur. Get rid of whether you can merge my PR #2308

request.go

+ var someMapMutex = sync.RWMutex{}
// executeRequest executes the actual generated request and returns error if occurred
func (request *Request) executeRequest(reqURL string, generatedRequest *generatedRequest, previousEvent output.InternalEvent, hasInteractMatchers bool, callback protocols.OutputEventCallback, requestCount int) error {
	request.setCustomHeaders(generatedRequest)
.....
+  someMapMutex.Lock()
		for k, v := range previousEvent {
			finalEvent[k] = v
		}
		for k, v := range outputEvent {
			finalEvent[k] = v
		}

		// Add to history the current request number metadata if asked by the user.
		if request.ReqCondition {
			for k, v := range outputEvent {
				key := fmt.Sprintf("%s_%d", k, requestCount)
				previousEvent[key] = v
				finalEvent[key] = v
			}
		}
+ someMapMutex.Unlock()

now fix request.go:573

@hktalent hktalent mentioned this issue Jul 28, 2022
Merged
4 tasks
@forgedhallpass forgedhallpass linked a pull request Jul 29, 2022 that will close this issue
Add substr and aes_cbc DSL functions #2361
Merged
4 tasks
@hktalent
Copy link
Contributor Author

hktalent commented Jul 30, 2022 •
edited

@ehsandeep
Concurrent multi-instance, a large number of：
“leveldb: closed”
0xProject/0x-mesh#319
image

@ehsandeep ehsandeep added the Investigation Something to Investigate label Jul 30, 2022
@Mzack9999 Mzack9999 added the Status: On Hold Similar to blocked, but is assigned to someone label Aug 1, 2022
@ehsandeep ehsandeep removed a link to a pull request Aug 22, 2022
Add substr and aes_cbc DSL functions #2361
Merged
4 tasks
ehsandeep pushed a commit that referenced this issue Aug 25, 2022
@hktalent @x51pwn @Mzack9999
Add substr and aes_cbc DSL functions (#2361)
606c361 
* 1、add DSL substr for #2304 By @hktalent
substr('xxtestxxx',2)。   testxxx
substr('xxtestxxx',2,-2)  testx
substr('xxtestxxx',2,6)   test

2、add DSL aes_cbc for #2243 By @hktalent
aes_cbc("key111key111key111key111", "dataxxxxxxdataxxxxxxdataxxxxxxdataxxxxxxdataxxxxxx")

3、fixed An error occurs when running nuclei with multiple instances #2301 By @hktalent

* refactoring helpers

* removing unwanted mutex

* commenting out test

* removing aes_cbc test due to random iv

Co-authored-by: 51pwn <51pwn@51pwn.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
@Mzack9999 Mzack9999 removed the Status: On Hold Similar to blocked, but is assigned to someone label Dec 28, 2022
@Mzack9999 Mzack9999 self-assigned this Jan 2, 2023
@Mzack9999 Mzack9999 added the Type: Question A query or seeking clarification on parts of the spec. Probably doesn't need the attention of all. label Jan 2, 2023
@Mzack9999
Copy link
Member

@hktalent It seems like that https://github.com/hktalent/scan4all/nuclei_Yaml doesn't exist anymore.
Is this issue still reproducible? Can you provide me a link to where the nuclei runner is being used?

@jimen0
Copy link
Contributor

jimen0 commented Jan 2, 2023

@Mzack9999 maybe https://github.com/hktalent/scan4all/tree/main/projectdiscovery/nuclei_Yaml is the new path for it?

@hktalent hktalent closed this as completed Jan 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Mzack9999 Mzack9999

Labels
Investigation Something to Investigate Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. Type: Question A query or seeking clarification on parts of the spec. Probably doesn't need the attention of all.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@jimen0 @ehsandeep @Mzack9999 @hktalent @l0n3rs @rijalrojan