fix data-race warnings #4036

RamanaReddy0M · 2023-08-09T16:34:07Z

Proposed changes

closes

WARNING: DATA RACE #4016

Data race at: github.com/projectdiscovery/utils/conn/connpool.(*OneTimePool).Close()

[INF] Using Interactsh Server: oast.pro
==================
WARNING: DATA RACE
Write at 0x00c0528bb3e8 by goroutine 3883:
  github.com/projectdiscovery/utils/conn/connpool.(*OneTimePool).Close()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.45-0.20230725161322-28ec1ee0ba40/conn/connpool/onetimepool.go:79 +0x4c
  github.com/projectdiscovery/utils/crypto/jarm.HashWithDialer.func1()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.45-0.20230725161322-28ec1ee0ba40/crypto/jarm/jarm.go:33 +0x2c
  runtime.deferreturn()
      /usr/local/go/src/runtime/panic.go:476 +0x30
  github.com/projectdiscovery/dsl.init.0.func84()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/dsl@v0.0.16/dsl.go:1121 +0xd0
  github.com/projectdiscovery/dsl.dslFunction.Exec()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/dsl@v0.0.16/func.go:52 +0x1e0
  github.com/projectdiscovery/dsl.dslFunction.Exec-fm()
      <autogenerated>:1 +0x88
  github.com/Knetic/govaluate.makeFunctionStage.func1()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/evaluationStage.go:245 +0x140
  github.com/Knetic/govaluate.EvaluableExpression.evaluateStage()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:232 +0x678
  github.com/Knetic/govaluate.EvaluableExpression.evaluateStage()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:172 +0xec
  github.com/Knetic/govaluate.EvaluableExpression.Eval()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:163 +0x148
  github.com/Knetic/govaluate.EvaluableExpression.Evaluate()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:137 +0xb4
  github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers.(*Matcher).MatchDSL()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/operators/matchers/match.go:191 +0x1e4
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).Match()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/operators.go:32 +0xfc
  github.com/projectdiscovery/nuclei/v2/pkg/protocols.Request.Match-fm()
      <autogenerated>:1 +0x58
  github.com/projectdiscovery/nuclei/v2/pkg/operators.(*Operators).Execute()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/operators/operators.go:271 +0x69c
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator.CreateEventWithAdditionalOptions()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/common/helpers/eventcreator/eventcreator.go:28 +0x1e0
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeRequestWithPayloads()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:300 +0x2870
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeAddress()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:108 +0x664
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).ExecuteWithResults()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:71 +0x52c
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/common/executer/executer.go:93 +0x708
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets.func2.1()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:128 +0x40c
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets.func2.2()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:135 +0x60

Previous read at 0x00c0528bb3e8 by goroutine 13972:
  github.com/projectdiscovery/utils/conn/connpool.(*OneTimePool).Run()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.45-0.20230725161322-28ec1ee0ba40/conn/connpool/onetimepool.go:63 +0x6c
  github.com/projectdiscovery/utils/crypto/jarm.HashWithDialer.func2()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.45-0.20230725161322-28ec1ee0ba40/crypto/jarm/jarm.go:34 +0x2c

Goroutine 3883 (running) created at:
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets.func2()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:105 +0x6d4
  github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.(*Input).Scan.func2()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:308 +0xb4
  github.com/projectdiscovery/hmap/store/disk.(*LevelDB).Scan()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.13/store/disk/leveldb.go:224 +0x250
  github.com/projectdiscovery/hmap/store/hybrid.(*HybridMap).Scan()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.13/store/hybrid/hybrid.go:258 +0x170
  github.com/projectdiscovery/nuclei/v2/pkg/core/inputs/hybrid.(*Input).Scan()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/inputs/hybrid/hmap.go:316 +0x108
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:76 +0x81c
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateSpray.func1()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/execute_options.go:102 +0x94
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateSpray.func2()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/execute_options.go:103 +0x44

Goroutine 13972 (running) created at:
  github.com/projectdiscovery/utils/crypto/jarm.HashWithDialer()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/utils@v0.0.45-0.20230725161322-28ec1ee0ba40/crypto/jarm/jarm.go:34 +0x2a0
  github.com/projectdiscovery/dsl.init.0.func84()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/dsl@v0.0.16/dsl.go:1121 +0xd0
  github.com/projectdiscovery/dsl.dslFunction.Exec()
      /Users/root/go/pkg/mod/github.com/projectdiscovery/dsl@v0.0.16/func.go:52 +0x1e0
  github.com/projectdiscovery/dsl.dslFunction.Exec-fm()
      <autogenerated>:1 +0x88
  github.com/Knetic/govaluate.makeFunctionStage.func1()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/evaluationStage.go:245 +0x140
  github.com/Knetic/govaluate.EvaluableExpression.evaluateStage()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:232 +0x678
  github.com/Knetic/govaluate.EvaluableExpression.evaluateStage()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:172 +0xec
  github.com/Knetic/govaluate.EvaluableExpression.Eval()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:163 +0x148
  github.com/Knetic/govaluate.EvaluableExpression.Evaluate()
      /Users/root/go/pkg/mod/github.com/!knetic/govaluate@v3.0.1-0.20171022003610-9aa49832a739+incompatible/EvaluableExpression.go:137 +0xb4
  github.com/projectdiscovery/nuclei/v2/pkg/operators/matchers.(*Matcher).MatchDSL()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/operators/matchers/match.go:191 +0x1e4
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).Match()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/operators.go:32 +0xfc
  github.com/projectdiscovery/nuclei/v2/pkg/protocols.Request.Match-fm()
      <autogenerated>:1 +0x58
  github.com/projectdiscovery/nuclei/v2/pkg/operators.(*Operators).Execute()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/operators/operators.go:271 +0x69c
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/helpers/eventcreator.CreateEventWithAdditionalOptions()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/common/helpers/eventcreator/eventcreator.go:28 +0x1e0
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeRequestWithPayloads()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:300 +0x2870
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).executeAddress()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:108 +0x664
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/network.(*Request).ExecuteWithResults()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/network/request.go:71 +0x52c
  github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/protocols/common/executer/executer.go:93 +0x708
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets.func2.1()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:128 +0x40c
  github.com/projectdiscovery/nuclei/v2/pkg/core.(*Engine).executeTemplateWithTargets.func2.2()
      /Users/root/projectdiscovery-workspace/forks/nuclei/v2/pkg/core/executors.go:135 +0x60
==================

test

✗ go run -race . -l ../functional-test/targets.txt

Note: ../functional-test/targets.txt contains 150 targets

Checklist

Pull request is created against the dev branch
All checks passed (lint, unit/integration/regression tests etc.) with my changes
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if appropriate)

RamanaReddy0M · 2023-08-09T20:08:33Z

upstream pr: projectdiscovery/utils#235

tarunKoyalwar

lgtm ! upstream PR might need some changes as suggested by @Mzack9999

Mzack9999 · 2023-08-23T21:26:05Z

v2/pkg/protocols/common/executer/executer.go

@@ -71,6 +72,7 @@ func (e *Executer) Execute(input *contextargs.Context) (bool, error) {
 	}
 	previous := make(map[string]interface{})

+	mtx := &sync.Mutex{}


this mutex seems unnecessary as the variable has write-only operations within two mutually exclusive code branches. Could you verify if removing it reintroduces somehow the data race?

yes @Mzack9999, I have tried again somehow it's reintroduces data race issue without a mutex wrapper.

[openssh-detect] [tcp] [info] scanme.sh:22 [SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.9] ================== WARNING: DATA RACE Write at 0x00c00d251da0 by goroutine 7295350: github.com/projectdiscovery/nuclei/v2/pkg/protocols/common/executer.(*Executer).Execute.func3() /Users/ramana/projectdiscovery-workspace/nuclei/v2/pkg/protocols/common/executer/executer.go:

fix data-race warnings

3946cee

RamanaReddy0M self-assigned this Aug 9, 2023

RamanaReddy0M linked an issue Aug 9, 2023 that may be closed by this pull request

WARNING: DATA RACE #4016

Closed

RamanaReddy0M mentioned this pull request Aug 9, 2023

WARNING: DATA RACE with OneTimePool projectdiscovery/utils#234

Closed

update utils pkg

bdc9660

RamanaReddy0M requested a review from tarunKoyalwar August 9, 2023 20:00

tarunKoyalwar approved these changes Aug 10, 2023

View reviewed changes

RamanaReddy0M added 2 commits August 22, 2023 15:57

resolve merge conflicts

5177d10

utils update

db7e150

RamanaReddy0M marked this pull request as draft August 22, 2023 17:33

RamanaReddy0M added 2 commits August 23, 2023 21:57

Merge branch 'dev' into issue-4016-data-race

c5dfa06

update utils

83320fd

RamanaReddy0M marked this pull request as ready for review August 23, 2023 18:31

RamanaReddy0M requested a review from Mzack9999 August 23, 2023 18:32

bumping utils

a4b0b0d

Mzack9999 requested changes Aug 23, 2023

View reviewed changes

Merge branch 'dev' into issue-4016-data-race

006443d

ehsandeep approved these changes Sep 4, 2023

View reviewed changes

ehsandeep merged commit 5bd4e68 into dev Sep 4, 2023
12 checks passed

ehsandeep deleted the issue-4016-data-race branch September 4, 2023 07:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix data-race warnings #4036

fix data-race warnings #4036

RamanaReddy0M commented Aug 9, 2023 •

edited

Loading

RamanaReddy0M commented Aug 9, 2023

tarunKoyalwar left a comment

Mzack9999 Aug 23, 2023

RamanaReddy0M Aug 24, 2023

fix data-race warnings #4036

fix data-race warnings #4036

Conversation

RamanaReddy0M commented Aug 9, 2023 • edited Loading

Proposed changes

test

Checklist

RamanaReddy0M commented Aug 9, 2023

tarunKoyalwar left a comment

Choose a reason for hiding this comment

Mzack9999 Aug 23, 2023

Choose a reason for hiding this comment

RamanaReddy0M Aug 24, 2023

Choose a reason for hiding this comment

RamanaReddy0M commented Aug 9, 2023 •

edited

Loading