Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix race condition #2 #614

Open
kosegor opened this issue Jan 10, 2023 · 2 comments
Open

Fix race condition #2 #614

kosegor opened this issue Jan 10, 2023 · 2 comments
Assignees
@ricardopinto
Labels
go Pull requests that update Go code good first issue Good for newcomers t/bug Something isn't working

Comments

@kosegor
Copy link
Contributor

kosegor commented Jan 10, 2023

Found this while testing the app:

WARNING: DATA RACE
Write at 0x00c001e8def0 by goroutine 146:
  runtime.mapdelete_faststr()
      /usr/local/go/src/runtime/map_faststr.go:301 +0x0
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).CleanCache()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:104 +0x1cb
  github.com/alicenet/alicenet/consensus/dman.(*DMan).CleanCache()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/dman.go:57 +0x7e4
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).UpdateLocalState.func1()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/lstate/engine.go:194 +0x79d
  github.com/alicenet/alicenet/consensus/db.(*rawDataBase).Update()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/db/rawdb.go:45 +0xbe
  github.com/alicenet/alicenet/consensus/db.(*Database).Update()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/db/db.go:47 +0xba
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).UpdateLocalState()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/lstate/engine.go:117 +0xac
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).UpdateLocalState-fm()
      <autogenerated>:1 +0x39
  github.com/alicenet/alicenet/consensus.(*Synchronizer).loop.func2()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:363 +0x7a1
  github.com/alicenet/alicenet/consensus.(*Synchronizer).loop()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:370 +0x214
  github.com/alicenet/alicenet/consensus.(*Synchronizer).setupLoops.func2()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:438 +0x47

Previous read at 0x00c001e8def0 by goroutine 452:
  runtime.mapaccess2_faststr()
      /usr/local/go/src/runtime/map_faststr.go:108 +0x0
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).doDownload.func5()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:241 +0x15c
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).doDownload()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:246 +0x115
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).download.func1()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:139 +0x64

Goroutine 146 (running) created at:
  github.com/alicenet/alicenet/consensus.(*Synchronizer).setupLoops()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:438 +0x1293
  github.com/alicenet/alicenet/consensus.(*Synchronizer).Start.func1()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:259 +0x184
  sync.(*Once).doSlow()
      /usr/local/go/src/sync/once.go:68 +0x101
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:59 +0x46
  github.com/alicenet/alicenet/consensus.(*Synchronizer).Start()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:253 +0x50
  github.com/alicenet/alicenet/cmd/node.validatorNode()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/cmd/node/node.go:376 +0x21e9
  main.runner.func1()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/cmd/main.go:68 +0x4d3
  github.com/spf13/cobra.(*Command).execute()
      /Users/kosegor/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920 +0xc5a
  github.com/spf13/cobra.(*Command).ExecuteC()
      /Users/kosegor/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044 +0x5dd
  github.com/spf13/cobra.(*Command).Execute()
      /Users/kosegor/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968 +0x7772
  main.main()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/cmd/main.go:317 +0x7765

Goroutine 452 (finished) created at:
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).download()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:139 +0x144
  github.com/alicenet/alicenet/consensus/dman.(*RootActor).DownloadBlockHeader()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/actors.go:130 +0xec
  github.com/alicenet/alicenet/consensus/dman.(*DMan).SyncOneBH()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/dman/dman.go:170 +0x10cd
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).Sync.func1()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/lstate/engine.go:590 +0xa75
  github.com/alicenet/alicenet/consensus/db.(*rawDataBase).Update()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/db/rawdb.go:45 +0xbe
  github.com/alicenet/alicenet/consensus/db.(*Database).Update()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/db/db.go:47 +0xba
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).Sync()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/lstate/engine.go:504 +0xac
  github.com/alicenet/alicenet/consensus/lstate.(*Engine).Sync-fm()
      <autogenerated>:1 +0x39
  github.com/alicenet/alicenet/consensus.(*Synchronizer).loop.func2()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:363 +0x7a1
  github.com/alicenet/alicenet/consensus.(*Synchronizer).loop()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:370 +0x214
  github.com/alicenet/alicenet/consensus.(*Synchronizer).setupLoops.func3()
      /Users/kosegor/workspace/madhive/vtleonardo/alicenet/consensus/synchronizer.go:454 +0x47

In order to reproduce:

  • run make race to build the binary with -race flag
  • start 5 validators, 1 of them using ./scripts/main.sh race <validator number>
  • complete ETHDKG ceremony
  • in a separate terminal run tx spammer with go run ./cmd/testutils/inject/main.go -workers 50

It took me more than 2 hours to reproduce it
@kosegor kosegor added t/bug Something isn't working good first issue Good for newcomers go Pull requests that update Go code labels Jan 10, 2023
@ricardopinto ricardopinto self-assigned this Jan 26, 2023
@iolloyd
Copy link

iolloyd commented Apr 15, 2023

Is this being worked on?

@ricardopinto
Copy link
Contributor

@iolloyd not anymore. I was able to patch the code for this specific case but was never able to reproduce it. While addressing this issue we were trying to fix a bug that halted consensus and made it difficult to test anything, including this ticket, and thus the patch never made it to the main branch. The code can be found on this branch https://github.com/alicenet/alicenet/tree/race-condition-adm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ricardopinto ricardopinto

Labels
go Pull requests that update Go code good first issue Good for newcomers t/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@iolloyd @ricardopinto @kosegor