Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: false-sharing observed on mheap for high core count environment #62472

Closed
gavinlichn opened this issue Sep 6, 2023 · 1 comment
Closed
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@gavinlichn
Copy link
Contributor

gavinlichn commented Sep 6, 2023

What version of Go are you using (go version)?

$ go version
go version devel go1.22-ac64a3628b Sat Aug 12 03:56:58 2023 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/ansible/ligang/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/ansible/ligang/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.22-ac64a3628b Sat Aug 12 03:56:58 2023 +0000'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/ansible/ligang/minio/minio_svr/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3702239426=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I benchmarked MinIO with master version of go runtime.
Server: https://github.com/minio/minio
Client: https://github.com/minio/warp
Configuration:

  • Single node MinIO Server, without TLS enabled,
  • Run warp GET request, 128KB object and 512 concurrent
  • GOGC=1000
  • 2 socket server(56c per socket),
  • pin MinIO Server and warp with specific CPU cores
  • MinIO bind with 6 tmpfs ramdisks

What did you expect to see?

No significant false-sharing

What did you see instead?

While pin MinIO server with high core count, significant false-sharing observed.
While pin MinIO with 100 CPU cores(0-99), Pin warp with 12 CPU Core(100-111) perf c2c result as below(omited some minor info)

#        ----- HITM -----  -- Store Refs --  -- Data address -- ---------- cycles ----------    Total       cpu                                           Shared               
#   Num  RmtHitm  LclHitm   L1 Hit  L1 Miss       Offset  Node  rmt hitm  lcl hitm      load  records       cnt                                   Symbol  Object  Source:Line  
# .....  .......  .......  .......  .......  ...........  ....  ........  ........  ........  .......  ........  .......................................  .....  ...........
  ------------------------------------------------------
      1      398     1056       31        0    0x62b2640
  ------------------------------------------------------
           2.26%    0.57%    3.23%    0.00%          0x0     0      4582      3858      3165      215        42  [.] runtime.(*mheap).initSpan            minio  types.go:420  
           2.26%    1.14%   41.94%    0.00%          0x8     0      4356      2245      3288      330        42  [.] runtime.(*sweepLocked).sweep         minio  types.go:344  
           1.76%    0.09%   16.13%    0.00%         0x30     0      4728      1534      3506      180        41  [.] runtime.(*mheap).reclaim             minio  types.go:401  
          63.07%   63.92%    0.00%    0.00%         0x38     0      1859      1646      1931     5996        42  [.] runtime.writeHeapBits.flush          minio  mbitmap.go:865
           6.03%    8.05%    0.00%    0.00%         0x38     0      2464      3614      2256     6443        42  [.] runtime.writeHeapBits.flush          minio  mbitmap.go:893
           5.28%    6.63%    0.00%    0.00%         0x38     0      2050      1462      2166      812        42  [.] runtime.writeHeapBits.write          minio  mbitmap.go:812
           1.01%    2.37%    0.00%    0.00%         0x38     0      2540      1707      2348      222        42  [.] runtime.spanOfHeap                   minio  mheap.go:700  

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Sep 6, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/525955 mentions this issue: runtime: Resolve false sharing for frequent memory allocate workloads

@mknyszek mknyszek added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 6, 2023
@mknyszek mknyszek added this to the Backlog milestone Sep 6, 2023
@mknyszek mknyszek self-assigned this Sep 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

3 participants