cmd/compile: ccxt oom killed on compilation

[carlosmiei]

Go version

go version 1.24.0 darwin/arm64 / go version go1.23.7 linux/amd64 / etc

Output of go env in your module/workspace:

go env
AR='ar'
CC='cc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='c++'
GCCGO='gccgo'
GO111MODULE='auto'
GOARCH='arm64'
GOARM64='v8.0'
GOAUTH='netrc'
GOBIN='/Users/cjg/go/bin'
GOCACHE='/Users/cjg/Library/Caches/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/Users/cjg/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/9z/bllf1k3s08q344pgrjkfybd80000gn/T/go-build340361704=/tmp/go-build -gno-record-gcc-switches -fno-common'
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMOD=''
GOMODCACHE='/Users/cjg/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/cjg/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/opt/go/libexec'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/Users/cjg/Library/Application Support/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/opt/go/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

CCXT is a crypto trading SDK that supports more than 100+ exchanges making it a pretty big package (hundreds of files and millions of loc).

We have recently released a GO version of the package, but the reports of users not able to compile it due to an OOM error are stacking up.

We have tried different go versions (1.22/23/24) and OSes but the problem pops up in all of them apparently.

Related issues:

• Can't use ccxt in golang ccxt/ccxt#25663
• Encountering an out-of-memory error when building CCXT with Go 1.23.7 ccxt/ccxt#25493

What did you see happen?

The build hangs indefinitely, or the process gets OOM-killed.

What did you expect to see?

The build succeeded in a timely manner

[prattmic]

cc @golang/compiler

[prattmic]

Is https://pkg.go.dev/github.com/ccxt/ccxt/go/v4 the problematic package?

I count "only" ~670k lines of code in that package (not millions), but that is still a lot for a single package. :)

The package does build successfully for me, but it takes ~2min and cmd/compile has a peak memory usage of ~10GB.

[gabyhelp]

Related Issues

• cmd/compile: Go 1.9 oom killed on compilation #21713 (closed)
• cmd/go: 1.15.5 crashes on compiling workspace on darwin/amd64 with goroutine stack exceeds 1000000000-byte limit #43279 (closed)
• cmd/cgo: panics when building crypto/x509 on darwin and android #28540 (closed)
• cmd/compile: provide error message for giant strings instead of crashing #16394 (closed)
• cmd/compile: infinite loop in types2 #48136 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

CPU profile: https://pprof.host/mw4g/flamegraph
Heap profile: https://pprof.host/n04g/flamegraph?si=alloc_space

Not much stands out as odd to me here. Perhaps other compiler folks notice something.

The one odd thing is that this compilation took ~120s wall time and used ~173s CPU time, which isn't very parallel. The SSA backend is concurrent, so I'm a bit surprised it didn't make better use of concurrency.

[prattmic]

FWIW, even if we fix this to use less memory, I recommend you split this package up.

This package contains ~1500 exported types and functions. Most users are going to browse documentation on https://pkg.go.dev/github.com/ccxt/ccxt/go/v4, which is nigh-unusable. It took me ~10s just to scroll through the index as quickly as I could!

IMO, a more natural design in Go would be to use one package per exchange, and then users only import the packages for the exchanges they use. I think this would be nicer to use as a user plus it would almost certainly fix your build time issues, as each binary would only need to build the packages that are actually used.

i.e., in https://github.com/ccxt/ccxt?tab=readme-ov-file#go-1, instead of

import (
	"github.com/ccxt/ccxt/go/v4"
)

func main() {
	exchange := ccxt.NewBinance(...)

You'd have

import (
	"github.com/ccxt/ccxt/go/v5/binance"
)

func main() {
	exchange := binance.NewExchange(...)

[carlosmiei]

@prattmic Thanks for your answer.

I count "only" ~670k lines of code in that package (not millions), but that is still a lot for a single package. :)

Sorry that was a rough estimation I did not count them 😅

IMO, a more natural design in Go would be to use one package per exchange, and then users only import the packages for the exchanges they us

I agree but we have some limitations since most of the code is transpiled from TS and not natively implemented.

Nonetheless I appreciate your input and it will be taken into consideration