math/big: panic: mismatched montgomery number lengths

[fisherFIAF]

go version 1.16.6

GRPC panicked when serving.

tracing stack:

panic: math/big: mismatched montgomery number lengths

goroutine 1787827 [running]:
math/big.nat.montgomery(0x0, 0x0, 0x0, 0xc0056b8f80, 0x10, 0x10, 0xc0056b7400, 0x1a, 0x26, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:219 +0x507
math/big.nat.expNNMontgomery(0xc0035926c0, 0x40, 0x44, 0xc003592b40, 0x10, 0x45, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1430 +0x327
math/big.nat.expNN(0xc0035926c0, 0x40, 0x44, 0xc003592000, 0x40, 0x44, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1262 +0xa06
math/big.(*Int).Exp(0xc003f733d8, 0xc003f733f8, 0xc0005ca3e0, 0xc0005ca340, 0xc000bbbf60)
        /usr/local/go/src/math/big/int.go:509 +0x1a5
crypto/rsa.decrypt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f733f8, 0xc003f734e8, 0x5d4506, 0x13fd3c0)
        /usr/local/go/src/crypto/rsa/rsa.go:536 +0x19f
crypto/rsa.decryptAndCheck(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f73580, 0x100, 0x100, 0xc0053d57a0)
        /usr/local/go/src/crypto/rsa/rsa.go:570 +0x53
crypto/rsa.signPSSWithSalt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0019e4ae0, 0x20, 0x20, ...)
        /usr/local/go/src/crypto/rsa/pss.go:217 +0x1c5
crypto/rsa.SignPSS(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0097e5fe0, 0x40df45, 0x14b45e0, ...)
        /usr/local/go/src/crypto/rsa/pss.go:281 +0x1da
crypto/rsa.(*PrivateKey).Sign(0xc0005d26c0, 0x18a0900, 0xc0000e8480, 0xc0019e4ac0, 0x20, 0x20, 0x18a0920, 0xc0097e5fe0, 0xc005580000, 0x25, ...)
        /usr/local/go/src/crypto/rsa/rsa.go:146 +0x9e
crypto/tls.(*serverHandshakeStateTLS13).sendServerCertificate(0xc003f739e8, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:636 +0x466
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc003f739e8, 0xc00103ac00, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:59 +0xd4
crypto/tls.(*Conn).serverHandshake(0xc005ab3c00, 0x0, 0x7f982045c2c0)
        /usr/local/go/src/crypto/tls/handshake_server.go:51 +0xd4
crypto/tls.(*Conn).Handshake(0xc005ab3c00, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1391 +0xc9
google.golang.org/grpc/credentials.(*tlsCreds).ServerHandshake(0xc00063c230, 0x18d9ee0, 0xc001212180, 0x244b200, 0xe9, 0x0, 0x0, 0xc0005176f0, 0x78)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/credentials/tls.go:115 +0xdf
google.golang.org/grpc.(*Server).useTransportAuthenticator(0xc000819500, 0x18d9ee0, 0xc001212180, 0x244b200, 0x0, 0x0, 0x244b200, 0xc0058ffbf0, 0xc0492d074d535ba5)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:651 +0x50
google.golang.org/grpc.(*Server).handleRawConn(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:776 +0xca
google.golang.org/grpc.(*Server).Serve.func3(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:762 +0x3f
created by google.golang.org/grpc.(*Server).Serve
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:761 +0x305

It happened when server is online, and very seldom, nearly once one or two weeks.
It seems happened when GRPC server handshake when new connection come.

linux kernel version: 3.10.0-1160.36.2.el7.x86_64

CPU:

processor       : 31
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
stepping        : 4
microcode       : 0x1
cpu MHz         : 3000.000
cache size      : 25344 KB
physical id     : 1
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 31
initial apicid  : 31
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat md_clear spec_ctrl intel_stibp flush_l1d
bogomips        : 6000.00
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:

[ericlagergren]

This looks like a security bug: https://golang.org/security

@FiloSottile

[randall77]

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

[fisherFIAF]

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

can I resolve this problem by upgrading go version to 1.17

[randall77]

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

[FiloSottile]

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

[fisherFIAF]

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

thanks very much, we will have a try.

[fisherFIAF]

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

I got it, thank you very much.

[randall77]

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

[fisherFIAF]

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

we run it on the server, every GRPC request create a new connection, hunders connection very seconds, it panic randomly recently, but not often, and I don't know how it happened either. I think we should reduce the connection times first.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[ericlagergren]

Just another data point (since this got bumped by gopherbot): I've been running Keith's program for over a cumulative 48 hours across 3 machines (both amd64 and arm64) and haven't seen any errors, either.