Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

math/big: panic: mismatched montgomery number lengths #48431

Open
fisherFIAF opened this issue Sep 17, 2021 · 9 comments
Open

math/big: panic: mismatched montgomery number lengths #48431

fisherFIAF opened this issue Sep 17, 2021 · 9 comments

Comments

@fisherFIAF
Copy link

@fisherFIAF fisherFIAF commented Sep 17, 2021

go version 1.16.6

GRPC panicked when serving.

tracing stack:

panic: math/big: mismatched montgomery number lengths

goroutine 1787827 [running]:
math/big.nat.montgomery(0x0, 0x0, 0x0, 0xc0056b8f80, 0x10, 0x10, 0xc0056b7400, 0x1a, 0x26, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:219 +0x507
math/big.nat.expNNMontgomery(0xc0035926c0, 0x40, 0x44, 0xc003592b40, 0x10, 0x45, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1430 +0x327
math/big.nat.expNN(0xc0035926c0, 0x40, 0x44, 0xc003592000, 0x40, 0x44, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1262 +0xa06
math/big.(*Int).Exp(0xc003f733d8, 0xc003f733f8, 0xc0005ca3e0, 0xc0005ca340, 0xc000bbbf60)
        /usr/local/go/src/math/big/int.go:509 +0x1a5
crypto/rsa.decrypt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f733f8, 0xc003f734e8, 0x5d4506, 0x13fd3c0)
        /usr/local/go/src/crypto/rsa/rsa.go:536 +0x19f
crypto/rsa.decryptAndCheck(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f73580, 0x100, 0x100, 0xc0053d57a0)
        /usr/local/go/src/crypto/rsa/rsa.go:570 +0x53
crypto/rsa.signPSSWithSalt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0019e4ae0, 0x20, 0x20, ...)
        /usr/local/go/src/crypto/rsa/pss.go:217 +0x1c5
crypto/rsa.SignPSS(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0097e5fe0, 0x40df45, 0x14b45e0, ...)
        /usr/local/go/src/crypto/rsa/pss.go:281 +0x1da
crypto/rsa.(*PrivateKey).Sign(0xc0005d26c0, 0x18a0900, 0xc0000e8480, 0xc0019e4ac0, 0x20, 0x20, 0x18a0920, 0xc0097e5fe0, 0xc005580000, 0x25, ...)
        /usr/local/go/src/crypto/rsa/rsa.go:146 +0x9e
crypto/tls.(*serverHandshakeStateTLS13).sendServerCertificate(0xc003f739e8, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:636 +0x466
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc003f739e8, 0xc00103ac00, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:59 +0xd4
crypto/tls.(*Conn).serverHandshake(0xc005ab3c00, 0x0, 0x7f982045c2c0)
        /usr/local/go/src/crypto/tls/handshake_server.go:51 +0xd4
crypto/tls.(*Conn).Handshake(0xc005ab3c00, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1391 +0xc9
google.golang.org/grpc/credentials.(*tlsCreds).ServerHandshake(0xc00063c230, 0x18d9ee0, 0xc001212180, 0x244b200, 0xe9, 0x0, 0x0, 0xc0005176f0, 0x78)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/credentials/tls.go:115 +0xdf
google.golang.org/grpc.(*Server).useTransportAuthenticator(0xc000819500, 0x18d9ee0, 0xc001212180, 0x244b200, 0x0, 0x0, 0x244b200, 0xc0058ffbf0, 0xc0492d074d535ba5)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:651 +0x50
google.golang.org/grpc.(*Server).handleRawConn(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:776 +0xca
google.golang.org/grpc.(*Server).Serve.func3(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:762 +0x3f
created by google.golang.org/grpc.(*Server).Serve
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:761 +0x305

It happened when server is online, and very seldom, nearly once one or two weeks.
It seems happened when GRPC server handshake when new connection come.

linux kernel version: 3.10.0-1160.36.2.el7.x86_64

CPU:

processor       : 31
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
stepping        : 4
microcode       : 0x1
cpu MHz         : 3000.000
cache size      : 25344 KB
physical id     : 1
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 31
initial apicid  : 31
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat md_clear spec_ctrl intel_stibp flush_l1d
bogomips        : 6000.00
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:
@ericlagergren
Copy link
Contributor

@ericlagergren ericlagergren commented Sep 17, 2021

This looks like a security bug: https://golang.org/security

@FiloSottile

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 17, 2021

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

Loading

@fisherFIAF
Copy link
Author

@fisherFIAF fisherFIAF commented Sep 18, 2021

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

can I resolve this problem by upgrading go version to 1.17

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 18, 2021

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

Loading

@seankhliao seankhliao changed the title panic: math/big: mismatched montgomery number lengths math/big: panic: mismatched montgomery number lengths Sep 18, 2021
@FiloSottile
Copy link
Contributor

@FiloSottile FiloSottile commented Sep 20, 2021

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

Loading

@fisherFIAF
Copy link
Author

@fisherFIAF fisherFIAF commented Sep 22, 2021

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

thanks very much, we will have a try.

Loading

@fisherFIAF
Copy link
Author

@fisherFIAF fisherFIAF commented Sep 22, 2021

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

I got it, thank you very much.

Loading

@randall77
Copy link
Contributor

@randall77 randall77 commented Sep 22, 2021

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

Loading

@fisherFIAF
Copy link
Author

@fisherFIAF fisherFIAF commented Sep 22, 2021

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

we run it on the server, every GRPC request create a new connection, hunders connection very seconds, it panic randomly recently, but not often, and I don't know how it happened either. I think we should reduce the connection times first.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants