Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase PFCP buffer size and update go-pfcp #366

Merged
merged 8 commits into from Dec 3, 2021
Merged

Conversation

pudelkoM
Copy link
Member

@pudelkoM pudelkoM commented Dec 1, 2021

We observed a pfcp agent crash when PFCP messages are larger than a certain size. The UDP receive buffer is too small and causes the message to be truncated.
Then, the library code reads beyond the buffer and causes the panic seen below:

panic: runtime error: slice bounds out of range [:50] with capacity 41
goroutine 13 [running]:
github.com/wmnsk/go-pfcp/ie.(*IE).UnmarshalBinary(0x300c00003fa00, {0xc0002eebb3, 0x2, 0x3})
    /pfcpiface/vendor/github.com/wmnsk/go-pfcp/ie/ie.go:367 +0x187
github.com/wmnsk/go-pfcp/ie.Parse({0xc0002eebb3, 0x29, 0x29}) 
    /pfcpiface/vendor/github.com/wmnsk/go-pfcp/ie/ie.go:339 +0x48
github.com/wmnsk/go-pfcp/ie.ParseMultiIEs({0xc0002ee610, 0x5dc, 0x5dc}) 
    /pfcpiface/vendor/github.com/wmnsk/go-pfcp/ie/ie.go:628 +0x8c 
github.com/wmnsk/go-pfcp/message.(*SessionEstablishmentRequest).UnmarshalBinary(0xc000232000, {0xc0002ee600, 0x51a530, 0xc000298008}) 
    /pfcpiface/vendor/github.com/wmnsk/go-pfcp/message/session-establishment-request.go:291 +0x6e
github.com/wmnsk/go-pfcp/message.Parse({0xc0002ee600, 0x5dc, 0x5dc}) 
    /pfcpiface/vendor/github.com/wmnsk/go-pfcp/message/message.go:116 +0x3ab
main.pfcpifaceMainLoop(0xc00037e500, {0xc0000ce240, 0x0}, {0xc0000ce250, 0xf}, {0xc0000ce260, 0x7}, {0x0, 0x0})
    /pfcpiface/conn.go:133 +0x8e5
created by main.main
    /pfcpiface/main.go:231 +0x94f

The proposed solution is to increase the buffer size to 4096 and check the PFCP header length field. The length check for IE parsing needs to happen in pfcp-go.

image

@pudelkoM pudelkoM force-pushed the pfcp-read-buf-size branch 3 times, most recently from b8ed5eb to e2f03a3 Compare December 1, 2021 23:06
pfcpiface/conn.go Outdated Show resolved Hide resolved
@pudelkoM pudelkoM force-pushed the pfcp-read-buf-size branch 2 times, most recently from 93a98e9 to 14f75c3 Compare December 2, 2021 00:07
@pudelkoM
Copy link
Member Author

pudelkoM commented Dec 2, 2021

Managed to reproduce the crash in a small test, see latest commit:

=== RUN   Test
--- FAIL: Test (0.00s)
panic: runtime error: slice bounds out of range [:20] with capacity 5 [recovered]
	panic: runtime error: slice bounds out of range [:20] with capacity 5

goroutine 34 [running]:
testing.tRunner.func1.2({0x15f2ae0, 0xc0003dc018})
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1212 +0x218
panic({0x15f2ae0, 0xc0003dc018})
	/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1038 +0x215
github.com/wmnsk/go-pfcp/ie.(*IE).UnmarshalBinary(0xc00006c6c0, {0xc0002a8d88, 0x5, 0x153f8e0})
	/Users/max/go/pkg/mod/github.com/wmnsk/go-pfcp@v0.0.11/ie/ie.go:367 +0x187
github.com/wmnsk/go-pfcp/ie.Parse({0xc0002a8d88, 0x5, 0x5})
	/Users/max/go/pkg/mod/github.com/wmnsk/go-pfcp@v0.0.11/ie/ie.go:339 +0x48
github.com/omec-project/upf-epc/pfcpiface.Test(0xc000292680)
	/Users/max/projects/upf-epc/pfcpiface/conn_test.go:12 +0x4d
testing.tRunner(0xc000292680, 0x1680060)
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/usr/local/Cellar/go/1.17.2/libexec/src/testing/testing.go:1306 +0x35a


Process finished with the exit code 1

Probably because of a missing range check here: https://github.com/omec-project/upf-epc/blob/a6b6a4e29cc56c902cf7ca03b10a7d01f9cd98ae/pfcpiface/vendor/github.com/wmnsk/go-pfcp/ie/ie.go#L367

I don't think this is fixable here, the library needs to be updated.

@pudelkoM pudelkoM changed the title Increase PFCP buffer size and check for message truncation Increase PFCP buffer size Dec 2, 2021
pfcpiface/conn_test.go Outdated Show resolved Hide resolved
krsna1729
krsna1729 previously approved these changes Dec 2, 2021
krsna1729
krsna1729 previously approved these changes Dec 2, 2021
@pudelkoM pudelkoM changed the title Increase PFCP buffer size Increase PFCP buffer size and update go-pfcp Dec 2, 2021
@pudelkoM
Copy link
Member Author

pudelkoM commented Dec 2, 2021

I got confirmation from QA team that this fix prevents the original crash.

@krsna1729 krsna1729 merged commit be08d25 into master Dec 3, 2021
@krsna1729 krsna1729 deleted the pfcp-read-buf-size branch December 3, 2021 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants