Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime error: slice bounds out of range #48

Closed
kkroo opened this issue Feb 18, 2023 · 13 comments · Fixed by corazawaf/coraza#825
Closed

runtime error: slice bounds out of range #48

kkroo opened this issue Feb 18, 2023 · 13 comments · Fixed by corazawaf/coraza#825
Assignees
Labels
awaiting feedback bug Something isn't working

Comments

@kkroo
Copy link

kkroo commented Feb 18, 2023

Description

Describe the issue that you're seeing.
caddy server container crashed with panic. hope this might help someone. great project thank you

cdn-in-a-box-caddy-1 | panic: runtime error: slice bounds out of range [862:0]
cdn-in-a-box-caddy-1 |
cdn-in-a-box-caddy-1 | goroutine 21235869 [running]:
cdn-in-a-box-caddy-1 | github.com/corazawaf/coraza/v3/internal/corazawaf.(*bodyBufferReader).Read(0xc047d6cd50, {0xc038774000?, 0x0?, 0x7fce60935688?})
cdn-in-a-box-caddy-1 | github.com/corazawaf/coraza/v3@v3.0.0-20230117071831-8b909c7fc345/internal/corazawaf/body_buffer.go:98 +0x139
cdn-in-a-box-caddy-1 | io.discard.ReadFrom({}, {0x1f8f960, 0xc047d6cd50})
cdn-in-a-box-caddy-1 | io/io.go:611 +0x72
cdn-in-a-box-caddy-1 | io.copyBuffer({0x1f97800, 0x2ca05e0}, {0x1f8f960, 0xc047d6cd50}, {0xc05461a000, 0x8000, 0x8000})
cdn-in-a-box-caddy-1 | io/io.go:413 +0x14b
cdn-in-a-box-caddy-1 | io.(*multiReader).writeToWithBuffer(0xc051094c48, {0x1f97800, 0x2ca05e0}, {0xc05461a000, 0x8000, 0x8000})
cdn-in-a-box-caddy-1 | io/multi.go:54 +0x125
cdn-in-a-box-caddy-1 | io.(*multiReader).WriteTo(0x19bcd00?, {0x1f97800, 0x2ca05e0})
cdn-in-a-box-caddy-1 | io/multi.go:45 +0x56
cdn-in-a-box-caddy-1 | io.copyBuffer({0x1f97800, 0x2ca05e0}, {0x7fce601809b8, 0xc047d4bbf0}, {0x0, 0x0, 0x0})
cdn-in-a-box-caddy-1 | io/io.go:409 +0x16e
cdn-in-a-box-caddy-1 | io.Copy(...)
cdn-in-a-box-caddy-1 | io/io.go:386
cdn-in-a-box-caddy-1 | net/http.(*transferWriter).doBodyCopy(0xc02cddcdc0, {0x1f97800?, 0x2ca05e0?}, {0x7fce601809b8?, 0xc047d4bbf0?})
cdn-in-a-box-caddy-1 | net/http/transfer.go:412 +0x4d
cdn-in-a-box-caddy-1 | net/http.(*transferWriter).writeBody(0xc02cddcdc0, {0x1f8e160, 0xc0293ac640})
cdn-in-a-box-caddy-1 | net/http/transfer.go:375 +0x418
cdn-in-a-box-caddy-1 | net/http.(*Request).write(0xc069109700, {0x1f8e160, 0xc0293ac640}, 0x0, 0x0, 0x0)
cdn-in-a-box-caddy-1 | net/http/request.go:701 +0xb46
cdn-in-a-box-caddy-1 | net/http.(*persistConn).writeLoop(0xc0595f9b00)
cdn-in-a-box-caddy-1 | net/http/transport.go:2395 +0x174
cdn-in-a-box-caddy-1 | created by net/http.(*Transport).dialConn
cdn-in-a-box-caddy-1 | net/http/transport.go:1752 +0x1791

Steps to reproduce

Clear steps describing how to reproduce the issue. Please please please link to a demo project if possible, this makes your issue much easier to diagnose (seriously).
reverse proxy server being hammered by bots

Expected result

What should happen?

Actual result

What happened.

@jcchavezs
Copy link
Member

jcchavezs commented Feb 18, 2023 via email

@jcchavezs
Copy link
Member

This is a weird issue and I guess it happens only because you run bots and at some point during the read you also write and then move towards a file.

@kkroo
Copy link
Author

kkroo commented Feb 23, 2023

Thanks for the fix @jcchavezs. I am seeing this crash a few times per day right now. Would appreciate a release into corazawaf/coraza-caddy when the change is landed.

@jcchavezs
Copy link
Member

jcchavezs commented Feb 24, 2023 via email

@kkroo
Copy link
Author

kkroo commented Feb 25, 2023

Just rebuilt from master and report back if I see another crash.
Doesn't look like the fix made it in the latest version though.
github.com/corazawaf/coraza-caddy v1.2.3-0.20230224222231-e5bec900d1d2 github.com/corazawaf/coraza/v3 v3.0.0-20230222164726-1ac270a3257d

@jcchavezs
Copy link
Member

jcchavezs commented Feb 25, 2023 via email

@kkroo
Copy link
Author

kkroo commented Feb 26, 2023

cdn-in-a-box-caddy-1 | panic: runtime error: slice bounds out of range [45:0]
cdn-in-a-box-caddy-1 |
cdn-in-a-box-caddy-1 | goroutine 22441553 [running]:
cdn-in-a-box-caddy-1 | github.com/corazawaf/coraza/v3/internal/corazawaf.(*bodyBufferReader).Read(0xc02e3c0b70, {0xc031f20000?, 0x2c6a100?, 0x2c63a78?})
cdn-in-a-box-caddy-1 | github.com/corazawaf/coraza/v3@v3.0.0-20230222164726-1ac270a3257d/internal/corazawaf/body_buffer.go:111 +0x12a
cdn-in-a-box-caddy-1 | io.discard.ReadFrom({}, {0x1f84c80, 0xc02e3c0b70})
cdn-in-a-box-caddy-1 | io/io.go:611 +0x72
cdn-in-a-box-caddy-1 | io.copyBuffer({0x1f8cb20, 0x2c94800}, {0x1f84c80, 0xc02e3c0b70}, {0xc02f500000, 0x8000, 0x8000})
cdn-in-a-box-caddy-1 | io/io.go:413 +0x14b
cdn-in-a-box-caddy-1 | io.(*multiReader).writeToWithBuffer(0xc02e27b6b0, {0x1f8cb20, 0x2c94800}, {0xc02f500000, 0x8000, 0x8000})
cdn-in-a-box-caddy-1 | io/multi.go:54 +0x125
cdn-in-a-box-caddy-1 | io.(*multiReader).WriteTo(0x19b4760?, {0x1f8cb20, 0x2c94800})
cdn-in-a-box-caddy-1 | io/multi.go:45 +0x56
cdn-in-a-box-caddy-1 | io.copyBuffer({0x1f8cb20, 0x2c94800}, {0x7fb53ee235b8, 0xc0c52a8870}, {0x0, 0x0, 0x0})
cdn-in-a-box-caddy-1 | io/io.go:409 +0x16e
cdn-in-a-box-caddy-1 | io.Copy(...)
cdn-in-a-box-caddy-1 | io/io.go:386
cdn-in-a-box-caddy-1 | net/http.(*transferWriter).doBodyCopy(0xc019535680, {0x1f8cb20?, 0x2c94800?}, {0x7fb53ee235b8?, 0xc0c52a8870?})
cdn-in-a-box-caddy-1 | net/http/transfer.go:412 +0x4d
cdn-in-a-box-caddy-1 | net/http.(*transferWriter).writeBody(0xc019535680, {0x1f83480, 0xc024ad7200})
cdn-in-a-box-caddy-1 | net/http/transfer.go:375 +0x418
cdn-in-a-box-caddy-1 | net/http.(*Request).write(0xc08c25c000, {0x1f83480, 0xc024ad7200}, 0x0, 0x0, 0x0)
cdn-in-a-box-caddy-1 | net/http/request.go:701 +0xb46
cdn-in-a-box-caddy-1 | net/http.(*persistConn).writeLoop(0xc043deeb40)
cdn-in-a-box-caddy-1 | net/http/transport.go:2395 +0x174
cdn-in-a-box-caddy-1 | created by net/http.(*Transport).dialConn
cdn-in-a-box-caddy-1 | net/http/transport.go:1752 +0x1791

@jcchavezs jcchavezs added the bug Something isn't working label Feb 27, 2023
@jcchavezs
Copy link
Member

jcchavezs commented Feb 27, 2023

Thanks for this. I think the issue is the way the middleware is handled vs how coraza expects it to handle. We need to upgrade caddy's middleware and make it somehow with http middleware. I am moving this issue to Caddy for now and do the work there.

@jcchavezs jcchavezs transferred this issue from corazawaf/coraza Feb 27, 2023
@jcchavezs
Copy link
Member

jcchavezs commented Mar 5, 2023 via email

@jcchavezs
Copy link
Member

@kkroo we completely rewrite the connector. Do you mind testing again with latest commit?

@jcchavezs
Copy link
Member

Ping @kkroo

@fzipi
Copy link
Member

fzipi commented May 21, 2023

@kkroo Can you please help us getting this solved?

jcchavezs added a commit to corazawaf/coraza that referenced this issue Jun 25, 2023
Currently, the body buffer would hold a piece of the body, hence a reader will be sent to the connector in order to pass it to upstream. The problem happens when the transaction is closed, the body buffer is resetted but the reader is still out there and the connector tries to drain it (using something like io.Copy(io.Discard, body) in order to reuse the connection, the body buffer is already empty (lenght 0) but the body buffer reader points to the end of the buffer (e.g. 512), hence attempting to read it till the end (from 512 to 0) trigger an out of range error.

Closes corazawaf/coraza-caddy#48.
jcchavezs added a commit to corazawaf/coraza that referenced this issue Jun 25, 2023
Currently, the body buffer would hold a piece of the body, hence a reader will be sent to the connector in order to pass it to upstream. The problem happens when the transaction is closed, the body buffer is resetted but the reader is still out there and the connector tries to drain it (using something like io.Copy(io.Discard, body) in order to reuse the connection, the body buffer is already empty (lenght 0) but the body buffer reader points to the end of the buffer (e.g. 512), hence attempting to read it till the end (from 512 to 0) trigger an out of range error.

Closes corazawaf/coraza-caddy#48.
jcchavezs added a commit to corazawaf/coraza that referenced this issue Jun 26, 2023
…825)

* fix: blocks body buffer reader once the body buffer has been reset.

Currently, the body buffer would hold a piece of the body, hence a reader will be sent to the connector in order to pass it to upstream. The problem happens when the transaction is closed, the body buffer is resetted but the reader is still out there and the connector tries to drain it (using something like io.Copy(io.Discard, body) in order to reuse the connection, the body buffer is already empty (lenght 0) but the body buffer reader points to the end of the buffer (e.g. 512), hence attempting to read it till the end (from 512 to 0) trigger an out of range error.

Closes corazawaf/coraza-caddy#48.
@jcchavezs
Copy link
Member

@iMashtak would you be up to create a PR on this repo upgrading to Coraza v3.0.2 addressing this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting feedback bug Something isn't working
Projects
None yet
3 participants