Invalid memory address or nil pointer dereference #48
Comments
|
This seems strange. Does it happen occasionally or does it happen to every requests? |
|
The file did not show how you used your RoundTripper created by NewRoundTripper. Is it use-and-throw or do you reuse it multiple times? As you can see, you'd close the client after each RoundTripper.RoundTrip() call. If you reuse that RoundTripper, you'd probably be using a handler on an already closed client connection, which would become The design of gofast is to have the ClientFactory reused, not the Client. |
|
I'll have a more friendly error for this situation in the future. |
|
@yookoala Thanks for your thorough investigation on this issue. I'm the author of the gofast implementation in Skipper. I will try to create a PR to include your suggestion. |
|
I checked, and the RoundTripper is actually created for every single request. It is never reused. |
|
Hmm... I'll do some further check. |
|
I'm not sure if it's related, but before the segmentation violation there are sometimes timeouts. |
|
I'm not working often in Golang, please forgive me if what I say makes no sense. I've been looking for potential memory that is not cleaned up.
|
@yookoala mentioned that the ClientFactory should be reused in the comment. However, if I read it right, the ClientFactory is created on every request. @yookoala @ruudk Should this not be changed in Skipper to only create the ClientFactory once? I think the problem is with creating new request IDs. When the ClientFactory is not reused, I guess that there are new ID pools created on every request? Explanation How to dump
|
|
Skipper creates a new client for every request. This is because Skipper handles a lot of different traffic to different servers. So only when the requests comes in, it knows where to send it to. It might be something that we can optimize in the future, but it is what it is right now. We found out that when gofast creates a new client, it also spawns a goroutine that handles the ID pool. These are the leaking goroutines. When Skipper sends the request to gofast, it never calls When that is merged and tagged, we can create a PR for Skipper to properly call |
|
I think I can stably reproduce this "Invalid memory address or nil pointer dereference" issue with this test: func TestClient_close_before_readRequest(t *testing.T) {
// Test closing client before writeRequest
//
// Note: closing connection could cause client.conn nil
// writing after close could cause error
// create a temp dummy fastcgi application server
p, err := newAppServer("client.test.sock", func(w http.ResponseWriter, r *http.Request) {
t.Logf("accessing FastCGI process")
fmt.Fprintf(w, "hello world")
})
if err != nil {
t.Fatalf("Unable to start server: %s", err.Error())
}
defer p.Close()
// create reusable client factory
c, err := gofast.SimpleClientFactory(
gofast.SimpleConnFactory(p.Network(), p.Address()),
)()
if err != nil {
t.Fatalf("Unable for client factory to connect to server: %s", err.Error())
}
// Mock creating a gofast.Request that still
// pending to write a body.
pr, pw := io.Pipe()
r, err := http.NewRequest("POST", "/add", pr)
r.Header.Add("Content-Length", "11")
if err != nil {
t.Errorf("unexpected error: %#v", err.Error())
}
writeBody := func() {
c.Close()
pw.Write([]byte("hello world"))
}
req := gofast.NewRequest(r)
// handle the result
go writeBody()
log.Printf("checkpoint 1")
resp, err := c.Do(req)
if err != nil {
t.Fatalf("web server: unable to connect to process request: %s", err.Error())
return
}
w := httptest.NewRecorder()
errBuffer := new(bytes.Buffer)
resp.WriteTo(w, errBuffer)
if want, have := "", w.Body.String(); want != have {
t.Errorf("expected %v, got %v", want, have)
}
}Still working on way to resolve it. |
I'm seeing this error in production in combination with skipper. Unfortunately I wasn't able to get the exact request yet.
Do you maybe have some pointers (pun intended) how to debug this?
Relevant code segment:
https://github.com/yookoala/gofast/blob/v0.4.0/client.go#L199-L239
The text was updated successfully, but these errors were encountered: