Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc: panic during addExport #54

Closed
et4te opened this issue Oct 19, 2016 · 6 comments
Closed

rpc: panic during addExport #54

et4te opened this issue Oct 19, 2016 · 6 comments
Assignees
Labels

Comments

@et4te
Copy link

et4te commented Oct 19, 2016

Hey there, just came across this possible bug, any idea what its about?

panic: runtime error: index out of range
goroutine 358 [running]:
panic(0xa4efc0, 0xc82000a150)
        /nix/store/jvsx0g86is0ryjxdrsql5q7nv349jnf1-go-1.6.3/share/go/src/runtime/panic.go:481 +0x3e6
zombiezen.com/go/capnproto2/rpc.(*Conn).addExport(0xc8202a41a0, 0x7f4327c1fb00, 0xc82009a060, 0x0)
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/tables.go:175 +0x3df
zombiezen.com/go/capnproto2/rpc.(*Conn).descriptorForClient(0xc8202a41a0, 0xc8201d4060, 0x800000050, 0xffffffffffff0001, 0xfffffffffffffffe, 0x1, 0x7f4327c1f9e8, 0xc8200c0800, 0x0, 0x0)
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/introspect.go:202 +0x156
zombiezen.com/go/capnproto2/rpc.(*Conn).makeCapTable(0xc8202a41a0, 0xc8201d4060, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/rpc.go:576 +0x3e2
zombiezen.com/go/capnproto2/rpc.(*answer).fulfill(0xc820076a50, 0xc8202ac060, 0x8, 0x100000000, 0xffffffffffffffff, 0x0, 0x0, 0x0)
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/answer.go:76 +0x366
zombiezen.com/go/capnproto2/rpc.joinAnswer(0xc820076a50, 0x7f4327d8b9e8, 0xc820362778)
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/answer.go:225 +0x30b
created by zombiezen.com/go/capnproto2/rpc.(*Conn).routeCallMessage
        /home/fader/golang/src/zombiezen.com/go/capnproto2/rpc/rpc.go:698 +0x69f

Thx for the lib by the way awesome work.

@zombiezen
Copy link
Contributor

Yup, this seems like a bug! Looks like there's something about how the ID allocation code is going out-of-bound of the export table. I'll take a look and hopefully have a fix within the next couple of days.

@zombiezen zombiezen self-assigned this Oct 20, 2016
@zombiezen zombiezen added the bug label Oct 20, 2016
@zombiezen zombiezen changed the title Possible Bug rpc: panic during addExport Oct 20, 2016
@et4te
Copy link
Author

et4te commented Oct 20, 2016

Btw this happened when I was looping over stuff in RPC:

    conn := rpc.NewConn(rpc.StreamTransport(client))
    defer conn.Close()

    nexp := core.RPCFrame{Client: conn.Bootstrap(ctx)}

    dt := sd
    d  = core.EmptyDomain

    s := nexp.NewFrame(ctx, func(p core.RPCFrame_newFrame_Params) error {
        return nil
    }).Result()

    for {
        r, err := s.Activate(ctx, func(p core.RPCActivationFrame_activate_Params) error {
            netExpS, err := core.Serialize(core.RPCExpression{uid, e, k, d, sd})
            if err != nil {
                return err
            }
            return p.SetFrame(netExpS)
        }).Struct()

        if err != nil {
            return nil, err
        }

        viS, err := r.Result()
        if err != nil {
            return nil, err
        }

        vi, err := core.Deserialize(viS)
        if err != nil {
            return nil, err
        }
                ...

When I moved the nexp.NewFrame (the bootstrap struct) inside the for loop it fixed the problem for some reason.

@zombiezen
Copy link
Contributor

zombiezen commented Oct 29, 2016

Odd. I'm still not sure how to reproduce this one. In the sample you give, you don't seem to be exporting a capability to the peer: none of those calls create a server. My current hypotheses of what is happening are:

  1. You've exported so many capabilities (>2<<31) that the ID is wrapping around. This seems unlikely to me, but is probably something I should be checking.
  2. There may be an issue where an export gets added as the connection is shutting down. This seems a bit more likely, but I'm not sure how it would happen.

Can you modify your rpc package locally to log a bit of information for me? In particular, I'd like to have:

  1. c.infof("setting export %d, len = %d", id, len(c.exports) before rpc/tables.go, line 175 (the assignment that is panicing)
  2. c.infof("tearing down conn") after rpc/rpc.go, line 189 (the c.workers.Wait() line).

The first one may log a little bit too much, so you can wrap it in a conditional:

// ...
if int(id) < 0 || int(id) >= len(c.exports) {
  c.infof("setting export %d, len = %d", id, len(c.exports)
}
// ...

@zombiezen
Copy link
Contributor

Actually, I just reproduced a similar crash in my own code. It is the second case (trying to add exports/questions while the connection is dying). I don't think I need any additional information to be able to fix. Thanks for the report!

zombiezen added a commit that referenced this issue Jan 16, 2017
The panic identified in #54 is a race between dying and teardown.
teardown only starts once Conn.workers is done, so every non-dispatch
goroutine checks the state and adds itself to the c.workers pool.

This conceptually addresses the issue, but I'm not totally confident
that this fixes the bug.  It is hard to write a reliable automated repro
case for this issue.  It seems strictly superior to prior iterations of
the code, so I'll run a few experiments.

Updates #54
@zombiezen
Copy link
Contributor

@et4te, would you mind pulling and seeing whether this new commit fixes your issue?

@zombiezen
Copy link
Contributor

Marking as fixed, since I haven't heard anything for a couple months.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants