rust agent crashes on shutdown

[jodh-intel]

Tracking issue for a problem I'm seeing related to adding tracing (which requires a clean shutdown):

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', src/libcore/result.rs:1188:5

This problem is seen calling destroySandbox on master and appears to be caused by a thread (created by a dependent crate?) failing when operating on an mpsc channel. Debugging is complicated by the fact that the crash is random (threads ;) and the generated backtrace is not super-helpful:

stack backtrace:
   0:     0x55555634a734 - <unknown>
   1:     0x555556370c6c - <unknown>
   2:     0x555556345537 - <unknown>
   3:     0x55555634cf4e - <unknown>
   4:     0x55555634cc41 - <unknown>
   5:     0x55555634d62b - <unknown>
   6:     0x55555634d1de - <unknown>
   7:     0x55555636cf0e - <unknown>
   8:     0x55555636d007 - <unknown>
   9:     0x5555560760be - <unknown>
  10:     0x555556091e9c - <unknown>
  11:     0x5555560958c2 - <unknown>
  12:     0x55555608ef03 - <unknown>
  13:     0x5555560962c1 - <unknown>
  14:     0x5555560967d0 - <unknown>
  15:     0x555556355eba - <unknown>
  16:     0x555556096658 - <unknown>
  17:     0x555556096393 - <unknown>
  18:     0x55555608ecee - <unknown>
  19:     0x555556065164 - <unknown>
  20:     0x55555633dacf - <unknown>
  21:     0x555556354f80 - <unknown>
  22:     0x7ffff7da1669 - start_thread
  23:     0x7ffff7ef3323 - clone
  24:                0x0 - <unknown>

@lifupan - I'll add more specific details to this issue on Monday but any thoughts? Can you recreate?

[lifupan]

Hi @jodh-intel , from you description, was the panic occur here: https://github.com/kata-containers/kata-containers/blob/master/src/agent/src/grpc.rs#L1234 ?

[jodh-intel]

Hi @lifupan - The problem actually seems to be the custom VSOCK grpc-rs/grpc-sys crate we're using: shutdown seems to be unreliable with it, but not with the an upstream version:

I now have a test program that can shutdown its gRPC server cleanly using grpcio = "0.4.6", but with the crate we're using it crashes catastrophically on shutdown intermittently with the error shown at the top of this issue.

I really don't want to have to fix this behaviour in our custom crate as it was only ever supposed to be a temporary fix. So, this is a good time to task about two things:

• How close we are to being able to switch to the upstream grpcio crate?

  In other words, how close are we to landing the VSOCK changes: Add vsock support grpc/grpc#21121 seems to have stalled...?

  (See also rust-agent: Land rust agent into kata-containers #57 (comment)).

• How close are we to being able to switch to ttrpc? (simplify kata agent: use ttrpc as agent protocol #148).

/cc @amshinde, @bergwolf, @gnawux.

[jodh-intel]

@teawater - could you provide any update on grpc/grpc#21121?

[jodh-intel]

Ping all 😄

If we are unable to find a way to get the vsock changes landed upstream, there is of course ttrpc.

@bergwolf - is there any more information to share (a branch?) relating to #148?

[lifupan]

Ping all 😄

If we are unable to find a way to get the vsock changes landed upstream, there is of course ttrpc.

@bergwolf - is there any more information to share (a branch?) relating to #148?

Hi @jodh-intel

The ttrpc branch https://github.com/lifupan/kata-containers/commits/ttrpc is a poc that we did
two month ago.

[amshinde]

@lifupan Is the ttrpc code ready to be PR'd /merged?
@jodh-intel Can we make changes in our custom crate instead, to get you unblocked while the grpc vsock changes or the ttrpc code lands?

@bergwolf @gnawux Do you have any insights on the above? @jodh-intel is currently blocked on adding tracing support because of agent crashes seen on shutdown.

[bergwolf]

This has been fixed. Please reopen if you still see it with the rust agent @jodh-intel