"network error: dropped by socket" on RPC

[cjbe]

With the below test experiment ~50% of RPCs take >1s:

class RpcTimingTest(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("pmt")
        self.setattr_device("ttl0")
        self.t_pulse = 100*ms

    def dummy_rpc(self):
        pass

    @kernel
    def run(self):
        self.core.reset()
        for _ in range(10):
            self.core.break_realtime()
            with parallel:
                self.pmt.gate_rising(self.t_pulse)
                self.ttl0.pulse(self.t_pulse)
            delay(10*us)
            self.pmt.count()
            self.dummy_rpc()

Removing the dummy_rpc() call or the pmt.count() call fixes this issue.
I am using the current master branch and nist_clock gateware.

The core log over this period is:

[  1012193933us]  INFO(runtime::session): new connection from 10.255.6.32:38290
[  1012224636us]  WARN(runtime): network error: dropped by socket
[  1012229791us]  WARN(runtime): network error: dropped by socket
[  1012462590us]  INFO(runtime::session): resetting RTIO
[  1015772027us]  WARN(runtime): network error: dropped by socket
[  1018775958us]  WARN(runtime): network error: dropped by socket
[  1021779965us]  WARN(runtime): network error: dropped by socket
[  1024783975us]  WARN(runtime): network error: dropped by socket
[  1028296028us]  WARN(runtime): network error: dropped by socket
[  1031404139us]  WARN(runtime): network error: dropped by socket
[  1034816002us]  WARN(runtime): network error: dropped by socket
[  1037820124us]  WARN(runtime): network error: dropped by socket
[  1040824006us]  WARN(runtime): network error: dropped by socket
[  1043932095us]  WARN(runtime): network error: dropped by socket
[  1046936200us]  WARN(runtime): network error: dropped by socket
[  1049940213us]  WARN(runtime): network error: dropped by socket
[  1050260130us]  INFO(runtime::session): no connection, starting idle kernel
[  1050266104us]  INFO(runtime::session): no idle kernel found
[  1090808488us]  WARN(runtime): network error: truncated packet

Here is the ttl0 output:

Here is a packet dump: kc705.zip

[whitequark]

This is the same issue as #685.

[cjbe]

Any idea when this will be fixed?

This make Artiq-3 unusable, and I am keen to move over to 3 as we are now being annoyingly limited by 'connection reset by peer' errors on Artiq-2

[whitequark]

@cjbe As a matter of fact, with the current master the entire experiment finishes in under 1s.