Subkernels calling subkernels on other destinations, subkernel DDMA, with bidirectional DRTIO routing

[Spaqin]

ARTIQ Pull Request

Description of Changes

This PR is a part of few, which will be built on top of each other:

• Bidirectional DRTIO routing, subkernels calling subkernels, (done)
• DMA and DDMA support for subkernels, (done)
• Free message passing between [sub]kernels, beyond call arguments and returns.

At the moment, subkernels cannot call other subkernels in other destinations, for simplicity. In general, that required adding possibility of bidirectional DRTIO routing, supporting more than typical 'master asks, satellite responds' communication scheme. Subkernels can ask for launching another subkernel at any time in any place, and we would prefer to not to involve the master if possible - only for re-routing if there's no path from the current satellite node downstream to the destination.

Thus, few DRTIO related packets were modified to also include information on the source, and they're routable - a satellite receiving an unsolicited packet, instead of discarding it, may now need to re-route or consume it, depending if the satellite was the intended target. Subkernels and DDMA also now know who called them, so they can return their work to the proper source.

As unsolicited/asynchronous aux packets are generally unstable (#2213), routed or asynchronous packets are not sent without notice. Satellites notify the master or higher ranking satellite that they have some packets with a small packet in the main protocol, async_messages_ready (see gateware changes). The master or higher ranking satellite, upon noticing a flag, will query the downstream device for whatever data it may have. That data can be consumed or re-routed, depending on the destination.

DMA on subkernels is an extension of DDMA. Subkernels themselves have no access to DMA gateware and as such, DDMA is always used, and playback is started by the main (satman/comm) core. Subkernels also have no DRTIO meaning they cannot manipulate RTIO devices on other satellites, so a DMA trace is broken down like it would've been in master, and appropriate slices are carved out and sent to other satellites if necessary, to be played back later.

Tested with 3 Kasli 2.0s, in both star ( sat1 <- master -> sat2 ) and daisychain ( master -> sat1 -> sat2 ) and subkernel experiments below:

Subkernel code

test_subkernel_multdest_sat:

from artiq.experiment import *


class TestSkMultiDestSat(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("led0")
        self.setattr_device("led3")
        self.setattr_device("led6")

    @subkernel(destination=1)
    def simple_self(self) -> TInt32:
        self.core.break_realtime()
        self.core.reset()
        delay(1000*ms)
        self.led3.pulse(1000*ms)
        self.simple_self_too()
        return 15 + subkernel_await(self.simple_self_too)

    @subkernel(destination=2)
    def simple_self_too(self) -> TInt32:
        self.core.break_realtime()
        self.core.reset()
        delay(1000*ms)
        self.led6.pulse(1000*ms)
        return 16

    @kernel
    def run(self):
        subkernel_preload(self.simple_self)
        self.core.reset()
        self.core.break_realtime()
        self.led0.pulse(1000*ms)

        delay(10000*ms)
        self.simple_self()
        val = subkernel_await(self.simple_self)
        print(val)

        self.led3.pulse(1000*ms)

        self.led6.pulse(500*ms)

test_subkernel_multdest_mst.py:

from artiq.experiment import *


class DMAPulses(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("led0")
        self.setattr_device("led3")
        self.setattr_device("led6")

    @subkernel(destination=1)
    def simple_self(self) -> TInt32:
        self.core.break_realtime()
        self.core.reset()
        delay(10*ms)
        self.led3.pulse(1000*ms)
        return 15

    @subkernel(destination=2)
    def simple_self_too(self) -> TInt32:
        self.core.break_realtime()
        self.core.reset()
        delay(10*ms)
        self.led6.pulse(1000*ms)
        return 16

    @kernel
    def run(self):
        subkernel_preload(self.simple_self)
        self.core.reset()
        self.core.break_realtime()
        self.led0.pulse(1000*ms)

        delay(1000*ms)
        self.simple_self()
        val = subkernel_await(self.simple_self)
        print(val)

        self.led3.pulse(1000*ms)

        delay(1000*ms)
        self.simple_self_too()
        val = subkernel_await(self.simple_self_too)
        print(val)

        self.led6.pulse(500*ms)

(D)DMA has been tested with test_ddma_subkernel_mult.py:

from artiq.experiment import *


class DMAPulses(EnvExperiment):
    def build(self):
        self.setattr_device("core")
        self.setattr_device("core_dma")
        self.setattr_device("led3")  # on sat1
        self.setattr_device("led6")  # on sat2

    @subkernel(destination = 1)
    def ddma_test(self) -> TInt32:
        self.core.reset()
        with self.core_dma.record("pulses", enable_ddma=True):
            for i in range(10):
                self.led3.pulse(500*ms)
                self.led6.pulse(500*ms)
        pulses_handle = self.core_dma.get_handle("pulses")
        self.core.break_realtime()
        delay(1000*ms)
        self.core_dma.playback_handle(pulses_handle)
        return 15

    @kernel
    def run(self):
        self.ddma_test()
        val = subkernel_await(self.ddma_test)
        print(val)

Summary of changes:

• Support subkernel messages are sent with proper destination
• Zynq port (at very least, make the changes to DRTIO packets to maintain compatibility) - can be done after this is merge
• More testing in different configurations (daisychain, maybe with 4 satellites)
• Change documentation as necessary (separate PR to keep this one clean)
• (D)DMA support
• (D)DMA Zynq port
• Support for free message passing

Related Issue

Type of Changes

	Type
✓	✨ New feature

Steps (Choose relevant, delete irrelevant before submitting)

All Pull Requests

• Use correct spelling and grammar.

Code Changes

• Test your changes or have someone test them. Mention what was tested and how.
• Add and check docstrings and comments
• Check, test, and update the unittests in /artiq/test/ or gateware simulations in /artiq/gateware/test

Git Logistics

• Split your contribution into logically separate changes (git rebase --interactive). Merge/squash/fixup commits that just fix or amend previous commits. Remove unintended changes & cleanup. See tutorial.
• Write short & meaningful commit messages. Review each commit for messages (git show). Format:

  topic: description. < 50 characters total.
  
  Longer description. < 70 characters per line
  

Licensing

See copyright & licensing for more info.
ARTIQ files that do not contain a license header are copyrighted by M-Labs Limited and are licensed under LGPLv3+.