Tilelink support / SoC negociation framework

[Dolu1990]

Hi,

I'm working on getting Tilelink support in SpinalHDL :
https://github.com/SpinalHDL/SpinalHDL/tree/tilelink/lib/src/main/scala/spinal/lib/bus/tilelink

The idea is to have a negociation framework to solve the parameters, but not in the same way as the rocket Diplomacy. Mostly it would be using the Handle / spinal.core.fiber framework to allow different elaboration thread to "solve" the design.

Here is an example of the current usage API :
https://github.com/SpinalHDL/SpinalHDL/blob/tilelink/lib/src/test/scala/spinal/lib/bus/tilelink/InterconnectTester.scala#L38

Let's me know if you are interrested into using / constributing to the feature ^^

[Dolu1990]

Here is a more concret example, with a fictive little SoC with two clock domain :

    val slowCd = ClockDomain.external("slow")
    val fastCd = ClockDomain.external("fast")
    implicit val interconnect = new Interconnect()

    val system = fastCd on new Area {
      val cpu0 = new CPU()
      val cpu1 = new CPU()

      val busA = interconnect.createNode()
      busA << cpu0.node
      busA << cpu1.node
    }
    val peripheral = slowCd on new Area {
      val peripheralBus = interconnect.createNode()
      peripheralBus at(0x10000000, 16 MiB) of system.busA

      val adapter = new Adapter()
      adapter.input << peripheralBus

      val uart0 = new UART()
      uart0.node at(0x100) of adapter.output

      val uart1 = new UART()
      uart1.node at(0x200) of adapter.output
    }

[kleinai]

I've thought about trying this for AXI switches or at least IDs. If you can figure out one way maybe we can make it generic or backport it to other buses?

[Dolu1990]

I think currently all the common feature are here. (on tilelink branch, not everything is in dev)

1. Multithreaded elaboration

  import spinal.core._
  import spinal.core.fiber._
  SpinalVerilog{new Component{
    //Fork a thread 
    val b1 = hardFork on new Area{
      println("Build 1")
      println("b2.miaou = " + b2.miaou) //Will wait until the b2 thread completed
      println("Build 1 Done")
    }
    //Fork another thread
    val b2 = hardFork on new Area{
      println("Build 2")
      val miaou = 42
      println("Build 2 Done")
    }
  }}

=>

Build 1
Build 2
Build 2 Done
b2.miaou = 42
Build 1 Done

2. Handle, which can store a value and be waited on.

    val miaou = Handle[Int]
    val b1 = hardFork on new Area{
      println("Build 1")
      println("miaou = " + miaou.get) //Will wait until the miaou handle is loaded
      println("Build 1 Done")
    }
    val b2 = hardFork on new Area{
      println("Build 2")
      miaou.load(42)
      println("Build 2 Done")
    }

Note that b1 and b2 are typed as Handle[...] and loaded with the given Area in the example.

Build 1
Build 2
Build 2 Done
miaou = 42
Build 1 Done

3. Having elaboration phases (Like NaxRiscv) in order to ensure everybody can run stuff before another set of things.
   Here, there is two phase, "setup" which alows everybody to do something first, then, when all those things, the second phase "build" will start.

  SpinalVerilog{new Component{
    //For a thread which will run in build phase (after all previous phases are done)
    val b1 = Elab build new Area{
      println("Build 1")
      println("b2.miaou = " + b2.miaou) //Will wait until the b2 thread completed
      println("Build 1 Done")
    }
    val b2 = Elab build new Area{
      println("Build 2")
      val miaou = 42
      println("Build 2 Done")
    }
    //For a thread which will run in setup phase, that phase is before "build" phase
    val s1 = Elab setup new Area{
      println("Setup 1")
      println("Setup 1  Done")
    }
    val s2 = Elab setup new Area{
      println("Setup 2")
      println("Setup 2 Done")
    }
  }}

=>

Setup 1
Setup 1  Done
Setup 2
Setup 2 Done
Build 1
Build 2
Build 2 Done
b2.miaou = 42
Build 1 Done

4. Lock

  SpinalVerilog{new Component{
    val lock = Lock()
    val b1 = Elab build new Area{
      println("Build 1")
      lock.await() //Will be blocked until b2 release the lock (see val s1)
      println("Build 1 Done")
    }
    val b2 = Elab build new Area{
      println("Build 2")
      lock.release()
      println("Build 2 Done")
    }
    val s1 = Elab setup new Area{
      println("Setup 1")
      lock.retain() //Let's lock the lock to allow b2 running stuff before b1
      println("Setup 1  Done")
    }
  }}

Setup 1
Setup 1  Done
Build 1
Build 2
Build 2 Done
Build 1 Done

I would say that's all we need to have a scalable negociation framework.
Then we can implement different tools as an AXI interconnect, Tilelink interconnect indepdently without having to worry much about interpolability, as a simple TileLink to Axi interconnect bridge should be able to carry on the negociations by himself.

[Dolu1990]

Here is some progress (it generate the hardware) of a simple pseudo SoC where master / slave buses are toplevel io

new Component{
      implicit val interconnect = new Interconnect()
      
      //A fictive CPU which has 2 memory bus, one from the data cache and one for peripheral accesses
      val cpu = new Area{
        val main = simpleMaster(coherentOnly)
        val io = simpleMaster(readWrite)
      }

      //Will manage memory coherency
      val hub = new CoherencyHubIntegrator()
      val p0 = hub.createPort()
      p0 << cpu.main.node
      p0 << cpu.io.node

      //Define the main memory of the SoC (ex : DDR)
      val memory = simpleSlave(20, 32)
      memory.node.addTag(PMA.MAIN)
      memory.node at 0x80000000l of hub.memGet
      memory.node at 0x80000000l of hub.memPut

      //Define all the peripherals / low performance stuff, ex uart, scratch ram, rom, ..
      val peripheral = new Area{
        val bus = interconnect.createNode()
        bus at 0x20000000 of hub.memGet
        bus at 0x20000000 of hub.memPut

        val gpio = simpleSlave(12, 32)
        val uart = simpleSlave(12, 32)
        val spi = simpleSlave(12, 32)
        val memory = simpleSlave(16, 32)
        val rom = simpleReadOnlySlave(12, 32)

        memory.node.addTag(PMA.MAIN)
        rom.node.addTag(PMA.MAIN)

        gpio.node at 0x1000 of bus
        uart.node at 0x2000 of bus
        spi.node at 0x3000 of bus
        memory.node at 0x10000 of bus
        rom.node at 0x20000 of bus
      }

      //Will analyse the access capabilities of the CPU buses
      Elab check new Area{
        val mainSupport = MemoryConnection.getSupportedTransfers(cpu.main.node)
        println("cpu.main.node can access : ")
        println(mainSupport.map("- " + _).mkString("\n"))
        val ioSupport = MemoryConnection.getSupportedTransfers(cpu.io.node)
        println("cpu.io.node can access : ")
        println(ioSupport.map("- " + _).mkString("\n"))
      }
    }

The elab check stuff will print out :

cpu.main.node can access : 
- (toplevel/memory_node 80000000 100000,TB)
- (toplevel/peripheral_memory_node 20010000 10000,TB)
- (toplevel/peripheral_rom_node 20020000 1000,B)
cpu.io.node can access : 
- (toplevel/memory_node 80000000 100000,GFP)
- (toplevel/peripheral_gpio_node 20001000 1000,GFP)
- (toplevel/peripheral_uart_node 20002000 1000,GFP)
- (toplevel/peripheral_spi_node 20003000 1000,GFP)
- (toplevel/peripheral_memory_node 20010000 10000,GFP)
- (toplevel/peripheral_rom_node 20020000 1000,G)

So it can track down for each master, exactly which slave can be accessed, and which kind of access can be done.
(G = get, F = putFull, P = putPartial, B = aquireBranch, T = aquireTrunc)

[Dolu1990]

Funny update, there is no more Interconnect class which centralize stuff, things are now completly distributed :

new Component{
      val m0, m1 = simpleMaster(readWrite)
      val s0, s1 = simpleSlave(8)

      val b0 = InterconnectNode()

      b0 << m0.node
      b0 << m1.node

      s0.node at 0x200 of b0
      s1.node at 0x400 of b0    
}

[Readon]

It's really a cool feature for Tilelink, is that hard to have other protocol support the feature the same as this?

[Dolu1990]

I think it isn't hard. The aproache so far is very different from Chisel diplomacy, in a way which i hope is more easily understandable.

Mainly a bus instance is a Node, and you can add connections between nodes.
Each node and each connections has its own elaboration thread.

A Node is mostly :

class Node() extends Area with SpinalTagReady with SpinalTag {
  // Some attributes
  val bus = Handle[Bus]()
  val ups = ArrayBuffer[InterconnectConnection]()
  val downs = ArrayBuffer[InterconnectConnection]()
  val clockDomain = ClockDomain.currentHandle

  //Some negociation handles
  val m2s = new Area{
    val proposed = Handle[M2sSupport]()
    val supported = Handle[M2sSupport]()
    val parameters = Handle[M2sParameters]()
  }
  val s2m = new Area{
    val proposed = Handle[S2mSupport]()
    val supported = Handle[S2mSupport]()
    val parameters = Handle[S2mParameters]()
  }

  //A elaboration thread to negociate parameters in functions of the connections, and generate the arbiter / decoder 
  val thread = Elab build new Area { 
     ...
  }
}  

While a connection is :

class InterconnectConnection(val m : Node, val s : Node) extends Area {
  m.downs += this
  s.ups += this
 
 //Addres mapping parameters
  val mapping = new Area{
    var addressSpec = Option.empty[BigInt]
    var mappingSpec = Option.empty[SizeMapping]
    var defaultSpec = Option.empty[Unit]
    val value = Handle[Seq[SizeMapping]]
  }

  //Negociation parameters
  val decoder, arbiter = new Area{
    val bus = Handle[Bus]()
    val m2s = new Area{
      val parameters = Handle[M2sParameters]()
    }
    val s2m = new Area{
      val parameters = Handle[S2mParameters]()
    }
  }
  //A elaboration thread to negociate stuff and generate the connection hardware (ex width adapter, cross clock, ...)
  val thread = Elab build new Area { 
     ...
  }
}  

[Dolu1990]

Just did some cleaning :

SpinalHDL/lib/src/main/scala/spinal/lib/bus/tilelink/crossbar/Node.scala

Line 24 in e139a3c

class Node() extends Area with SpinalTagReady with SpinalTag {

SpinalHDL/lib/src/main/scala/spinal/lib/bus/tilelink/crossbar/Connection.scala

Line 12 in e139a3c

class Connection(val m : Node, val s : Node) extends Area {

[kleinai]

Weird request, could there be an option, or something other hack, to move the interconnect logic into a component?
Areas are nice, and while the code doesn't need components, it would help with things like floor planning in EDA tools. Personally, I use the GUI to spot check synthesis or insert debug cores.

[Dolu1990]

Ahhh like moving every part of the interconnect into a single component ? I think it could be possible yes.

[andreasWallner]

Looks quite nice!
Would this also work if it's not defined in a single file, i.e. if the peripherals were in a subcomponent? (doesn't make much sense for the specific example, but looking at bigger SoC with possibly multiple layers, clock switching logic etc. this might make sense.

I assume that TileLink is currenly hardcoded?

[Dolu1990]

i.e. if the peripherals were in a subcomponent

Currently, crossing to multiple component isn't supported, but i think that would be fesable.

I assume that TileLink is currenly hardcoded?

It isn't realy hardcoded. The idea is more to have a distributed system.
So the framework is about having elaboration threads (Elab build xxx) + Handle + a few set of neutral SpinalTag.

If for instance you want to add AXI in the mix, what you need is to implement a new axi.Node / axi.Connection in a standalone way, and if you want to connect tilelink.Node to axi.Node, you just need to implement a Bridge which will convert the negociation skim + add the hardware.
Here is an example of tilelink bridge, which will filter the trafic from up to down , only allowing the trafic with proper address / opcode.

[Dolu1990]

Those developments will be presented in the FSiC 2023 (10-12 july)

Here are the slides :
https://github.com/SpinalHDL/SpinalDoc/blob/master/presentation/en/fsic_100723/fsic_100723.pdf

Let's me know if you have any question / comment ^^

[kleinai]

After looking at your informative slides, some future features which might be fun: automatic address allocation, and whole bus register map generation (RegIf and introspection).

[jtredux]

Hi - this looks really interesting, thanks for all your hard work!

My brain is still stuck in the world of traditional RTL flows, and I'm a bit unsure how would I go about creating a single tilelink master component (e.g. a CPU) so that I can stick it into the FPGA tools to check timing / area etc. My first guess was to create a wrapper level with a dummy slave component as elaboration obviously gets stuck if you try and elaborate only one end of the bus! However, it would seem that I need all my tilelink Fibres to be defined within Areas rather than Components to allow them to communicate during elaboration. Do I have to abandon my traditional component-hierarchy if I want to embrace this tilelink flow?

[Dolu1990]

Hi,

Did you spoted the doc ?
https://spinalhdl.github.io/SpinalDoc-RTD/master/SpinalHDL/Libraries/Bus/tilelink/tilelink_fabric.html

It isn't plenty, so, let's me know if it miss something :)

However, it would seem that I need all my tilelink Fibres to be defined within Areas rather than Components to allow them to communicate during elaboration.

For now, yes that's right.
The idea is that you can still define your individual component in a traditional way, and then you create that fiber wrapper which will feed the parameters to it.
All the negociation being handled in the fiber area stuff.

Overall it would be possible to bring the fiber stuff into sub component, the issue is just that currently the tilelink interconnect assume all the Node are in the same component. Upgrading that to support cross component would probably not be that hard, i think. Just that so far i didn't had the need, so it wasn't take care of (that' often how things move forward XD)

[jtredux]

Did you spoted the doc ?
https://spinalhdl.github.io/SpinalDoc-RTD/master/SpinalHDL/Libraries/Bus/tilelink/tilelink_fabric.html

Hi, yes thanks, I had a read of the docs and watched both the video links yesterday :)

Overall it would be possible to bring the fiber stuff into sub component, the issue is just that currently the tilelink interconnect >assume all the Node are in the same component. Upgrading that to support cross component would probably not be that hard, i >think. Just that so far i didn't had the need, so it wasn't take care of (that' often how things move forward XD)

Fair enough! I guess I'm just used to building bottom-up and so just gravitate to wrapping up things in component hierarchy.

I think I can get most of what I wanted to do done by building all of the tilelink stuff in a single component with the Fibre API and then manually connecting that to hand-built components with tilelink bus ports on their I/O bundle.

I do however seem to be getting illegal VHDL out of my experimental code - it seems to be missing some naming prefix somewhere:

package pkg_enum is
  type  is (PUT_FULL_DATA,PUT_PARTIAL_DATA,GET,ACQUIRE_BLOCK,ACQUIRE_PERM);
  type _1 is (ACCESS_ACK,ACCESS_ACK_DATA,GRANT,GRANT_DATA,RELEASE_ACK);

  function pkg_mux (sel : std_logic; one : ; zero : ) return ;
  subtype _enc_type is std_logic_vector(2 downto 0);
  constant _enc_PUT_FULL_DATA : _enc_type := "000";
  constant _enc_PUT_PARTIAL_DATA : _enc_type := "001";
  constant _enc_GET : _enc_type := "100";
  constant _enc_ACQUIRE_BLOCK : _enc_type := "110";
  constant _enc_ACQUIRE_PERM : _enc_type := "111";

  function pkg_mux (sel : std_logic; one : _1; zero : _1) return _1;
  subtype _1_enc_type is std_logic_vector(2 downto 0);
  constant _1_enc_ACCESS_ACK : _1_enc_type := "000";
  constant _1_enc_ACCESS_ACK_DATA : _1_enc_type := "001";
  constant _1_enc_GRANT : _1_enc_type := "100";
  constant _1_enc_GRANT_DATA : _1_enc_type := "101";
  constant _1_enc_RELEASE_ACK : _1_enc_type := "110";
```

Any idea what I could have forgotten to set?

[Dolu1990]

I do however seem to be getting illegal VHDL out of my experimental code - it seems to be missing some naming prefix somewhere:

Ahhh likely it is because the enum is defined as a val instead of as a object :

SpinalHDL/lib/src/main/scala/spinal/lib/bus/tilelink/Bus.scala

Line 15 in 68b6158

val A = new SpinalEnum{

Could you give a try ?

[jtredux]

Thanks for that - it took me a bit of time to figure out how to get sbt to build and use a custom version of SpinalHDL, but changing those enums to be objects produced sensible VHDL output. BTW there is a similar issue in the coherent tilelink Cache.

--- a/lib/src/main/scala/spinal/lib/bus/tilelink/coherent/Cache.scala
+++ b/lib/src/main/scala/spinal/lib/bus/tilelink/coherent/Cache.scala
@@ -50,11 +50,11 @@ case class CacheParam(var unp : NodeParameters,
 
 
 object Cache{
-  val CtrlOpcode = new SpinalEnum {
+  object CtrlOpcode extends SpinalEnum {
     val ACQUIRE_BLOCK, ACQUIRE_PERM, RELEASE, RELEASE_DATA, PUT_PARTIAL_DATA, PUT_FULL_DATA, GET, EVICT = newElement()
   }
 
-  val ToUpDOpcode = new SpinalEnum {
+  object ToUpDOpcode extends SpinalEnum {
     val NONE, ACCESS_ACK, ACCESS_ACK_DATA, GRANT, GRANT_DATA, RELEASE_ACK = newElement()
   }
 

Do you have any more examples of how to use the tilelink fibre stuff? I've managed to replicate the examples in the docs, but I seem to get unstuck when trying more complex stuff. For example, I tried extending the CpuFibre example to include a bridge from tilelink -> AXI4, but it gets stuck trying to elaborate:

...
 val axiConfig = bus.amba4.axi.Axi4Config(
    addressWidth = 64,
    dataWidth    = 64,
    idWidth      = 8
  )

  // Define the I/O
   val io = new Bundle {
    val axiDown = master(bus.amba4.axi.Axi4(axiConfig))
  }

val cpu = new CpuFiber()
  val ram = new RamFiber()
  val axibr = new fabric.Axi4Bridge()
  ram.up at (0x10000,0x1000) of cpu.down
  axibr.up at (0x20000,0x10000) of cpu.down

  io.axiDown << axibr.down   // <<<<---- THIS assignment seems to cause the elaboration to get stuck

[Dolu1990]

tilelink -> AXI4, but it gets stuck trying to elaborate:

Yes right, mostly the elaboration system is made in a way that no fiber phase start before the main thread is done executing.

So, if you want to interact with stuff generated from one of the fiber phase (axibr.down) you have to fork a new fiber, or at leat a new hardware elaboration thread (to not block the main thread)

So

hardFork(io.axiDown << axibr.down) // the short and kinda hacky way, as it bypass Fiber API

Or : 

val binding = Fiber build new Area{
  io.axiDown << axibr.down 
  ..
}

[jtredux]

Many thanks! Despite building very parallel hardware for a living, my brain is stuck in a world of single-threaded SW thinking!

[Dolu1990]

It is curious when i try on my side the tilelink Opcode.A stuff, it seems fine :

  type A is (PUT_FULL_DATA,PUT_PARTIAL_DATA,GET,ACQUIRE_BLOCK,ACQUIRE_PERM);

Which version of scala do you use ? do you have anything specific to you setup ?

For the Cache stuff the proper fix is to provide a naming space to the parent scope :)

object Cache extends AreaObject{
  val CtrlOpcode = new SpinalEnum {
    val ACQUIRE_BLOCK, ACQUIRE_PERM, RELEASE, RELEASE_DATA, PUT_PARTIAL_DATA, PUT_FULL_DATA, GET, EVICT = newElement()
  }

I will push that fix

[Dolu1990]

what component is actually objecting to the masks

What do you mean by objecting ? i don't understand the meaning.

dev branch

SpinalHDL one ? One thing, maybe five a try the SpinalHDL "composable-exp" branch, it reworked how the error messaging from live-lock is done.

It work a lot better with VexiiRiscv, but for NaxRiscv / Tilelink thing it may not realy improve things.
Basicaly, for it to work well, threads need to specify what handle they will load in advance, this allow to create a graph of dependencies and sort out dead lock loops

[jtredux]

Sorry - when I say 'objecting', I mean 'something' was hitting an assertion on the exact cycle this transaction with the odd masks exited the cache, so my assumption was that some tilelink component had checked them and 'objected'.

I will give the composable-exp branch a go now, thanks!

[jtredux]

Hmm, it would seem that some of my code that would elaborate with the dev or master branches will not elaborate in the composable-exp branch due to fiber live-locks that don't seem to happen on the other branches! It did however help me spot some bugs (e.g. a setIdle() on the b-channel of a tilelink bus that doesn't support s2m transfers).

I think I'm going to switch back to master or dev and see how far I get...

[Dolu1990]

how far behind you were before updating SpinalHDL ? (git hash)
Also, what error do you get ? (copypast)

[jtredux]

Oops - sorry, I worked around the issue, so I don't have those details. I think I was having an issue with elaborating with the cache connected to the Axi4Bridge, I'll see if I can come up with a testcase. For now I'm just using the non-fiber-wrapped version of the Axi4Bridge and it seems to work OK.

I did figure out what that assertion I was getting was about. It's nothing to do with the masks, but the cache seems to think that I'm doing an ACQUIRE_BLOCK on a non-coherent address. I'm not really sure how to set up the coherent range though, I tried:

val cache = new CacheFiber()
    cache.parameter.blockSize = 64
    cache.parameter.cacheWays = 4
    cache.parameter.cacheBytes = 4 * 1024
    cache.parameter.generalSlotCount = slotCount
    cache.parameter.coherentRegion = addr => True

But that didn't seem to help. I seem to be triggering the assertion

   when(!preCtrl.PROBE_REGION){
        assert(!(isValid && preCtrl.ACQUIRE))
        askProbe := False
      }

I've hacked my local copy for to just tie preCtrl.PROBE_REGION high as I have a single coherent region that covers all the address space.