Redefine memory layout in a future-compatible way

[DL8]

Memory address space should be defined such that region and CORE_ID will be in the MSBs. This definition fits all possible memory sizes without the need to redefine and it is easier for software to work with a contiguous memory space.

Possible structure:

31:30 (region) | 29:22 (CORE_ID) | 21:0 (memory offset) |

Notes:

1. The order between region doesn't have to be as described (e.g. replacing between region and CORE_ID is acceptable)
2. The number of bits in CORE_ID should be discussed (balance between number of cores and possible amount of memory)
3. This proposed layout assumes I and D memories are of equal size and are a power-of-2. Removing this requirement may slightly complicate software interface

New CRs for software:

1. MEMORY_OFFSET_BITS - number of valid bits in the memory offset (e.g. 12 in case of I+D==4K)
2. CORE_ID_BITS - number of bits in CORE_ID (can be hardcoded in current releases)

[amichai-bd]

Are there "standard" addresses that are reserved for that we should take into consideration?
For example the global variables?
Are there standard reserved memory spaces we should try to avoid?

using the MSB [31:30] might be ok - but I'm not sure if it has other effects and impacts I'm not aware of.
Maybe we should keep them reserve and move the "region" bits down to [21:20]?

[DL8]

Not really: the hardware should provide the memory space and it's up to software to partition it.

Regarding bits [31:30], I think it's better to use them if there are no problems and not waste precious address bits. One concern I can think of is how it will work with 64 bits, but that's far away.

Regarding the actual layout, it may be better to put CORE_ID in the MSBs because it's more intuitive, but on the other hand it may be more difficult to extend when we add more cores.

[amichai-bd]

This makes me think maybe we should return to the original Address core location.

This is my rationale:

1. The "local memory" that is located physically next to the core (direct access without ring, "0" latency)
2. Having "local offset" bits [13:0] + [15:14] region is the maximum expected "on-die" local memory for each core. 15/16 bits D_MEM+I_MEM -> aka 64 KB of memory. that is the maximum I can imagine we may have. (in many-core architecture this may be close to 2-8KB)
3. This way when we want to add more memory space - it won't be the cores "local memory" it will be a "shared" bigger memory on the Fabric (similar to Cache L3 -> but there is no HW cache-control - its all done using SW)
4. having the bits [23:16] as core ID and bits [31:24] as Reserved will allow us to add more cores in the future (more than 256) and allocate as much "non-local memory" (on-die like cache L3 or even external DDR memory)

Having the core/CR region bits in the MSB - will make a lot of memory interleave which in my opinion will be confusing.

[DL8]

If we want to add more cores, a layout like this is what I have in mind:
REGION [31:30] | CORE_ID [29:29-C+1] | reserved | memory offset [M:0] |
Where:

• C - number of bits in CORE_ID (currently 8 => 29:22)
• M - number of bits in memory offset
  M and C should be exposed to software via CR. They may be hardwired, of course.

Regarding shared memory, we have two bits of region but only 3 are occupied. Perhaps we can take the reserved region type and use it for shared memory access (let's call it SHARED_MEM). If I understand correctly, CORE_ID is irrelevant when we access shared memory, so we can use it as address offset bits to have a bigger address space:
[31:30] == SHARED_MEM | TYPE [29] | reserved | shared memory offset [SH_M:0]

SH_M is the number of bits in shared memory offset and should also be exposed via CR.

The TYPE bit is used to indicate whether it is memory or MMIO space. This is an optional addition if we want to have shared CR space.

[amichai-bd]

This is what I had in mind:
(my origin proposal with some tweaks)
Each core has “continues” memory Space – very simple to take a core as an Embedded HW and use it as is without spreading it to the entire memory space.

1. Having shared memory outside the core region can be distinguished with a bit higher from MSB. (similar to your latest proposal )
2. The “reserved” memory (bits [15:14]==10) may be the second layer of cores memory in the “fabric” – may be used as cores local memory but not within the cores “local-memory” in the “lotr_tile”. Accessing this memory will have a latency penalty (which we don’t have in the D_MEM region).
   

I see you as the SW expert – so if you think my layout is less scalable – and your latest suggestion makes more sense from the SW point of view – I’m willing to “fold” and agree with you.
But let's talk about it.

I set us a meeting for 18/5/2021