question about mem initialization

[damaoew]

when I using spinalhdl to generate a verilog file specifically for mem simulation, if the array used for initialization is too large (probably width * depth exceeding 240MB), it will cause the following suspected JVM memory leakage problem. Is there a solution to this phenomenon? Alternatively, initialize the mem using another method.

code：

import spinal.core._
import spinal.lib._
import scala.collection.mutable.ArrayBuffer
import java.lang.Runtime
class test28 extends Component{
val init_data_seq_test = new ArrayBufferBigInt
(0 until 80000000).map( idx => {
init_data_seq_test.append(0)
})
val mem_test = new Mem(Bits(32bits),80000000)
mem_test.initBigInt(init_data_seq_test)
}
object test28 extends App{
SpinalVerilog(new test28())
}

log:

[Warning] Elaboration failed (0 error).
Spinal will restart with scala trace to help you to find the problem.

[Progress] at 30.452 : Elaborate components
[Progress] at 35.448 : Checks and transforms
[Progress] at 35.450 : Generate Verilog

Exception in thread "main" java.lang.OutOfMemoryError
at java.base/java.lang.AbstractStringBuilder.hugeCapacity(AbstractStringBuilder.java:214)
at java.base/java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:206)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:173)

[Dolu1990]

Hi,

The mem initisalisation is currently designed to init a relatively small Mem with data (not 320 MB :D)
For the simulation, are you using SpinalSim ? Or you have your own VHDL / Verilog testbench ?
Because in that case, there is some API there to initialize a memory at simulation time instead than from the netlist itself.

Else, i guess a feature could be added to fill a mem with zero in the netlist, but still, that isn't synthetisable, so i'm not sure that would be the good aproach.

[damaoew]

I said before may not be detailed enough. I need to use spinalhdl to generate a verilog simulation file based on different parameters, and provide it to users as a substitute for the actual design，it isn't synthetisable. This requires our simulation mem to be able to read initialization data from json or txt files. this function similar to readmemb in Verilog. We first store the initialization data read from the json file into an array. next, using it for mem initialization. But, we discovered the problem and conducted an experiment using the above code

[Dolu1990]

Does the json/txt file cover the whole memory ?
You are reading those json/txt file in SpinalHDL / Scala iself right ?

[damaoew]

Yes，the json cover the whole memory. SpinalHDL code read the json file, then generate verilog files for customer simulation use

[Dolu1990]

One thing which will help a bit is to not use initBigInt but directly going to the low level api :

    val mem = Mem.fill(1024*1024*80)(Bits(32 bits))

    val values = new Array[BigInt](mem.wordCount)
    for(i <- 0 until mem.wordCount){
      values(i) = BigInt(i.toLong*i)
    }
    mem.initialContent = values

So after testing, there is 3 bottlenecks :

• Mem init is stored via Array[BigInt], which doesn't scale well at all, should be moved to Array[Byte]
• Durring the generation, SpinalHDL store the whole init file in memory
• The init file use readmemb instead of readmemh

Each of those are fixable, but i'm quite under water those days.

Maybe, try the workaround (Array[BigInt]) and to bump up JVM maximum memory.

What is your reported [Runtime] JVM max memory : xxxx when you run SpinalHDL ?

[andreasWallner]

It looks to me like they are not using SpinalSim, so a possible solution would be to allow inference of a ROM from just a size and either create the needed bin files filled with 0's or expect one to be present for simulation. That would remove the need to have that data in memory during generation.
But yes, that also needs changes...

[Dolu1990]

Ahhh right, either a filled with 0, either the user would need to provide an iterator / lambda function, that would also avoid having it in memory.

[Dolu1990]

@andreasWallner Would having the user providing a iterator / landa function (or something like that) sound good ?

[andreasWallner]

I guess either an operator or a lambda sound good, in both cases we could stream the content out.

I was thinking of not writing the file since it sounded easiest, especially if one already has the file from a build. But either of the options above sound better ;-)

[damaoew]

Thank you for your answer. The maximum mem for JVM should be 32GB. In the past, I used spinalhdl directly read initialization data from json files, then generate verilog and corresponding bin files. Now I am trying to embed a module written in Verilog into the spinalhdl code, so that I can use readmemb to complete initialization within this module. The corresponding bin file is also generated separately to avoid being stored in memory during generation. I hope this can solve the problem.

[Dolu1990]

I was thinking of not writing the file since it sounded easiest, especially if one already has the file from a build.

Agree, it would be good aswell.