Perpetual Looping of Autotuner?

[dalyles]

I am trying to do a run on the swerv example circuit from the library nangate45.
What seems to be happening is that there is a supposed infite loop of running the iterations when
I run the command:

python3 distributed.py --design swerv --platform nangate45 --config ../designs/nangate45/swerv/autotuner.json tune

Luckily, the contents in the config.mk, constraint.sdc, and autotuner.json are small enough to display here. If this helps I'm using python version 3.10.9.
All other files in OpenROAD-flow-scripts/flow/designs/nangate45/swerv are left unchanged.

config.mk

constraint.sdc

autotuner.json

Thank you very much in advance.

[vijayank88]

@dalyles
Are you running locally or jobs distributed?

[dalyles]

I'm running it on a remote server. I'm not sure if that completely answers your question.

[vijayank88]

You have to follow instruction from here: https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/tools/AutoTuner/README.md to distribute jobs for complex designs. Otherwise it may run for log time.

[dalyles]

@vijayank88

Do you have a tutorial or a walkthrough on how to run this with Autotuner?
Thanks

[dalyles]

Another thing I want to add is that it looks like during the supposed infinite loop, each iteration is exactly the same.

[vijayank88]

Best parameters found:

2023-06-22 11:00:03,104	INFO tune.py:639 -- Total run time: 10852.13 seconds (10851.97 seconds for the tuning loop).
[INFO TUN-0002] Best parameters found: {'_SDC_CLK_PERIOD': 3.2433794787353936, 'CORE_UTILIZATION': 98, 'CORE_ASPECT_RATIO': 1.288325817455164, 'CORE_MARGIN': 2, 'CELL_PAD_IN_SITES_GLOBAL_PLACEMENT': 0, 'CELL_PAD_IN_SITES_DETAIL_PLACEMENT': 1, '_FR_LAYER_ADJUST': 0.39813722784548444, 'PLACE_DENSITY_LB_ADDON': 0.42029200119725857, '_PINS_DISTANCE': 1, 'CTS_CLUSTER_SIZE': 126, 'CTS_CLUSTER_DIAMETER': 313, '_FR_GR_OVERFLOW': 1}
(save_best pid=10002) [INFO TUN-0003] Best parameters written to /home/vijayan/OPENROAD_FLOW/ORFS_LOCAL/OpenROAD-flow-scripts/flow/logs/nangate45/swerv/test-tune-2023-06-22-07-59-06/autotuner-best-ac039f88.json

[vvbandeira]

@dalyles, what makes you think the iterations are the same? Note that the "Status" section is printed periodically and does not reflect an iteration. Note also that the run that finished, i.e., with the "TERMINATED" status, took over 4h to run. Finally, are you interested in the swerv design or only in testing the AutoTuner? If it is the latter, I would suggest you start with gcd, which has a much smaller runtime.

[dalyles]

Hi @vvbandeira,

For the Autotuner, I'm trying to test it, yes.
However I am using swerv since it has a variety of violations such as
maximum capacitance violations and maximum slew rate violations.
gcd only gives a setup time violation when running with default parameters.

For swerv I'm adding the parameters CAP_MARGIN and SLEW_MARGIN into the autotuner.json file and the constraint.sdc to see if will successfully converge and then run the flow using the best parameters found.

[vvbandeira]

@dalyles, thanks for the info. Was my explanation about the log files clear/useful? Could you expand a little on why you think the experiments are always the same?

[dalyles]

Hi @vvbandeira,
Yes your explanation was helpful. Just to confirm what you were saying, I reran the autotuner and let it run overnight and I saw that progress is being made although it is quite slow being slow. From what I understand, just because the console output periodically displays the status, doesn't mean it is redoing the same iteration. The same iteration is still in progress. I'll take your recommendation that you posted in your other comment.

[vijayank88]

@vvbandeira Trimming autotuner.json variable will cause any issue, because user mention it is running infinitely.

@dalyles can you share OS name/RAM/No.of cores in remote server?

[dalyles]

Thank you very much for this, I'll give this a shot, so I just append these options in my original command?
Sorry I'm a bit of a novice when it comes to these things.

[dalyles]

So I tried using this command:

:~/OpenROAD-flow-scripts/flow/util$ python3 distributed.py --design swerv --platform nangate45 --config ../designs/nangate45/swerv/autotuner.json tune --resources_per_trial 10 --jobs 10

usage: distributed.py [-h] --design <gcd,jpeg,ibex,aes,...> --platform <sky130hd,sky130hs,asap7,...> --config [--experiment ] [--timeout ] [--git_clean] [--git_clone] [--git_clone_args ] [--git_latest] [--git_or_branch ]
[--git_orfs_branch ] [--git_url ] [--build_args ] [--jobs ] [--openroad_threads ] [--server <ip|servername>] [--port ] [-v]
{tune,sweep} ...

distributed.py: error: unrecognized arguments: --jobs 10

It sees to not recognize the argument for jobs but it is fine with accepting resources_per_trial

[vvbandeira]

Global arguments should go before the tune or sweep keywords, i.e., the --jobs -- the other three can go in any order after the tune keyword.

---

I would suggest, based on your HW above, --resources_per_trial 14 --samples 24 --iterations 3. This will run 24 trials, in 3 batches of 8 -- since each trial uses 14 CPUs it can consume all 112 threads of your system (14*8=112) if the 8 parallel traials reach DRT at the same time. Expected runtime for this would be 3 * single run of swerv, in your case from the original post 4h, thus 4h*3 = 12h (overnight :) )

[dalyles]

But why would you want to have a longer runtime? Wouldn't it make more sense to utilize the resources to reduce the runtime?

[vvbandeira]

AutoTuner needs data to improve the results it provides. The higher the number of iterations, the higher the probability of better parameters. A higher number of samples per iteration means more data for AT to use in the next iteration. Choosing the "correct" amount of samples and iterations is an open research question. In short, if everything is well set up, the higher the runtime, the better parameters you will get -- note that this is a very generalized statement; there's an upper bound limit for the tool and the search space (i.e., --config <FILE>.json).