Wafer thin Bazel layer on top of ORFS

[oharboe]

We are writing a wafer thin Bazel layer on top of ORFS. This is because we want ORFS functionality, but Bazel artifacts and remote builds.

I'm aware that there are ground up Bazel implementations, but I want the ORFS scripts & flow, so that it is easier to report issues to ORFS and that I can use the latest version of ORFS.

This is going well, and when I have something, I will share it, but what would be helpful is to separate the concern of ORFS as a configuration and build flow system, from ORFS's use of make.

Example of what I mean:

• I want to run make synth, this generates some files. Bazel will put these files somewhere and restore them at some later point when downloaded as an artifact
• Now I want to run the rest of the flow, but if I invoke make, then there are no guarantees w.r.t. dates for the files restored as a synth artifact. To make things more interesting, Bazel also uses symlinks; symlinks and make do not work together(looong story).

So what I need is the ability to move the dependency checking to Bazel, but to leave the command invocation to ORFS.

Any ideas on how this might that be achieved?

[maliberty]

I have no Bazel experience but maybe @QuantamHD might?

[oharboe]

This is not a Bazel question, it is a make or ORFS question.

I simply want to unconditionally run the commands that are run when I type make synth with the configuration variables from ORFS, but without having make checking if the files are up to date.

I want to separate the concern of configuration in ORFS from the use of make, effectively.

[QuantamHD]

I've actually thought a lot about this as the author of the ground up implementation. https://github.com/hdl/bazel_rules_hdl

I think the easiest way to achieve this is to make ORFS generate a ninja build file dynamically https://ninja-build.org/. I've done something similar internally at Google, and then run the ninja file under bazel. Let me know if you find something different.

You're going to run into some pain if you try to run abtrirary make scripts under bazel. Namely, that bazel typically wants a hermetic accounting of all input and output files which is hard to achieve with arbitrary make files. Definitely plausible, but some head banging will be required around the edges.

[oharboe]

This approach makes sense to me.

Do you have anything you can share in terms of details of what you have in mind or even code snippets?

[maliberty]

What does "ORFS generate a ninja build file dynamically" mean?

[oharboe]

Here is an idea on how to separate the concern of a executing a stage and checking dependencies.

This requires a bit of copy and paste in the ORFS Makefile, but add targets such as the do-floorplan below.

From Bazel, I can then run:

• make synth do do the first stage. Bootstrapping is special, in that the current make code is fine. If Bazel decides to run it, it will be run unconditionally. Leave this stage as is in the ORFS makefile as it works as-is.
• For floorplan, place, cts, route, finish, generate-abstract we need to create a new "do-xxx" phony target that is executed unconditonally. The steps are copied and pasted from the existing stages.

Although it is annoying to copy and paste the code, it is relatively simple to do. If ORFS agrees to support this, then a CI smoke-test stage could be added to check that it doesn't break too easily.

NOTE!!! Invoking $(MAKE) do-floorplan is problematic because it would cause environment variables to be duplicated, hence I don't see a way around copying and pasting the code.

Floorplan is a single stage here, because it's convenient and it saves on the number of copies of .odb used in artifacts.

.PHONY: do-floorplan
do-floorplan:
	mkdir -p $(RESULTS_DIR) $(LOG_DIR) $(REPORTS_DIR)
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/floorplan.tcl -metrics $(LOG_DIR)/2_1_floorplan.json) 2>&1 | tee $(LOG_DIR)/2_1_floorplan.log
ifndef IS_CHIP
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/io_placement_random.tcl -metrics $(LOG_DIR)/2_2_floorplan_io.json) 2>&1 | tee $(LOG_DIR)/2_2_floorplan_io.log
else
	cp $(RESULTS_DIR)/2_1_floorplan.odb $(RESULTS_DIR)/2_2_floorplan_io.odb
endif
ifeq ($(MACRO_PLACEMENT),)
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/tdms_place.tcl -metrics $(LOG_DIR)/2_3_tdms.json) 2>&1 | tee $(LOG_DIR)/2_3_tdms_place.log
else
	$(info [INFO][FLOW] Using manual macro placement file $(MACRO_PLACEMENT))
	cp $(RESULTS_DIR)/2_2_floorplan_io.odb $(RESULTS_DIR)/2_3_floorplan_tdms.odb
endif
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/macro_place.tcl -metrics $(LOG_DIR)/2_4_mplace.json) 2>&1 | tee $(LOG_DIR)/2_4_mplace.log
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/tapcell.tcl -metrics $(LOG_DIR)/2_5_tapcell.json) 2>&1 | tee $(LOG_DIR)/2_5_tapcell.log
	($(TIME_CMD) $(OPENROAD_CMD) $(SCRIPTS_DIR)/pdn.tcl -metrics $(LOG_DIR)/2_6_pdn.json) 2>&1 | tee $(LOG_DIR)/2_6_pdn.log
	cp $(RESULTS_DIR)/2_6_floorplan_pdn.odb $(RESULTS_DIR)/2_floorplan.odb

[maliberty]

I'm not excited about having copies of the code - there will be inevitable bugs when they get out of sync.

[oharboe]

@QuantamHD I'm curious. What is it about Ninja that makes it easier to use from Bazel than make?

I'm in favour of the best solution to using Bazel on top of ORFS that is acceptable to ORFS in terms of maintenance burden... Whatever that is :-)

[maliberty]

What is the main motivation for Bazel? I'm wondering if there are other ways to achieve it.

[oharboe]

Distributed builds, parallel macro building, artifacts.

[maliberty]

Most steps involve the running of a single script. I've considered switching from passing envars to treating the scripts as jinja templates and building evaluated copies instead. Would that enable you to write the Bazel rules on those scripts as they would be self-contained?

[oharboe]

I only have a vague sense of what you are suggesting.

Can you give a bit more detail and an example on how this would then be used?

@QuantamHD Wanted to have a ninja build script. Can he go from this to a ninja build script?

[maliberty]

today we have

  initialize_floorplan -die_area $::env(DIE_AREA) \
                       -core_area $::env(CORE_AREA) \
                       -site $::env(PLACE_SITE)

which require three envars to be passed in by make. If we generated a platform/design/variant specific script it might have for ng45/ariane133

  initialize_floorplan -die_area "0 0 1500 1500" \
                       -core_area "10 12 1448 1448" \
                       -site FreePDK45_38x28_10R_NP_162NW_34O

Then Bazel could run this script without requiring the envars from make.

[oharboe]

I see. So there might be a "build_scripts" target in the Makefile to set this up and then the existing targets would depend on individual scripts?

[oharboe]

This would make debugging easier too, not relying on environment variables to invoke openroad...

[maliberty]

Would this be enough to enable Bazel?

[oharboe]

I think so, though...

What about synthesis?

In a pinch, I think we can get away with not having this in synthesis, because we can run make synth and have it always run in Bazel as there are no preceding dependencies to worry about.

[oharboe]

Some thoughts on difficulties in injecting environment variables into scripts:

• the scripts/ folder is fairly straightforward. Search and replace $::env()
• can we use the same strategy in the util/ folder?
• the designs themselves have .tcl scripts that use $::env(): (.cfg, .tcl, .sdc and .strategy scripts). How are they handled?
• the platform/ folder uses environment variables. Can search and replace $::env() work there?
• distributed.py generates .tcl files that use $::env()

[maliberty]

today we have

  initialize_floorplan -die_area $::env(DIE_AREA) \
                       -core_area $::env(CORE_AREA) \
                       -site $::env(PLACE_SITE)

which require three envars to be passed in by make. If we generated a platform/design/variant specific script it might have for ng45/ariane133

  initialize_floorplan -die_area "0 0 1500 1500" \
                       -core_area "10 12 1448 1448" \
                       -site FreePDK45_38x28_10R_NP_162NW_34O

Then Bazel could run this script without requiring the envars from make.

@QuantamHD would this be useful to you in terms of keeping up to date with flow changes?

[oharboe]

@maliberty Another idea add a target to generate vars.sh, which sets up environment variables. This will make it possible to invoke openroad and with a .tcl script directly, without having to go via make. All the dependencies have to be expressed in Bazel carefully, but that's what we want anyway...

See #1163 for a draft pull request as to what I have in mind.

[maliberty]

We could easily move the if conditions into the OR script and have OR just start up, invoke cp from tcl, and quit. The overhead would be a lot less than duplicating the code in the Makefile.

[oharboe]

Still.... looots of copy and paste that has to be maintained for these stages.... Exact names of scripts, log files, etc.

[maliberty]

It is a lot of copy-paste in the Makefile as proposed.

[oharboe]

Not a proposal, just the best we have come up with so far, in some ways... I'm hoping for something better.

What I'm saying is that if there is no way around copying and pasting, better to have it in one place, close together and under CI.

[oharboe]

@maliberty Here's what copying in files look like in Tcl... Ran into an interesting snag.... #1169

[oharboe]

@maliberty @QuantamHD With #1175, a step can be executed unconditonally without checking dependencies.

Example:

make do-2_1_floorplan.odb

Obviously dependencies have to be listed in Bazel. It is at least possible to list the dependencies from the makefile.

make -p __phony__ | grep -E '^([^#.\t][^:]*)\:' | grep 2_1_floorplan.odb
make: *** No rule to make target '__phony__'.  Stop.
[deleted]
results/asap7/mock-array/base/2_1_floorplan.odb: results/asap7/mock-array/base/1_synth.v results/asap7/mock-array/base/1_synth.sdc platforms/asap7/lef/asap7_tech_1x_201209.lef platforms/asap7/lef/asap7sc7p5t_28_R_1x_220121a.lef results/asap7/mock-array_Element/base/Element.lef

[oharboe]

ORFS now separates the action and checking dependencies and doing the action.

make floorplan checks dependencies and executes steps out of date, make do-floorplan executes the floorplan steps without checking dependencies.

[oharboe]

For anyone googling, work continues here... hdl/bazel_rules_hdl#225