new build system

[foonlyboy]

Hello everyone at simh

I'd like to suggest a new build system.
This could improve the build times a lot.

With the current makefile,
every rebuild of simh takes a lot of time.

I think this is because, every .c file
is compiled again and again for each executable.

But I do not want to touch the makefile.
I want to introduce a second road.

I looked into meson.
That looked good to me.
I'd like to give that a try.

The meson.conf should look like:
executable('vax', [vax.c, simh.c, simh-scsi.c, ...])

• can we generate a meson.conf from the legacy makefile dependencies?
• gmake -d | parse_to_meson_conf

The old makefile was build on the assumption,
that the old compilers needed to see all code for proper inlining.
This is no longer true.

I could even imagine, that we make every module a seperate object,
and let the lto (link time optimization) do the job.

// big fun
Let us build the fastest vax of all times

[markpizz]

The old makefile was build on the assumption, that the old compilers needed to see all code for proper inlining.

Not true. For the most part, you really can't compile the common pieces of all the simulators only one time since each of the common components builds differently with the changing set of -D values they are built with.

Most folks are only interested on one or at most a couple of simulators, so building everything is a waste anyway. Building just your personally interesting simulators merely is done by mentioning the names of the interesting simulators on the command line.

Apart from this, someone who is actively considering changes to their favorite simulator now has the option of only building the files that they are messing with during their development a debug process. This is achieved by either invoking make with BUILD_SEPARATE=1 on the command line, or having an exported environment variable defined called BUILD_SEPARATE with the value of 1.

[foonlyboy]

Hi Mark, nice to get in contact I'm looking forward to talk with you. Obviously, I was completely wrong. Best regards, ~eike

…

On Mon, Jun 19, 2023 at 12:51 AM Mark Pizzolato ***@***.***> wrote: The old makefile was build on the assumption, that the old compilers needed to see all code for proper inlining. Not true. For the most part, you really can't compile the common pieces of all the simulators only one time since each of the common components builds differently with the changing set of -D values they are built with. Most folks are only interested on one or at most a couple of simulators, so building everything is a waste anyway. Building just your personally interesting simulators merely is done by mentioning the names of the interesting simulators on the command line. Apart from this, someone who is actively considering changes to their favorite simulator now has the option of only building the files that they are messing with during their development a debug process. This is achieved by either invoking make with BUILD_SEPARATE=1 on the command line, or having an exported environment variable defined called BUILD_SEPARATE with the value of 1. — Reply to this email directly, view it on GitHub <#1177 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKLJCHQOJM4NZ4VAID2VOMTXL6A75ANCNFSM6AAAAAAZLEMCJY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[larsbrinkhoff]

Hello @foonlyboy,

I feel compelled to ask: are you interested in Foonly, the company, or the computers?

Best regards,
Lars

[eli-schwartz]

With the current makefile, every rebuild of simh takes a lot of time.

I think this is because, every .c file is compiled again and again for each executable.

That's a project choice which isn't really tied to the build system (or shouldn't be). e.g. Makefiles that build several binaries from a common set of source files typically do so with pattern rules that create foo.c -> foo.o, every binary depends on foo.o so it only gets built once.

In some cases, binaries may depend on e.g. foo.c -> progname-foo.o -> progname.exe and that's usually quite deliberate.

The old makefile was build on the assumption,
that the old compilers needed to see all code for proper inlining.

Doubtful. A compiler could only do this by using unity builds, or amalgamations. This usually requires significant ongoing maintenance investment to prevent name clashes once the entire project becomes a single Translation Unit, plus it leaks macro definitions everywhere. Also unity builds kill incremental builds entirely.

...

If every rebuild takes a lot of time, that sounds like a caching issue more than anything else. Meson is definitely one way to solve caching issues, but there are other solutions too.

(Meson does this by getting a very good view of the build graph, separating configuration from building, and using ninja to enforce rebuilding on changed command lines, rather than e.g. rebuilding every time or having objects depend directly on the Makefile, which are, sadly, common tactics for build systems that use something which is not Ninja. There are some highly custom approaches to tracking command lines with pure make, e.g. the linux kernel.)

[markpizz]

With the current makefile, every rebuild of simh takes a lot of time.

I think this is because, every .c file is compiled again and again for each executable.

That's a project choice which isn't really tied to the build system (or shouldn't be). e.g. Makefiles that build several binaries from a common set of source files typically do so with pattern rules that create foo.c -> foo.o, every binary depends on foo.o so it only gets built once.

In some cases, binaries may depend on e.g. foo.c -> progname-foo.o -> progname.exe and that's usually quite deliberate.

It isn't merely a project choice. The files which are common to all simulators produce different object results based on the -D defines passed in at compile time which are different for each simulator.

The old makefile was build on the assumption,
that the old compilers needed to see all code for proper inlining.

Doubtful. A compiler could only do this by using unity builds, or amalgamations. This usually requires significant ongoing maintenance investment to prevent name clashes once the entire project becomes a single Translation Unit, plus it leaks macro definitions everywhere. Also unity builds kill incremental builds entirely.

Inlining has nothing to do with building each simulator with all of the common source modules. See above.

...

If every rebuild takes a lot of time, that sounds like a caching issue more than anything else. Meson is definitely one way to solve caching issues, but there are other solutions too.

(Meson does this by getting a very good view of the build graph, separating configuration from building, and using ninja to enforce rebuilding on changed command lines, rather than e.g. rebuilding every time or having objects depend directly on the Makefile, which are, sadly, common tactics for build systems that use something which is not Ninja. There are some highly custom approaches to tracking command lines with pure make, e.g. the linux kernel.)

An alternate build system wouldn't solve the problem!

In general, given that there are 78 simulators that can be produced by the source code in this project, and if someone chooses to build ALL of them, then that may take significant time. However, it is very rare that any one user actually cares about more than 1 or possibly a handful of simulators. Therefore, building 78 when all they really care about is one, is a user choice and the long built time is a consequence of their choice.

[eli-schwartz]

It isn't merely a project choice. The files which are common to all simulators produce different object results based on the -D defines passed in at compile time which are different for each simulator.

That's exactly what I said, yes. It's a project choice based on how the project has chosen to lay out its source code.

Inlining has nothing to do with building each simulator with all of the common source modules. See above.

And... I'm not the person who mentioned inlining?

I explained why simh's Makefile has nothing to do with inlining, in fact. I explained this by demonstrating how real inlining would require functionality (such as unity builds) which is not present in "build each source file with different target-specific defines".

However, it is very rare that any one user actually cares about more than 1 or possibly a handful of simulators. Therefore, building 78 when all they really care about is one, is a user choice and the long built time is a consequence of their choice.

Well I really have no idea. The reporter mentioned rebuilds, not builds. If one is having long build times on a rebuild there could indeed be caching issues. There's a variety of ways to make re builds faster, none of which will help first-time builds. Building fewer targets will of course help with first time builds as well.

[markpizz]

It isn't merely a project choice. The files which are common to all simulators produce different object results based on the -D defines passed in at compile time which are different for each simulator.

That's exactly what I said, yes. It's a project choice based on how the project has chosen to lay out its source code.

OK, that's true. However, it was a project choice made some 25 years ago, so it's not like there's much choice about this being made recently.

However, it is very rare that any one user actually cares about more than 1 or possibly a handful of simulators. Therefore, building 78 when all they really care about is one, is a user choice and the long built time is a consequence of their choice.

Well I really have no idea. The reporter mentioned rebuilds, not builds. If one is having long build times on a rebuild there could indeed be caching issues. There's a variety of ways to make re builds faster, none of which will help first-time builds. Building fewer targets will of course help with first time builds as well.

There are several things here:

1. The makefile, by default, compiles and links all source in a single compile command for a given target, so build or rebuild is the same thing, no intermediate files are saved between build when compiling this way.
2. The VAST MAJORITY of users merely (like mentioned above about 1 or at most a handful of simulators) are only users of the simulators and as such just pick up the current repository state and build the one or handful they care about. Generally these folks very rarely update anything locally with any frequency so they're working with changes across everything anyway.
3. The VAST MINORITY (a handful of folks at most) are actively working on simulators or shared code. Someone doing this now has the option of only building changed modules if make is invoked with BUILD_SEPARATE=1 (or an exported environment variable is set to this). This report was @foonlyboy's first comment here, so I'm guessing he isn't in the group of developers....

If he wants to use a different build system to build only changed modules, he's free to do that, but that would merely be a different way to solve an already solved problem.