Initial pure D build script #8293

wilzbach · 2018-05-25T12:17:55Z

(inspired by @JinShil's recent dream on the NG about being able to build DMD easily on any platform)

Similarly to the test/run.d, this allows building DMD from source without depending on the Makefile.
At the moment it's a simplistic translation of more or the less the bare minimum of the posix.mak which is needed to build DMD, but the following is intended (not in this PR though):

support all OS (getting rid of the win32/win64.mak files is a major motivation)
allow this script to be used by dub (this should allow to build/expose full DMD via DUB)
target compatibility with the posix.mak Makefile (i.e. all targets that are supported in posix.mak should be supported here)

I opened this PR to avoid duplicated efforts and give others already a sneak-peek at this.
This already compiles DMD on Linux and even supports partial and parallel compilation (almost as good the Make build).

I kept the rule system as simplistic as possible to avoid adding a huge chunk of code that we would have to maintain + it's not really needed here anyhow.
Also the builder script got quite long, because instead of using one text blob for the source files, I used single lines for each file to avoid the merge conflicts that sometimes happen (though I have no strong feeling on this style).

Why such a naive approach and not cmake, ninja, reggae, ...?

There has never been a consensus on which third-party build system should be used.
Using a pure D script avoids any further dependencies and after all, D should be powerful to handle any task!
Also the dependency flow is rather simplistic here. The backend and lexer need to be built before the main DMD built, but they can be built in parallel once the initial dependencies have been generated.

But the Makefile is working so nicely

Well, I guess I'm one of the very few persons who understands the posix.mak (it looks more scary than it is), but I doubt many others do.
Also we there have been numerous complaints about the more or less unmaintained win32.mak and win64.mak files.
For now, like run.d this is an experiment and only intended to slowly replace the Makefiles once it has been thoroughly tested.

But isn't depending on D another dependency?

Nope, D requires a host D compiler since 2015. The Makefile comes with an AUTO_BOOTSTRAP option to automatically download a D compiler. While I added support for this in build.d too (it makes sense when you want to lock the D compiler version to a specific one), it's only of limited help to someone on a fresh machine without a D compiler.
Imho we can expect a D compiler dev to have a working D installation, but if this AUTO_BOOTSTRAP option is convenient for some (I think the Jenkins CI uses it), we could opt to add a small Bash bootstrapping script around it (or keep this part of the Makefile).

What's missing from this PR?

This PR contains everything essential to use the build.d script on x86_64 Linux.
Though there's still much work that needs to be done in future PRs.
Check the TODO header.

What won't be part of this PR?

I don't plan to support Windows nor all targets of posix.mak in this initial PR.

dlang-bot · 2018-05-25T12:17:57Z

Thanks for your pull request, @wilzbach!

Bugzilla references

Your PR doesn't reference any Bugzilla issue.

If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog.

Testing this PR locally

If you don't have a local development environment setup, you can use Digger to test this PR:

dub fetch digger
dub run digger -- build "master + dmd#8293"

JinShil · 2018-05-25T14:47:07Z

src/build.d

+    }
+    Sources sources = {
+        frontend: [
+	        "arraytypes",


Do the source files need to be explicitly enumerated? I envisioned something more like sourceDir.dirEntries("*.d", SpanMode.depth).join(" ")

I tried to stay as close as possible to the Makefile, but in general there's no big reason not search for them though there are a few dead files which are in the codebase, but never used.

JinShil · 2018-05-27T09:41:49Z

src/build.d

+
+Examples:
+
+    ./run.d dmd                                                  # build DMD


I think this should be "build.d" instead of "run.d", but I think you can get that dynamically.

wilzbach

Okay, so while there's still a ton of things that should be done with this script, it's already quite usable on x86_64 Linux (to verify this I added this script to CircleCi).
Also perfect is the enemy of progress and this PR is already quite big.
So I would suggest we use this as a base for future improvements and PR which should then be a lot easier to review and manage.
Also with an existing build.d other people can more easily join this effort.

Everyone ok with starting of with the minimal subset and slowly increasing it?

wilzbach · 2018-06-01T15:01:37Z

.circleci/run.sh

-    codecov)
-        echo "removed - use 'all'"
-        # Fall-through is used to maintain compatibility with the existing PRs
-        ;&


As I was already touching this code, I removed these long-dead targets.

wilzbach · 2018-06-01T15:01:56Z

.circleci/run.sh

+    rm -rf generated # just to be sure
+    # TODO: add support for 32-bit builds
+    ./src/build.d MODEL=64
+    ./generated/linux/release/64/dmd --version | grep -v "dirty"


I will fix the 32-bit build in a follow-up PR.

wilzbach · 2018-06-01T15:02:59Z

src/build.d

+- test on OSX
+- test on Windows
+- allow appending DFLAGS via the environment
+- evaluate whether to use globbing for the file listings


I know it's a long list, but imho it's better to have something as an initial base on which incremental reviews can be done.
Also with an existing build.d other people can more easily join this effort.

wilzbach · 2018-06-01T15:26:33Z

src/build.d

+    auto target = env["G"].buildPath("dmd.conf");
+    auto commandFun = (){
+        conf.toFile(target);
+    }; // defined separately to support older D compilers


Due to a recently DMD bug declaring the function directly doesn't work.
See also:

https://issues.dlang.org/show_bug.cgi?id=18199
#8051

wilzbach · 2018-06-01T15:31:41Z

src/build.d

+
+            case "man":
+                "TODO: man".writeln; // TODO
+                break;


All these targets are nice-to-have (as we eventually want to get rid of the Makefile), but obviously not required for building dmd. I added them here, s.t. adding them doesn't get forgotten.

wilzbach · 2018-06-01T15:32:33Z

src/build.d

+
+            default:
+            case "all":
+                goto dmd;


This is to maintain "compatibility" with the Makefile whose default goal was "all" and is just an alias for "dmd".

wilzbach · 2018-06-01T15:33:18Z

src/build.d

+    env.getDefault("C", d.buildPath("backend"));
+    env.getDefault("TK", d.buildPath("tk"));
+    env.getDefault("ROOT", d.buildPath("root"));
+    env.getDefault("EX", d.buildPath("examples"));


This variable isn't used yet, but it will very likely soon be needed (i.e. when the example goal gets added)

wilzbach · 2018-06-01T15:33:41Z

src/build.d

+    env.getDefault("GIT_HOME", "https://github.com/dlang");
+    env.getDefault("SYSCONFDIR", "/etc");
+    env.getDefault("TMP", "/tmp");
+    env.getDefault("PGO_DIR", srcDir.buildPath("pgo"));


This variable isn't used yet, but once ENABLE_PGO gets added, it should be used.

wilzbach · 2018-06-01T15:35:03Z

src/build.d

+    {
+        env["HOST_DMD_PATH"] = ["which", env["HOST_DMD"]].execute.output.strip;
+        env["HOST_DMD_RUN"] = env["HOST_DMD"];
+    }


Auto-bootstrapping doesn't make so much sense anymore as it did with the Makefile (after all a D compiler is already required to run this script).
However, the use case of looking the compilation to a specific D version still exists.

wilzbach · 2018-06-01T15:39:31Z

src/build.d

+    {
+        // TODO: remove unused file lists
+        string[] frontend, lexer, lexerRoot, root, glue, dmd, backend;
+        string[] backendHeaders, tkHeaders, backendC, tkC, backendObjects;


We could use some globbing here to find all the files which would probably cut down half of the size of this script.
The problem is that the frontend is separated in multiple parts and we would have to do a bit of filtering and thus maybe not save so much.

jacob-carlborg · 2018-06-02T07:40:49Z

Everyone ok with starting of with the minimal subset and slowly increasing it?

Please do.

jacob-carlborg · 2018-06-02T07:45:47Z

Do not work on macOS:

object.Exception@./build.d(1135): dmd/backend/cgelem.c:1253:35: error: '&&' within '||' [-Werror,-Wlogical-op-parentheses]
                    op == OPu8_16 && (e->Eoper == OPand || !(i & ~0xFF)) ||
                    ~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~

Looks like it's missing some flags to compile the C++ code.

jacob-carlborg · 2018-06-02T07:51:21Z

src/build.d

+                "-Wno-implicit-fallthrough",
+            ];
+        else if (env["CXX_KIND"] == "clang++")
+            warnings ~= "-Wno-logical-op-parentheses";


The check for clang++ should be moved to the else branch of env["ENABLE_WARNINGS"] != "0", i.e. in the else after line 503.

JinShil · 2018-06-02T09:31:13Z

Everyone ok with starting of with the minimal subset and slowly increasing it?

Yes, though I would like to see a brief description of the strategy.

wilzbach · 2018-06-02T13:55:15Z

Do not work on macOS:

Yep, sorry but I don't have a macOS machine for testing.
Though I do plan to add this script to auto-tester-test target in the main posix.mak, s.t. we/I can test it for OSX.

Yes, though I would like to see a brief description of the strategy.

Here's how I plan to proceed over the next weeks:

Fix 32-bit build mode
Add different ENABLE modes
Test with LDC as host
OSX + Windows support
Remaining TODO tasks (e.g. allow appending DFLAGS from the environment)
Add other Makefile targets

(2), (4) and (6) might result in multiple PRs.
Is this what you were asking for or did you ask for the strategy of this build script?

The strategy of this script is to emulate what the Makefile is doing, but without a complicated dependency and dependency system.
The "dependency system" used here is rather naive and only parallelizes the build of the backend and lexer (writing a few config files doesn't take much time).
However, it does skip steps when the source files are younger than the target and thus supports partial rebuilds.

All individual dependencies of DMD are defined. They have a target path, sources paths and an optional name. When a dependency is needed either its command or custom commandFunction is executed.
A dependency will be skipped if all targets are older than all sources. The build script is by default part of the sources and thus any change to the build script, will trigger a full rebuild.

The function buildDMD defines the build order of its dependencies.

jacob-carlborg · 2018-06-02T19:26:39Z

Yep, sorry but I don't have a macOS machine for testing.

I haven't tested everything but by fixing the warnings it will now at least build DMD successfully.

JinShil · 2018-06-03T23:42:23Z

src/build.d

+  ./build.d dmd
+
+TODO:
+- add different ENABLE modes


~~What are "ENABLE modes"?~~ Answered my own question

JinShil · 2018-06-04T00:36:23Z

Its appears that we will be bringing up build.d in parallel with the existing makefiles. We are using Circle CI as the CI for build.d. At what point will we replace posix.mak with build.d? After steps 1~3 are completed, can we make the switch?

JinShil · 2018-06-04T00:39:08Z

As much as I really want this to happen, I think we should wait to pull this until @WalterBright is finished porting the backend to D. That process seems to involve a lot of makefile changes, and putting the burden on @WalterBright and reviewers to modify and verify two different builds scripts is going to be a bit of a nuisance.

wilzbach · 2018-06-08T12:35:04Z

At what point will we replace posix.mak with build.d? After steps 1~3 are completed, can we make the switch?

In theory we could already make the switch for e.g. CircleCi (only the enable modes are missing and they are quick to add), though I wanted to wait a few months to make sure there aren't any sneaky bugs in the build script somewhere and as you mentioned yourself it makes sense to wait with adding this build script until the backend conversion is complete.

As much as I really want this to happen, I think we should wait to pull this until @WalterBright is finished porting the backend to

Fair enough, but how about adding the script without adding it to the CIs (for now)?
That would still come with the advantage of being able to make incremental progress on it and expose it for other people who want to play with/test it.

Though of course, we could replace the hard-coded file list with file globbing and thus eliminate the need for manually updating this build script during the backend conversion.
Maybe I try this as a follow-up.

JinShil · 2018-06-08T13:50:14Z

Fair enough, but how about adding the script without adding it to the CIs (for now)?

But we need to test it as we go.

Though of course, we could replace the hard-coded file list with file globbing and thus eliminate the need for manually updating this build script during the backend conversion.

That would be great. If you can pull it off, I'd support adding it to the CI now, but we should be at @WalterBright's beck and call should he run into any issues.

wilzbach · 2018-06-19T19:49:40Z

we could replace the hard-coded file list with file globbing and thus eliminate the need for manually updating this build script during the backend conversion.
That would be great. If you can pull it off

I think I managed to do so.
It now builds with only a minimally hard-coded lists and globs for all other source files.

jacob-carlborg · 2018-06-20T10:46:37Z

src/build.d

+        return "dragonflybsd";
+    else version(Solaris)
+        return "solaris";
+    else version(SunOS)


Is this a predefined version identifier?

jacob-carlborg · 2018-06-20T10:48:55Z

src/build.d

+*/
+auto detectModel()
+{
+    version(Solaris)


Should SunOS be used here as well? Or should detectOS be used instead?

wilzbach · 2018-06-26T03:34:30Z

BTW when the recent tocvdebug.d PR was merged no modifications where required, which is a promising sign that the newly adding globbing does its job.
I addressed @jacob-carlborg's points. Anything else you want to see in this initial version?
(as mentioned there's a plan to extend this step by step over the next weeks).

JinShil

It's a big step in the right direction. Let's move forward. I don't want this to hang around in an unfinished state for years, though. I hope we can keep at it regularly until it's done.

andralex · 2019-10-29T16:34:52Z

src/build.d

+
+void main(string[] args)
+{
+    int jobs = totalCPUs;


This variable is never used. A primary feature of a build system is to facilitate parallel builds. Are there plans to implement such?

It always was (though of course the user should have been able to regulate this with -j):

https://github.com/wilzbach/dmd/blob/e417f0e0dfd3ea117609cd48a9de53695aaaa3ad/src/build.d#L244

This is still the case for master:

dmd/src/build.d

Lines 141 to 142 in 9e877e0

foreach (target; targets.parallel(1))

target();

dmd/src/build.d

Lines 1317 to 1320 in 9e877e0

foreach (dep; deps.parallel(1))

{

dep.run();

}

dlang-bot added the Review:WIP Work In Progress - not ready for review or pulling label May 25, 2018

wilzbach force-pushed the build branch from ae429e8 to c4cf309 Compare May 25, 2018 12:50

JinShil reviewed May 25, 2018

View reviewed changes

JinShil reviewed May 27, 2018

View reviewed changes

wilzbach force-pushed the build branch 11 times, most recently from efbd9f5 to e84b802 Compare June 1, 2018 15:21

wilzbach removed the Review:WIP Work In Progress - not ready for review or pulling label Jun 1, 2018

wilzbach changed the title ~~[WIP] Initial pure D builder~~ Initial pure D builder Jun 1, 2018

wilzbach commented Jun 1, 2018

View reviewed changes

wilzbach changed the title ~~Initial pure D builder~~ Initial pure D build script Jun 1, 2018

wilzbach force-pushed the build branch from e84b802 to d63dcc8 Compare June 1, 2018 15:42

wilzbach commented Jun 1, 2018

View reviewed changes

jacob-carlborg reviewed Jun 2, 2018

View reviewed changes

wilzbach force-pushed the build branch 3 times, most recently from 1f3b4f9 to f906850 Compare June 2, 2018 13:49

JinShil reviewed Jun 3, 2018

View reviewed changes

wilzbach force-pushed the build branch from ca5b1bf to ae61278 Compare June 8, 2018 12:29

wilzbach force-pushed the build branch 4 times, most recently from e2617f0 to 3060159 Compare June 19, 2018 19:39

wilzbach force-pushed the build branch from 3060159 to 0077a2e Compare June 19, 2018 19:59

jacob-carlborg reviewed Jun 20, 2018

View reviewed changes

wilzbach force-pushed the build branch 8 times, most recently from f6dec02 to e417f0e Compare June 26, 2018 02:00

Initial raw builder

e417f0e

JinShil approved these changes Jun 26, 2018

View reviewed changes

jacob-carlborg approved these changes Jun 26, 2018

View reviewed changes

jacob-carlborg added the Merge:auto-merge label Jun 26, 2018

dlang-bot merged commit fb22391 into dlang:master Jun 26, 2018

wilzbach deleted the build branch June 28, 2018 08:27

andralex reviewed Oct 29, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial pure D build script #8293

Initial pure D build script #8293

wilzbach commented May 25, 2018 •

edited

Loading

dlang-bot commented May 25, 2018

JinShil May 25, 2018

wilzbach May 25, 2018

JinShil May 27, 2018 •

edited

Loading

wilzbach left a comment

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

wilzbach Jun 1, 2018

jacob-carlborg commented Jun 2, 2018

jacob-carlborg commented Jun 2, 2018

jacob-carlborg Jun 2, 2018 •

edited

Loading

JinShil commented Jun 2, 2018

wilzbach commented Jun 2, 2018

jacob-carlborg commented Jun 2, 2018

JinShil Jun 3, 2018 •

edited

Loading

JinShil commented Jun 4, 2018

JinShil commented Jun 4, 2018 •

edited

Loading

wilzbach commented Jun 8, 2018

JinShil commented Jun 8, 2018 •

edited

Loading

wilzbach commented Jun 19, 2018

jacob-carlborg Jun 20, 2018

jacob-carlborg Jun 20, 2018

wilzbach commented Jun 26, 2018

JinShil left a comment

andralex Oct 29, 2019

wilzbach Oct 29, 2019

Initial pure D build script #8293

Initial pure D build script #8293

Conversation

wilzbach commented May 25, 2018 • edited Loading

Why such a naive approach and not cmake, ninja, reggae, ...?

But the Makefile is working so nicely

But isn't depending on D another dependency?

What's missing from this PR?

What won't be part of this PR?

dlang-bot commented May 25, 2018

Bugzilla references

Testing this PR locally

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JinShil May 27, 2018 • edited Loading

Choose a reason for hiding this comment

wilzbach left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacob-carlborg commented Jun 2, 2018

jacob-carlborg commented Jun 2, 2018

jacob-carlborg Jun 2, 2018 • edited Loading

Choose a reason for hiding this comment

JinShil commented Jun 2, 2018

wilzbach commented Jun 2, 2018

jacob-carlborg commented Jun 2, 2018

JinShil Jun 3, 2018 • edited Loading

Choose a reason for hiding this comment

JinShil commented Jun 4, 2018

JinShil commented Jun 4, 2018 • edited Loading

wilzbach commented Jun 8, 2018

JinShil commented Jun 8, 2018 • edited Loading

wilzbach commented Jun 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wilzbach commented Jun 26, 2018

JinShil left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wilzbach commented May 25, 2018 •

edited

Loading

JinShil May 27, 2018 •

edited

Loading

jacob-carlborg Jun 2, 2018 •

edited

Loading

JinShil Jun 3, 2018 •

edited

Loading

JinShil commented Jun 4, 2018 •

edited

Loading

JinShil commented Jun 8, 2018 •

edited

Loading