[RFC] Add -Xcc= compiler switch #10441
Conversation
A global on/off switch, intended to become the default, which caused all pre-existing dmd.conf files to be invalid and broke linking scripts, might not be the optimal solution.
|
I have had success with my first test of this PR on Linux with the test_d project! So that is a promising start that should encourage you to quickly mature this PR. The test_d project is a small self-contained project to test my CMake D language support under a wide variety of circumstances for any Posix or Windows platform for any D compiler (e.g., your PR for dmd) which is installed or can be built and installed on those platforms. @CyberShadow I highly encourage you to try test_d yourself now and later as you mature your PR. Although test_d is completely self-contained (it refers to no files outside its own source tree) the top-level directory of this test_d source-tree is located at cmake/test_d within the PLplot git repository. Here is how to gain access to test_d and comprehensively test any version of dmd with that project: Then look in the README file in that top-level directory for further instructions about how to comprehensively test CMake D language support for dmd with this project. Note that currently test_d does not test the -pthread C compiler option that via -Xcc for this PR could be passed to the C compiler that is called by dmd. But see the plplot git log message for commit 9251b9540 for further plans concerning implementing such a test, and thus testing -Xcc with test_d. And once -pthread works for test_d it should be straightforward to update the plplot build system so it also works for the plplot project (which has its own comprehensive test script). So there is a lot to look forward to here as you mature your PR and as I mature the CMake build systems for first test_d, and second plplot. |
|
I have now (PLplot commit ab8a90546) changed the C library of the test_d project to use the -pthread link option, and for a dmd version corresponding to this pull request (and ldc2) showed the -Xcc=-pthread option (which translates to the -pthread option for the underlying C compiler that is used for linking by dmd and ldc2) works well. (For further details see the commit message for ab8a90546.) So these are quite promising results that indicate this version of dmd (and also ldc2) should (with a bit of extra work on my part) also work well for the much more complex PLplot project. (The third choice of D compiler, gdc, already works well for both test_d and PLplot.) @CyberShadow In sum, the test_d support of a version of dmd corresponding to this PR has matured (at least on my Linux platform and likely Mac OS X as well), and I am pretty sure I will soon be able to say the same thing concerning the PLplot project. So I am fairly rapidly reaching my -pthread test goals here, and as a result I look forward to you maturing this PR so it is in good shape to be merged for the next release of dmd. |
CyberShadow
force-pushed
the
pull-20190926-222551
branch
from
7a04443
to
0ac23c8
Compare
CyberShadow
force-pushed
the
pull-20190926-222551
branch
from
0ac23c8
to
c45c173
Compare
The switch in LDC is defined as follows: cl::list<std::string>
ccSwitches("Xcc", cl::CommaSeparated,
cl::desc("Pass <ccflag> to GCC/Clang for linking"),
cl::value_desc("ccflag"), cl::cat(linkingCategory));I'm slightly worried about
Great. I had a look at |
CyberShadow
requested review from
Geod24,
MartinNowak,
rainers and
WalterBright
as code owners
|
@CyberShadow How do I get access to your -Xcc work now? Here is what happened with the old method (after a git fetch): But executing "git pull" lead to two conflicts which I don't think should be my business to resolve, but I thought I should let you know about those in case you were unaware of them. In case it makes any difference here are my notes of exactly what I did to get access to your |
|
I did some TODOs and squashed the commits into one, so that caused the merge failure. $ git fetch CyberShadow
$ git checkout pull-20190926-222551
$ git reset --hard CyberShadow/pull-20190926-222551should work. |
|
Thanks for that git suggestion which did work. For the record (since these "remote" manipulations are all new to me) here is exactly what I did, and the git response. So this indicates there have been no code changes on this branch since I last pulled it. I understand the reason for this lack of activity is you are waiting for those reviews. But I do admit to getting a little antsy about how soon the decision will be made about whether your work or the changed -L semantics approach will be getting into the next release which is only 3 weeks from now. So I hope those reviews are completed soon so you can move closer to making the decision. |
|
@kinke Any thoughts on the LDC comma question? @WalterBright probably wants to know whenever a compiler switch is to be added. |
|
I think commas support would be nice, similar to gcc/clang More important though IMO is the correct order wrt. |
If we are doing this, then it raises the question of what to do with existing switches, where DMD interprets the comma literally but LDC as a delimiter.
I think we're good here... |
|
CyberShadow said "I think we're good here..." [with regard to preserving order of -Xcc and -L options in generated gcc result]. @kinke I just confirmed option order was preserved with my own build of Cybershadow's work. |
| @@ -266,6 +264,7 @@ extern (C++) struct Param | |||
| // Linker stuff | |||
| Array!(const(char)*) objfiles; | |||
| Array!(const(char)*) linkswitches; | |||
| Array!bool linkswitchIsForCC; | |||
There was a problem hiding this comment.
We could have also defined a struct of both properties and then just used a single array.
There was a problem hiding this comment.
Yes. Though, that would have involved touching all the other platforms' linker driver code. Can refactor in a follow-up.
|
CC @atilaneves |
|
Thanks for your pull request, @CyberShadow! Bugzilla referencesYour PR doesn't reference any Bugzilla issue. If your PR contains non-trivial changes, please reference a Bugzilla issue or create a manual changelog. Testing this PR locallyIf you don't have a local development environment setup, you can use Digger to test this PR: dub fetch digger
dub run digger -- build "master + dmd#10441" |
|
I wanted to test this merge to the master branch to make sure it continues to work as well as this PR. I am not sure whether this issue is due to some issue with the present merge (as opposed to the pull request which is fine) or some other independent issue with the master branch. In any case, I thought I should report this master branch issue here first. If you have any trouble replicating it please let me know what else you think I should try with regard to building libphobos. Note, I am using my standard bash script to make a consistent build, test, and install of dmd, libphobos, etc. that starts from scratch (with all old results removed). That script worked roughly a month ago with no configuration, build, test, or install issues. But if you are having difficulty replicating the above issue with the installed version of libphobos, then please let me know if you are aware of any change in the build procedure for libphobos in the last month, and meanwhile I will be researching that possibility as well. |
|
Digger works fine for me, if it's the same for you then you could narrow down the problem by examining the differences in what commands the two run. |
|
This looks like stale built artifacts. |
core.thread has been refactored to a package, so it's likely that there is still a core/thread.d in your druntime import folder that phobos uses instead of the package. |
|
@CyberShadow The short story is thanks for your quick response which helped me to (eventually) solve this issue. The (good) "ldd -r" results I get now are And I should mention that my test_d project runs without issues for this latest build and install. The longer story is that response showed you could not replicate the issue with digger which inspired me to review exactly what my script and the relevant Posix Makefiles were doing, and it turned out the ldd error for phobos that I documented in my previous post was caused by not building the "clean" target for druntime before the build and install of dmd and phobos. That is, somehow stale build files for druntime were so severely interfering with the phobos build that ldd was failing for the installed version of libphobos! This obscure interaction between phobos and druntime certainly violates the "least surprise" principle. And something similar happens for the install target for druntime (which I fixed when I first implemented this script). In my defense, all of these issues in my build script that I have had to address are due to the seriously incomplete documentation at https://wiki.dlang.org/Building_under_Posix. A minor issue is that page does not state the default target name (which turns out to be "all" for dmd and phobos). Much more seriously, that wiki page says absolutely nothing about the "clean" (dmd, phobos, and druntime) targets that must be built before the "all" (dmd and phobos) targets; the "install" (dmd, phobos, and druntime) targets that must be be built after the "all" targets; and the test (dmd) and unittest(phobos) targets that must be built after the "install" targets. See the attached latest version of my build script. All these additional undocumented targets and especially the order in which they should be built and for which repositories should be of interest to any serious builder/tester of dmd and phobos such as me. So I hope you view these remarks as an informal bug report concerning the incomplete content of https://wiki.dlang.org/Building_under_Posix which I hope you would be willing to address in your copious spare time. :-) Seriously, though, if you cannot make such a commitment any time soon, let me know, and I will open a separate bug report about this dmd wiki content issue.
Will do! |
|
@airwin1 if you could file a bug report with the text you just posted, that would be good. |
You shouldn't need to run
Because you typically don't need to and this page documents the first-timer build from scratch.
FYI: You don't need to run
I'm sorry that this caused troubles for you, but it's really just: for repo in dmd druntime phobos
cd repo
git fetch upstream && git merge --ff-only upstream/master
make -f posix.mak -j10
cd ..Though tester can just use |
|
@Geod24 commented > This looks like stale built artifacts. It turns out you were absolutely right. But as soon as I saw Cybershadow's post I put my head down and figured out the issue which could be immediately solved by the brute-force approach of starting with a fresh clone of each repository each time, but it took me a lot longer to figure out a more sophisticated approach, i.e., find the missing clean target build in my script and figure out how to test for any remaining stale build files (with "git status --ignored") after the build of each clean target. Which reminds me that although your clean target for phobos was fine, your clean targets for dmd and druntime need work, i.e.,that git status test after building the clean target showed the following results for dmd: and the following result for druntime: Evidently, these stale build files are not a serious issue that interferes with good test results. Nevertheless, I assume the dmd developers would like these issues to be fixed. So again, if nobody here wants to fix those issues immediately, I would be willing (sigh) to fill in a separate bug report for each, but as you can no doubt tell from that sigh, I hope this informal report is sufficient for what I assume are trivial, easy-to-fix issues. |
|
@wilzbach said:
Sure, for a perfect build system you should not have to build clean targets. But we do live in an imperfect world so I disagree with you that the clean targets do not need documentation in what is recommended as the wiki entry on building dmd on Posix systems (so that is definitely not just a page for beginners). I agree a nightly build would likely have been much easier for me, but as a matter of principle I do like to figure out how to build free software from that software's documentation. Your assertion that dmd can be run from the build tree is correct. In fact, I was forced to do that for early versions of my build script that did not deal with the druntime install I mentioned. However, experienced builders of lots of different free software like me tend to like to run install-tree results whenever possible. The substantial saving of disk space between the build tree and install tree obviously does not matter much if either (a) you have a build system that forces you to build in the source tree rather than a separate build tree or (b) your are building using a source tree that is the working directory of a local git repository where that git repo size is obviously going to be huge compared to either the build tree or install tree size. But in any case most free software projects I have been experienced with do implement a separate build tree because users of those projects prefer pristine source trees that contain no stale build results by definition regardless of whether the source tree is a git working directory or a tarball unpack. Anyhow, out of habit I did try the dmd install, and I had to do some extra steps to debug that install because of the seriously incomplete documentation of the Posix build instructions that I have already mentioned. By the way, I do realize your experience developing and building and testing dmd is likely far outside my experience with other free software projects. Also, I haven't used proprietary software since 1996 (except for my direct use of the BIOS on the computers I have bought in the two decades since and my indirect use via web servers) as a long-standing free software principle I live by. So I am pretty sure that building and testing like I do from a locally cloned repository because I want to avoid the github proprietary add-ons as much as possible may be far outside your experience. So we may have to simply write off the differences in our two approaches as a culture clash and leave it there since it appears that other dmd developers are definitely interested in getting that Posix build wiki entry updated properly for this J. Random Builder that doesn't do building, testing, and using dmd in exactly the same way that dedicated dmd developers likely do that. :-) |
|
To sum up my dmd build experiences so far, from my "J. Random Builder" perspective, I don't like the dmd build system because there doesn't appear to be any way to have a separate build tree that allows me to maintain an absolutely pristine source tree; the obscure interactions between the phobos build and druntime build, clean, and install targets that violates the principle of least surprise and demonstrably has screwed up my recent install; apparently a separate build procedure for Windows versus Posix; and a hard-coded install directory that makes it gratuitously difficult to package dmd for free software distributions (which is likely one of the reasons that Debian Buster, for example, has not packaged dmd whereas they have long since already packaged gdc and ldc2). But that is the end of my bitching about these dmd, phobos, and druntime build-system issues in this post, and the rest of this is dedicated to an idea I have for permanently fixing these issues. I am quite busy continuing to develop my own free software projects on Linux so despite being a retiree, I don't have a lot of time to help out with dmd. Nevertheless, if there is any dmd developer here that is similarly concerned with what I think of as pretty bad deficiencies in the dmd, phobos, and druntime build systems, please get in touch with me at airwin@users.sourceforge.net to see whether we can address these issues together. But to summarize what I had in mind, I would be willing to collaborate with such a dmd developer or developers to implement a "super-build" CMake-based build system for dmd, phobus, and druntime that could almost trivially handle all the git aspects of keeping all repositories up to date with each other and to configure the clean, all, install, test, and unittest targets I have been discussing in a reliable, cross-platform way. Note that CMake typically co-exists happily with other build systems. Which makes it trivially easy to carefully compare results for building the targets configured by one CMake-based "super-build" system with what your current Posix and Windows builds for each separate repository are currently generating as a result of building the targets that are implemented for each of those build systems. I do have much more than casual experience with CMake. For example, I have had more than 10 years of intense experience with being the go-to guy for the CMake-based build system for PLplot. Typically, the plplot software project (as opposed to certain parts of the PLplot source tree) is not organized as a super-build project, but within the PLplot source tree I implemented roughly 5 years ago a large super-build project (located at cmake/epa_build within the PLplot source tree) that attempted to build PLplot and most of the large number of external libraries it uses (e.g., Qt and a large subset of the GTK suite of software libraries) on both Linux and Windows. I had complete success with epa_build on Linux, and I almost (aside from a small fraction of the GTK software libraries) also had complete success for traditional MinGW/MSYS (not MinGW-w64/MSYS2 since MSYS2 was not available back then, you don't need something like epa_build on that platform because that distribution provides pre-built system versions of most PLplot prerequisites and, in any case, MSYS2 currently does not work properly on the Wine version of Windows). Therefore due to that old super-build implementation experience I am confident I could implement the framework for such a super-build system that did all the relevant cleans, build, and installs for each repository in the right order on Linux. And that super-build would be quite near to working on non-Linux free software distributions such as fink, MacPorts, and Homebrew on Mac OS X and MSYS2 on Windows. Of course, that framework would just be the start that could likely be implemented by me in just a week or so, and the hard grafting (which I would cheerfully leave to my dmd colleagues!) would be to implement the configuration of the many different test targets in the same way they are implemented now with your present build systems for each of your repositories and to test the result on all Linux and non-Linux platforms my potential dmd collaborators have access to. Of course, once that hard grafting was done on the testing front by my collaborators, it would be completely straightforward to generalize those results from test targets to Ctest results and the associated dashboards of results you can submit from those to any free (in both senses) dashboard server such as my.cdash.org. So I would be willing to contribute that extension as well if there is any interest by dmd developers in it. However, I assume from a previous discussion here there would likely be a large culture clash between my typical development experience and that of my potential dmd collaborators. For example, you may be completely happy using github proprietary software (digger?) that does the equivalent of CTest (unless and until github starts charging for use of their proprietary software). But even if you ignore that good (from the free software perspective) CTest possibility, I still think a CMake-based super-build system that configures clean, all, install, and many different test targets for each of dmd, phobos, and druntime in an absolutely reliable (no violation of the principle of least surprise) and cross-platform way is well worth having. So if somebody here is truly interested and inspired by this possibility (i.e., they would be prepared to carry on with the test target implementation and on-going maintenance of this proposed CMake-based super-build system once I gave them a big start), then please get in touch with me if you want me to start working on my (start-up/proof-of-concept) part of this. |
|
From the "better late than never" department, I wanted to generate new issues today concerning the incomplete clean targets I mentioned above for the dmd and druntime repositories, and the incomplete documentation of the wiki item mentioned above. However, today I seem to be blocked from creating new issues. I logged in, and followed the directions at https://help.github.com/en/github/managing-your-work-on-github/creating-an-issue but to no avail. What do I need to do to create new issues for dmd, druntime, etc.? |
As documented in the README, issues are to be filed at https://issues.dlang.org/ . |
|
@CyberShadow Thanks for that hint (which I needed because I am working simultaneously with too many different bug trackers and development platforms). As promised I have now collected all build issues I noticed above into the following four bug reports: |
This aims to replace
-preview=noXlinkerand address the same problems it aimed to address, however, without breaking pre-existingdmd.conffiles or linker scripts.-Xin DMD already has a meaning (it deals with emitting JSON metadata and related options); the switch name was chosen to match the LDC switch with the same name and behavior.TODO:
-preview=noXlinker. (The supplied test does work.)CC='cc ...'-helpentry