[WIP] Merged the 2.064 frontend #519

Closed
wants to merge 74 commits into from

6 participants

@klickverbot
LDC member

This brings LDC to the level of dlang/dmd@a913ce4, i.e. somewhere around the second 2.064 beta.

dmd-testsuite has not been updated yet, the script for handling the repository (ldc-developers/ldc-scripts/dmd-testsuite) would have to be modified to also extract all the branches (no beta has been tagged yet).

Currently failing test cases: https://gist.github.com/klickverbot/7075137

This also includes an implementation of the new ModuleInfo discovery/druntime startup scheme on Linux which supports shared libraries. It didn't cause any regressions, at least not other than in the test cases failing anyway. Actually using it with shared objects has not been tested yet, though.

@redstar
LDC member

Cool. This is really a strong argument to create the 0.12.0 release now.
I thought about doing something similar but did not have too much time the last weeks.

@klickverbot
LDC member

Seems like the new startup scheme still isn't reliable, as it seems to crash on all the Travis CI boxes, while it works fine locally. This was supposed to be more of a proof-of-concept for what I was thinking to do than a finished implementation anyway, though.

Regarding the release, I'll push it out later today. Ideally, we'd then be able to release 2.064 at the same time (or shortly after) DMD is released.

@klickverbot
LDC member

For sensible shared library support, we notably need to split up druntime and Phobos (i.e. not just copy druntime a second time into the Phobos library). I'd also take this opportunity to handle -debuglib and -defaultlib in a sensible way: For them to be of any use, the user must be able to override the choice made in the config file, i.e. multiple occurrences must append the previous values instead of appending to them. It would probably make sense to allow use of , or ; as a separator for multiple libraries (so after the split, the config file could contain something along the lines of -defaultlib=druntime-ldc;phobos2-ldc -debuglib=druntime-ldc-debug;phobos2-ldc-debug. See also #417.

@klickverbot
LDC member

I probably won't be able to finish this in time, but if somebody else wants to: You can just revert the last (druntime module discovery-related) commit and most of the tests should again pass reliably.

You maybe also have to add back in the explicit druntime initialization in dmain2.d (search for version (linux)), don't recall OTOH if I pushed the commit that disables the Linux (resp. rt.sections_linux) -specific branch for LDC before removing it again.

klickverbot and others added some commits Oct 18, 2013
@redstar
LDC member

The problem with the new module discovery seems to be that on shutdown, gc_term() is called before _d_dso_registry. Then the call to unregisterGCRanges dies with a segmentation fault.

@redstar
LDC member

It turns out that in rt.dmain2, the version InitInRtSections must be used, otherwise the runtime is initialized and terminated twice. It works then as before.

@redstar
LDC member

I merged the latest changes to druntime and phobos and created a branch from this pull request. I now try to update to latest beta and fix the errors. :-)

@klickverbot
LDC member

@redstar: Must be a botched merge, sorry for the hassle. The druntime version linked in as a submodule was supposed to have that commit reverted.

@redstar
LDC member

It seems that the unit test failure in the rt.aaA is caused by different function signatures.

@klickverbot
LDC member

@redstar: Yes, they are. Off the top of my head, there are two ways of resolving this: Either change the druntime functions to accept/return void pointers (like the LDC-internal declarations do), or simply disable the tests/move them to a different file. I think I did the latter for one unittest added in an earlier releases, there is probably still an LDC_FIXME somewhere in that file.

Apart from the tests, the mismatch isn't really a problem in practice as the code in rt.aaA, i.e. the module that implements AAs, obviously can't use constructs that generate AA calls anyway.

Hm, thinking of it, I'm not sure what happens if you try to bitcode-link druntime into one module to use it for LTO – I never checked how the LLVM linker handles function type mismatch.

redstar added some commits Oct 30, 2013
@redstar redstar Update source to eb1b004.
This brings the source beyond beta4.
72f34e3
@redstar redstar Enable delete again.
Some Travis-CI errors may be caused by out-of-memory.
d48c72d
@redstar redstar Update to b52c07.
Also includes latest druntime changes.
054d789
@redstar redstar Fix version number 81775df
@redstar redstar _d_hidden_func needs a parameter.
The DMD implementation is a bit odd. A parameter is passed to the function but the function
itself is declared without a parameter. Then inline assembly is required to retrieve the
parameter.
Just do the same as gdc: declare the passed parameter.
a225ee9
@redstar redstar Make std.math.ldexp @trusted 27ccce3
@redstar
LDC member

The compile error in std.conv is fixed now. Unfortunately, the release unit test segfaults now. Root cause is the segfault in std.array, Appender.ensureAddable (line 2217). This single fault causes std.conv, std.utf, std.string and std.net.isemail to fail.

@redstar
LDC member

... and std.array segfaults because typeid(T[]).next is null in case of T being const(char)[].
Don't know why yet.

@redstar
LDC member

Reduced test case is:

void main()
{
    immutable(char)[] x = "abc";
    const(char)[] y = "abc";
    assert(typeid(x).next !is null);
    assert(typeid(y).next !is null);
}

Assertion is triggered with -O1/-O2/-O3.

@redstar
LDC member

With my last commit the segfaults are gone. But I am unsure if there are still other errors regarding TypeInfo. There are definitively some unit tests missing in druntime.

@redstar
LDC member

Current state is:

  • failure in rt.container.util is a bug in the jump threading pass (works with LLVM 3.4)
  • rt.aaA can be solved by signature changes
  • runnable/arrayop from the test suite fails because it is not yet updated to 2.064 (works with the code from 2.064)

This leaves the link errors in std.uni the last bug without explanation / workaround.

@klickverbot
LDC member

I just ran update-dmd-testsuite.sh from ldc-scripts to update the test suite repository.

@redstar
LDC member

Thanks.

@redstar
LDC member

The link errors in std.uni looks like a name mangling problem. I deleted uni.o from libphobos-ldc.a and compiled the unit test. Then ld tells me that the symbol

_D3std3uni159__T11TrieBuilderTbTwVi1114112TS3std3uni22__T9sliceBitsVm13Vm21Z9sliceBitsTS3std3uni21__T9sliceBitsVm9Vm13Z9sliceBitsTS3std3uni20__T9sliceBitsVm0Vm9Z9sliceBitsZ11TrieBuilder5putAtMFNaNbNembZv

referenced from array.o is missing. A check with nm revealed that this symbol only exists in the non-unittest version of std.uni. Any idea?

redstar added some commits Nov 8, 2013
@redstar redstar Update test suite to 2.064 4d856b8
@redstar redstar Add new testsuite.
This adds the DMD 2.064 test suite. It includes a workaround for a LLVM bug in the unit tests, too.
9ecd0da
@klickverbot
LDC member

Any idea?

Not really. It's probably worth checking the new logic in DtoDefineFunc, and look at the DMD history if there have been any changes to FuncDeclaration::toObjFile.

@dcousens

Just ran the tests for this branch, and most of them passed bar std.uni and rt_aaA.

99% tests passed, 8 tests failed out of 1080                                                                                                                              

Total Test time (real) = 932.80 sec                                                                                                                                       

The following tests FAILED:                                                                                                                                               
      425 - druntime_src_rt_aaA_debug_build (Failed)                                                                                                                      
      426 - druntime_src_rt_aaA_debug_run (Not Run)                                                                                                                       
      427 - druntime_src_rt_aaA_release_build (Failed)                                                                                                                    
      428 - druntime_src_rt_aaA_release_run (Not Run)                                                                                                                     
      903 - phobos_std_uni_release_build (Failed)                                                                                                                         
      904 - phobos_std_uni_release_run (Not Run)                                                                                                                          
      1078 - dmd-testsuite_debug (Failed)                                                                                                                                 
      1080 - dmd-testsuite_release (Failed)       

Respective errors were:

../ldc/runtime/phobos/std/uni.d:(.text+0x390): multiple definition of `_D3std3uni10compressToFNaNbNfkKAhZv'                                           
../ldc/runtime/phobos_std_uni_release.o:/home/daniel/workspace/ldc/runtime/phobos_std_uni_release.o:(.text+0x1bff0): first defined here 

...

For (afaik) every function in std.uni.

And for rt_aaA:

../ldc/runtime/druntime/src/rt/aaA.d(565): Error: Function type does not match previously declared function with the same mangled name: _d_assocarrayliteralTX  

Not sure if that helps at all, seems like you've already pointed out the issue.

@redstar
LDC member

Yeah, I know both of them. There is also runnable/foreach5.d and runnable/eh2.d which are failing. In the first test, a destructor is called twice. The second test seems to have problems with -fPIC.

redstar added some commits Nov 20, 2013
@redstar redstar Wrong break target choosen for labeled break.
This fixes a test failure in runnable/foreach5.d.
3ef3e35
@redstar redstar Enable NRVO for static arrays.
This is new in 2.064 and fixes a test failure in runnable/sdtor.d.
18d53f8
@redstar redstar Update to LLVM 3.5 f662221
@redstar redstar Added new LLVM 3.4 intrinsics 711a04b
@redstar redstar Merge branch 'master' into merge-2.064
Conflicts:
	runtime/druntime
87965cd
@redstar redstar Merge branch 'master' into merge-2.064 eb85713
@redstar redstar CCP classes do not have a monitor.
This fixes a failure in runnable/xtest46.d. I still need to check
the structure of the emitted class for compability with C++.
e1464fe
@redstar redstar Replace general error with more accurate error message (does not reso…
…lve issue)
1a36f9b
@MartinNowak

Good to see, that the hidden global weak symbols seem to work as section brackets.
At some point I want to switch dmd's backend implementation to something similar to this.

LDC member

Yep, it works in theory, although the current implementation is not too pretty due to the LLVM limitations mentioned in the above (the custom section/weak linkage bug also causes the _minfo_beg/_minfo_end to be duplicated for each module right now, which is rather ugly, even though the amount of bloat caused by that is not too bad).

@MartinNowak

You should actually be able to just emit one function and a reference into .ctors/.dtors each.
The _d_dso_registry implementation can tag the shared library using the dsoSlot to detect whether it should run ctor or dtor.
Also you could save the dsoInitiaized guard.

LDC member

I should be able, yes, but I didn't find a way to get this to work on the LLVM IR level. Module-level inline assembly would be an option in theory, but then, we can't do LTO anymore, as the LLVM linker wouldn't know how to coalesce them, and I couldn't find a way to make the MC asm parsers just ignore the subsequent instances. The proper solution would be to use linkonce_odr globals with the section attribute set to .ctors.<something>, but unfortunately, MC doesn't currently handle this case properly (specifying a custom section causes the weak linkage part to be disregarded).

And once the constructor is called multiple times, just using dsoSlot to distinguish between initialization/destruction doesn't work anymore.

Thinking of it, I should add the above as a source comment, or at least write that blog post on the topic I have been meaning do to for ages. -.-

And once the constructor is called multiple times, just using dsoSlot to distinguish between initialization/destruction doesn't work anymore.

Why is it called multiple times? Can't you mark the .ctors entries as weak symbols too?

LDC member

With current LLVM MC, you can either mark a symbol as weak, or specify a custom section, but not both. It's likely an unintended restriction/bug, but I wanted a solution that works now, with stock LLVM.

With current LLVM MC, you can either mark a symbol as weak, or specify a custom section, but not both.

I see, using a boolean gate won't have noticeable performance implications.

redstar and others added some commits Dec 2, 2013
@redstar redstar Fix logic to decide which new/delete operator to use. 5e6f696
@redstar redstar Fix issue with unions and initializers 08e3112
@redstar redstar Fix a failure in runnable/funclit.d 133ae1b
@redstar redstar Merge branch 'master' into merge-2.064 9c590a1
@redstar redstar Merge branch 'master' into merge-2.064 c0673fe
@redstar redstar Merge branch 'master' into merge-2.064
Conflicts:
	runtime/phobos
4faa8d8
@redstar redstar Fix wrong #if 2fa997e
@AlexeyProkhin AlexeyProkhin Fix DMD Issue 10425 - Link error with templates.
Merged dmdfe changes in typeinf.c
d235e4c
@AlexeyProkhin AlexeyProkhin Update struct type info generation to reflect changes in dmd frontend 5a181d9
@AlexeyProkhin AlexeyProkhin Remove redundant FuncDeclaration::nestedVars.
The change not only makes the code cleaner but also fixes compilation of multiple files at once.
Previously, fd->nestedVars may have been filled twice if fd was a template function instantiated in two modules simultaneously.
5a10a23
@AlexeyProkhin AlexeyProkhin Implement cast to void dcf0052
@AlexeyProkhin AlexeyProkhin Implement ClassReferenceExp::toElem 9e724b3
@AlexeyProkhin AlexeyProkhin Fix DMD Issue 10451 - Array of pointers to opaque struct gives forwar…
…d reference errors.
89573a7
@AlexeyProkhin AlexeyProkhin Fix DMD Issue 11238 - Codegen error when this is a ref parameter to …
…the method
ffe26e4
@AlexeyProkhin AlexeyProkhin Fix DMD Issue 8360 - Destruction of uninitialized temporary struct wi…
…th assert
669319b
@redstar redstar Merge remote-tracking branch 'origin/master' into merge-2.064
Conflicts:
	gen/runtime.cpp
	runtime/CMakeLists.txt
bdc1f44
@redstar redstar Extend error message if -noruntime is used.
The name of the implicit called runtime function gives a hint about
what goes wrong.
d25d28f
@redstar redstar Add -nogc switch.
This is a replacement for the -noruntime switch. With the new DSO implementation
every module generates runtime calls. The main motivation for the -noruntime switch
is to avoid implicit GC calls. Therefore a dedicated -nogc switch is introduced.
bea0b59
@redstar redstar Deprecate -noruntime switch (which has now no function). 038855e
@redstar redstar Merge branch 'master' into merge-2.064 2188ef9
@redstar redstar Fix for issue #553.
This version uses the skipboundscheck member variable.
dd3cc5f
@redstar redstar Fix generation of position-independent code.
The new module discovery scheme requires the following section order:

.minfo_beg
.minfo
.minfo_end

This works for non-PIC code because the segments have the same attributes.
However, if -relocation-model=pic is passed to ldc2 then the .minfo section
becomes writeable and the sequence of sections is changed.

The quick fix is to mark the data always as writeable. Then all sections are
always writeable.

I think a better solution would be to base this in the used relocation model.
But this information is currently only available in the driver.

This fixes the failure in test case runnable/eh2.d
e7fdbaf
@redstar redstar Merge remote-tracking branch 'origin/master' into merge-2.064
Conflicts:
	gen/arrays.cpp
c231ae0
@klickverbot klickverbot Merge branch 'master' into merge-2.064. 68f8e38
@klickverbot klickverbot Added test for GitHub #556. bcbb133
@klickverbot klickverbot Merge branch 'master' into merge-2.064. 6a17b38
@klickverbot klickverbot Return undef instead of null on toConstElem failure.
This fixes a segfault with associative array literals of
arrays of associative array literals, which occured because
of arrayLiteralToConst not handling null values properly.
Ensuring that null pointers are handled correctly in all
toConstElem callers is much more error-prone than just
returning an LLVM undef, an error is emitted anyway.

The root of the problem is actually in the kludgly
implementation of AssocArrayLiteral::toElem, we should
revisit this at some point.
3ec084d
@klickverbot klickverbot Merge pull request #560 from klickverbot/assoc-literal
Return  undef instead of null on toConstElem failure.
33e5909
@dymk

I'm curious what the status is on this branch; it seems to work fine for all of my projects. Is there an estimate for when it might be merged?

Thanks for all the hard work!

@redstar
LDC member

@dymk All phobos/druntime unit tests passes except one. Therefore it should be quite usable (and thanks for trying out!).
There are still about 5 failures in the test suite. This delays a new major release. But I try to create an alpha release soon.

@klickverbot
LDC member

The std.uni discussion is in #594.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment