[DDMD] Current patch list

[yebblies]

DO NOT MERGE

This is a bunch of patches currently required to get the autoconversion working. Not all of these are planned to go into the compiler as-is.

This is mostly so I can tell if I'm breaking things on other platforms, but feedback is also welcome.

DO NOT MERGE

Current status:

• Produces a working D compiler on win32.
• Produces a working D compiler on linux32
• Produces a working D compiler on linux64
• 1 outstanding patch

[MartinNowak]

One remaining problem with the conversion is the inability to model the spreaded class implementations, e.g. expression.c, in D. I do remember this came up as an argument for why we need better header/implementation separation. I will try to find that post. Given that a solution for header/implementation files will take a while, do you have any other approach to solve this.

[yebblies]

This is not a problem. With extern(C++) classes, it is quite reasonable to implement some of the member functions in C++ (just no special member functions).

Long term, this problem can be avoided by using real visitor classes, but that can wait until the port is complete.

[MartinNowak]

OK, you can have some C++ methods, but it's still not possible to split the implementation to expression.d and e2ir.d for example. We'll have a solution to this, but I wonder if you have any ideas on your own.

[yebblies]

Oh right, that.

e2ir is a bad example because it is in the glue layer, and will be left in C++.

Interpret on the other hand... I see three options (and one horror):

1. All the method bodies end up inside the class declaration. Already implemented. We then refactor to a real visitor interface after the port.
2. Build forwarding functions automatically. I've done this before, it isn't so hard, but it sure is ugly.
3. Refactor now. This is quite invasive, but will result in the least ugliness.
4. Change the language to allow out-of-class method implementations. I hate this.

[yebblies]

Could anybody on linux64 please bisect the ice for me?

[MartinNowak]

1. All the method bodies end up inside the class declaration. Already implemented. We then refactor to a real visitor interface after the port.

Problematic, because you end up mixing different compiler parts in a file.

1. Build forwarding functions automatically. I've done this before, it isn't so hard, but it sure is ugly.

This one is feasible, but it's ugly boilerplate.

1. Refactor now. This is quite invasive, but will result in the least ugliness.

It's the same refactoring as in 1 though, so I think it would be a better option.
The interesting question is how small we could keep the impact.

1. Change the language to allow out-of-class method implementations. I hate this.

Fairly simple though.

[yebblies]

My main reasons to prefer 1 over 3 are that a) it will break lots of pull requests (but I guess this will happen anyway eventually) and b) large refactorings are harder to get merged. If you or anybody else would like to work on this I will be happy to merge it. An easy first target would be the linux C++ mangling code, as this only need to touch Type and friends.

If you have time, please take a look at my DDMD pull requests. They are multiplying.

[yebblies]

Now only 68 link errors (44 constructors called from the glue layer, 4 destructors, 9 Array/ArrayBase methods, 6 variadics, 5 misc)

[andralex]

thisisawesome

[yebblies]

Down to 9 link errors - almost up to the yummy wrong code and memory corruption errors!

[andralex]

Idea: we should publish the C++->D conversion program in tools/. If it can convert dmd, it should also be able to convert a variety of idiomatic C++ programs to D, too!

[yebblies]

Idea: we should publish the C++->D conversion program in tools/. If it can convert dmd, it should also be able to convert a variety of idiomatic C++ programs to D, too!

I'll take it you haven't had a look at the actual code! While this sounds nice in theory, the reality is the conversion tool is saturated with special cases and makes some very bold assumptions about the code. The approach used could be very useful to other projects, but the tool itself is one-shot.

[MartinNowak]

Down to 9 link errors - almost up to the yummy wrong code and memory corruption errors!

This sounds great.

[yebblies]

Update: I hacked around all the remaining link errors, and it now compiles and links!

Then it fails with a stack trace showing functions that never call each other calling each other, but at least it's progress.

[andralex]

Amazing!

[yebblies]

Exciting update!

With the glue layer/backend stubbed out, all of the compilable tests and almost all of the fail_compilation tests generate the correct result!

The 'failing' fail_compilation tests are all caused by errors being incorrectly detected in the glue layer instead of the frontend. Eg invalid array ops, bad vtables, vtable shadowing, struct init circular reference, multiple mains, bad casts.

diag2452
diag6373
fail266
fail2740
fail293
fail4
fail5634
fail77
fail91
fail9418

Bugs I recently encountered came from:

• The parts I manually ported, oops
• GC.calloc not taking the same args as calloc
• GC.realloc not working properly when going from 4096 -> 4100
• passing string literals to C-style variadic functions (they don't get converted to const(char)*)
• cast(Class) meaning dynamic cast in D, along with some nasty hacks in the C++ source

[yebblies]

Now self-hosts on win32, and passes the dmd test suite on easy mode (no flag permutations) for everything except runnable\test4 and runnable\xtest46.

It compiles in ~3.3 seconds and compiles itself in ~4 seconds!

Currently segfaults when trying to compile druntime, but it's only a matter of time now...

[yebblies]

Now self-hosts, passes the test suite, and builds druntime and phobos (passing all unittests) on win32.

[ghost]

@yebblies:

Now self-hosts, passes the test suite, and builds druntime and phobos (passing all unittests) on win32.

What's the next step?

[yebblies]

@AndrejMitrovic

What's the next step?

Sorry, I thought I replied to this ages ago. I posted an update on the newsgroup.

[yebblies]

Down to 21!

[yebblies]

17

[yebblies]

15 remaining - now builds itself on linux64!

[ghost]

Keep up the good work!

[MartinNowak]

15 remaining - now builds itself on linux64!

Yay, linux. What do I have to do to try it out?

[yebblies]

Yay, linux. What do I have to do to try it out?

Pull this branch and build dmd with it. (and druntime/phobos)
Clone git@github.com:yebblies/magicport2.git.
I have it set up like:

./
    dmd
    druntime
    phobos
    magicport2

So you may need to change magicport2/posix.mak and/or magicport2/magicport.d if yours is different.
In magicport2, run make -fposix.mak. You may need to create the (initially empty) directory magicport2/port.

It segfaults trying to build the tester tool, but does seem to build itself successfully. Good luck!

[yebblies]

Only 9 remaining:

• 1 is a makefile change to allow treating the frontend as a lib ([DDMD] Build dmd as static libraries on posix #2932)
• 1 removes some C++ that isn't valid D (self-referencing struct pointers [DDMD] Remove SMALLARRAYCAP #3002)
• 5 are C++ mangling patches/hacks that will be fixed with the new C++ mangler
• 1 is a visitor patch that I should probably take out of here
• 1 is a hack around gcc emitting the vtbl into the C++ objects, and should be fixed by moving toSymbol to the new visitor

The converter has recently been extended to emit converted declarations into multiple D source files. It is in the process of being upgraded to preserve comments. Expect source previews soon!

[MartinNowak]

The converter has recently been extended to emit converted declarations into multiple D source files. It is in the process of being upgraded to preserve comments. Expect source previews soon!

Wow, this list has become short.

5 are C++ mangling patches/hacks that will be fixed with the new C++ mangler

The one by Igor Stepanov (#2074)?
That's in need for an update to use the new visitor, right?

[yebblies]

Wow, this list has become short.

Yeah, we might make it to D before dconf after all!

The one by Igor Stepanov (#2074)?
That's in need for an update to use the new visitor, right?

Yep! Hopefully that will be ready soon.

[yebblies]

New milestone: We've reached zero patches needed for the _host_ compiler. The rest are all for treating the source code that is fed into the converter. This means, unless new issues come up, the 2.066 release will be capable of building the d compiler.

[ghost]

This means, unless new issues come up, the 2.066 release will be capable of building the d compiler.

Good stuff. Keep the pulls coming, I'll be on standby and review as they come.

[MartinNowak]

Great, maybe you also want to add some DDMD milestones to the Agenda.

[ghost]

Fixed up your triple t in htttp linky. :p

[yebblies]

Slowly getting there...
In case anybody is bored and wants to help out, here is the list I'm working off for visitor/free function refactoring:

doc.c
Dsymbol::emitDitto
ScopeDsymbol::emitMemberComments
Dsymbol::emitComment
Dsymbol::toDocBuffer

hdrgen.c
Module::genhdrfile
Module::toCBuffer

traits.c
TraitsExp::isTypeX
TraitsExp::isFuncX
TraitsExp::isDeclX
TraitsExp::semantic

builtin.c
FuncDeclaration::isBuiltin

clone.c
AggregateDeclaration::hasIdentityOpAssign
StructDeclaration::needOpAssign
StructDeclaration::buildOpAssign
StructDeclaration::needOpEquals
AggregateDeclaration::hasIdentityOpEquals
StructDeclaration::buildOpEquals
StructDeclaration::buildXopEquals
StructDeclaration::buildXopCmp
StructDeclaration::buildCpCtor
StructDeclaration::buildPostBlit
AggregateDeclaration::buildDtor
AggregateDeclaration::buildInv

toobj.c (g)
Module::genmoduleinfo
Dsymbol::toObjFile
ClassDeclaration::baseVtblOffset

tocvdebug.c (g)
Dsymbol::toDebug
Dsymbol::cvMember

toir.c (g)
FuncDeclaration::buildClosure
TypeFunction::retStyle

iasm.c (g)
AsmStatement::semantic

toelfdebug.c (g)
Dsymbol::toDebug
Dsymbol::cvMember

opover.c
ForeachStatement::inferAggregate
ForeachStatement::inferApplyArgTypes

cast.c
SliceExp::toStaticArrayType
Expression::inferType
BinExp::scaleFactor
BinExp::typeCombine

glue.c (g)
Module::genobjfile
FuncDeclaration::toObjFile

template.c
Type::deduceWildHelper
Type::deduceTypeHelper
Type::deduceType

todt.c (g)
ClassReferenceExp::toDtI
ClassReferenceExp::toInstanceDt
ClassReferenceExp::toDt2
ClassDeclaration::toDt
ClassDeclaration::toDt2
StructDeclaration::toDt

typinf.c (g)
Type::getInternalTypeInfo
Type::getTypeInfo
Type::getTypeInfo*Declaration
Type::builtinTypeInfo

tocsym.c (g)
Dsymbol::toSymbolX
Dsymbol::toSymbol
Dsymbol::toImport
FuncDeclaration::toThunkSymbol
ClassDeclaration::toVtblSymbol
AggregateDeclaration::toInitializer
TypedefDeclaration::toInitializer
EnumDeclaration::toInitializer
Module::toModuleAssert
Module::toModuleUnittest
Module::toModuleArray
TypeAArray::aaGetSymbol
StructLiteralExp::toSymbol
ClassReferenceExp::toSymbol

[yebblies]

@WalterBright @andralex @AndrejMitrovic @9rnsr @MartinNowak @ibuclaw @klickverbot @braddr

Today we've passed a very important milestone - the conversion tool can now work on our unmodified git-head. (tested on win32/linux32/linux64)

Huge thanks to everyone who has reviewed my pull requests over the last ~10 months - over 250 pulls in total.

The next step will be cleaning up some minor formatting issues in the generated source and pushing it into this repo. My plan is to add new targets to the makefiles to convert the source and build ddmd, while leaving the C++ build intact (for now).

[MartinNowak]

Getting this into our build system and later into the auto-tester sounds like a good plan.
Should we host the converter in the tools repo to make it a little more public?

[dnadlinger]

@MartinNowak: I'm not sure if tools is the way to go, as magicport is a rather specialized tool. I think the best way to get people to notice the tool is just to write some documentation about how to build DDMD along with a few nice blog posts.

Also, we should set up a CI system for the D port ASAP to make sure master doesn't regress continuously, particularly as there have been quite a few adjustments to the source to make it convert cleanly.