Translated MinGW Windows headers - complete set

[adamdruppe]

Windows programming with D quickly finds that the core.sys.windows.windows header is incomplete. Work has been done to port the public domain win32 headers to D:

https://github.com/CS-svnmirror/dsource-bindings-win32

(among other repos) and we sometimes copy/paste definitions too. This brings them all under the druntime banner, in the core.sys.windows namespace - same as we do with Posix and stdc headers. Now, Windows libraries don't need to face incompatibilities anymore from using the different modules to access the same functionality and D will work for low-level Windows devs out of the box (at least 64 bit, we still have the .lib file issue on 32).

I realize this is enormous (I just manually verified the changes of win32->core.sys.windows is good by doing it in each individual file myself!) but these files are typically unchanged for years. It really isn't any more work than us all maintaining our private bindings anyway. Yes, it is 4 MB of code uncompressed, but does that really matter? I think not.

Besides, this will solve the common complaint that D sucks on Windows because most the API isn't even available out of the box. Totally worth doing.

[adamdruppe]

Note also: http://forum.dlang.org/thread/cqgfyjbcftcjzqpveulx@forum.dlang.org?page=2#post-m875ga:2428a:242:40digitalmars.com

[adamdruppe]

And an OLD OLD bug: https://issues.dlang.org/show_bug.cgi?id=317

Those of us who use D on Windows have wanted this for many years.

[Orvid]

This file probably shouldn't exist, as it is nothing but the module definition and a private import.

[Orvid]

Part of this does use explicitly private exports and I believe some of the imports might be expecting to be public, but I'm not sure. (sorry if that doesn't make sense)

src/core/sys/windows/windows.d should probably be the package definition, and src/core/sys/windows/core.d may need to go away, depending on how and where it's used.

[adamdruppe]

On Fri, Jan 02, 2015 at 03:14:05PM -0800, Orvid King wrote:

Part of this does use explicitly private exports and I believe some of the imports might be expecting to be public, but I'm not sure. (sorry if that doesn't make sense)

Yeah, I need to try it on some real world code and see what happens
before we actually pull this in, I just wanted to open the PR ASAP so
folks can start to weigh in early too.

This is potentially the most exciting week in D for me for a while!

[PetarKirov]

@adamdruppe Can this be used for testing? It is a translation of 5th Edition Programming Windows book by Charles Petzold which is quite outdated, but still covers a lot of Win32 stuff that are relevant today.

[ghost]

FYI I'm not sure about the build state of that project with the latest compiler. I haven't touched that codebase in a while.

[andralex]

test fail

[DmitryOlshansky]

@adamdruppe any chance to revive this?

[adamdruppe]

On Wed, Jul 08, 2015 at 11:33:19AM -0700, Dmitry Olshansky wrote:

@adamdruppe any chance to revive this?

Yeah, it is still something I want to do, gotta do a little more
translation though and I've been busy with a lot of other things.

Remind me, i'll try to do it this saturday.

[adamdruppe]

So I think this thing is actually working now (lol i thought it would take 7 minutes to throw these in, I'm now at over 7 hours on it)...

But the tester failed. Can someone take a look at this?

https://auto-tester.puremagic.com/show-run.ghtml?projectid=1&runid=1642588&isPull=true

Do I need to manually copy the files from /src to /import? I kinda assumed that would be done automatically.... or is something else causing that?

[DmitryOlshansky]

I guess yes. Just look into win32/win64 makefile. They are (as usual) just a ton of of copy-pasty copy commands:
https://github.com/D-Programming-Language/druntime/blob/master/win32.mak#L613

P.S. I really-really wish we had that reggae tool to build D.

[adamdruppe]

ugh, all done in a list too :( this is like 188 files... really easy to manage with *.d but a list like that will make me miserable. Easy to generate a script to make the list but still, come on.

[DmitryOlshansky]

If your Walter-foo is high this might be a good time to convince him that Makefiles are evil.

But that's OT, I think generating these rules is the way to go to merge this.

[DmitryOlshansky]

@adamdruppe The first merge confilic commit is NOT good, you pull a whole lot of work done on master into a single commit. Please go the pull --rebase route. In fact you can just redo the pull on top of fresh upstream master.

Anyhow pinging to see how it goes.

[adamdruppe]

On Wed, Jul 15, 2015 at 12:30:57PM -0700, Dmitry Olshansky wrote:

@adamdruppe The first merge confilic commit is NOT good, you pull a whole lot of work done on master into a single commit. Please go the pull --rebase route. In fact you can just redo the pull on top of fresh upstream master.

How do I do that?

tbh I don't really know how to use git... I just do the
basics over and over again.

Anyhow pinging to see how it goes.

I still have to fix the makefile then it should be set.

[DmitryOlshansky]

How do I do that?

Okay, If I had to do it I'd :

git remote -v # check to see if this has upstream repo as remote pointing to DLang
git checkout master
git pull --rebase upstream master
git checkout -b new_branch_name

# apply your changes from this pull to the working tree
git add std/ # add all files in std/ 
git commit
git push

Then anytime you want to update this pull:

git checkout branch_name
git pull --rebase upstream master

[adamdruppe]

OK, I think I got the commit rebase worked out. Last thing should just be fixing the makefile.

[DmitryOlshansky]

@adamdruppe - any progress on makefiles?

[DmitryOlshansky]

Should also add fix Issue 317 to commit message

[adamdruppe]

On Tue, Jul 28, 2015 at 09:10:56PM -0700, Dmitry Olshansky wrote:

@adamdruppe - any progress on makefiles?

I'm on it again now. What a pain!

[etcimon]

I added the version comparison code in case you're interested: etcimon/windows-headers@96be320#diff-469c8eb924b62375a9d80614bfc3351eR177

[adamdruppe]

On Thu, Sep 10, 2015 at 03:26:05PM -0700, Dmitry Atamanov wrote:

And not necessary to add a list of new files in mak\SRCS?

I want to avoid that since then the pragma(lib)'s are liable
to trigger when you don't actually want them (especially since
dmd 32 bit doesn't even include them all).

Arguably, those pragmas should just be removed, but I like having
them there anyway. So trying to get the best of both worlds.

I'm finally to the point where Phobos actually builds, there's
just test failures.... for some reason.

[adamdruppe]

Well, I might have to concede at least the winbase.d on which Phobos depends being added to the sources. Seems that dmd, even if the initializers are all zero, wants to reference the .init if they aren't zero by default. (If they are zero by default, it works fine without it. I think this is actually a minor bug, but meh, I REALLY do want to get this merged some time this year, so i'll work with it for now)

Hopefully, last run through the autotester.

[data-man]

@adamdruppe Which version of MinGW Windows headers translated?
I'am using mingw-w64 and noticed that many of the headers missing in translated D sources.

[adamdruppe]

On Thu, Sep 10, 2015 at 06:19:03PM -0700, Dmitry Atamanov wrote:

@adamdruppe Which version of MinGW Windows headers translated?

They are actually pretty old. I don't even remember where they
came from at first. I borrowed someone else's D translation and
tried to get it to fit into the druntime pattern.

We've been talking about this for years so I figured it is time
to sit down and actually make it happen (and now, almost a year
later than that, it might finally be getting there...)

[adamdruppe]

AAAAAAAARRRRRRRRGGGGGGGGGHHHHHHHHH all tests passed except win64 with this delicious error:
std\process.d(1014): Error: core.stdc.stdio.SEEK_END at ..\druntime\import\core\stdc\stdio.d(230) conflicts with core.stdc.stdio.SEEK_END at ..\druntime\import\core\sys\windows\mmsystem.d(1019)

A symbol conflicts with ITSELF! !@$##*(ing module bugs! I blame bugzilla 314.

[DmitryOlshansky]

AAAAAAAARRRRRRRRGGGGGGGGGHHHHHHHHH all tests passed except win64 with this delicious error

Too bad. However I seriously doubt that it's bugzilla 314. You can seek around for aliases but I've seen this bug with wrong module names supplied that had nothing to do with 314.

[adamdruppe]

On Thu, Sep 10, 2015 at 09:58:49PM -0700, Dmitry Olshansky wrote:

Too bad. However I seriously doubt that it's bugzilla 314. You can seek around for aliases but I've seen this bug with wrong module names supplied that had nothing to do with 314.

I was able to work around that anyway... now it is only giving a
fail on Win64.... and it is doing so with no error message. Come on.

[rainers]

now it is only giving a fail on Win64.... and it is doing so with no error message. Come on.

If you translate the error code to hex, it's 0xC0000005: an access violation.

[adamdruppe]

On Fri, Sep 11, 2015 at 07:15:36AM -0700, Rainer Schuetze wrote:

If you translate the error code to hex, it's 0xC0000005: an access violation.

aye, I noticed that too... but where? I'll have to set up
a win64 box to run it locally (I only have win32 set up)

[DmitryOlshansky]

aye, I noticed that too... but where? I'll have to set up
a win64 box to run it locally (I only have win32 set up)

I'll try your PR on Win64 and see how it goes.

[adamdruppe]

Thanks. I'm pretty sure there's just a DWORD where there
should be a SIZE_T somewhere and the compiler didn't catch
it at the usage point. I'm also going to look at some point
today - I have win64 available here now, just gotta install
VS and dmd on it.

[adamdruppe]

The win64.mak is unbelievably awful. My dmd install on 64 bit just worked well for my programs but fails miserably to build this. Why does the druntime build have to be this bad? It drives me nuts.

[adamdruppe]

TESTS FINALLY PASS!!!!!

[DmitryOlshansky]

Splendid! Squash it and I'll do the honors

[CyberShadow]

@ all, please see #1402 which is a different approach, based on a recent copy of the headers and hopefully will result in less code breakage. Currently blocked on a DMD regression.

[CyberShadow]

Sample of regressions identified by my test script introduced in this PR:

wintest32.d(62): Error: undefined identifier 'PCCH' in module 'core.sys.windows.windows', did you mean alias 'PCH'?
wintest32.d(64):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of PCCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 64, (_error_).stringof, "const(char)*"))
wintest32.d(107): Error: undefined identifier 'LPCTCH' in module 'core.sys.windows.windows', did you mean alias 'LPTCH'?
wintest32.d(109):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LPCTCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 109, (_error_).stringof, "const(char)*"))
wintest32.d(113): Error: undefined identifier 'PCTCH' in module 'core.sys.windows.windows', did you mean alias 'PTCH'?
wintest32.d(115):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of PCTCH is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 115, (_error_).stringof, "const(char)*"))
wintest32.d(331): Error: type of FARPROC is:
extern (Windows) int function() nothrow @nogc
but should be:
extern (Windows) int function() nothrow

wintest32.d(344): Error: undefined identifier 'LPCOLORREF' in module 'core.sys.windows.windows', did you mean alias 'COLORREF'?
wintest32.d(346):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LPCOLORREF is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 346, (_error_).stringof, "uint*"))
wintest32.d(352): Error: type of DLGPROC is:
extern (Windows) int function(void*, uint, uint, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, uint, uint, int) nothrow

wintest32.d(355): Error: type of TIMERPROC is:
extern (Windows) void function(void*, uint, uint, uint) nothrow @nogc
but should be:
extern (Windows) void function(void*, uint, uint, uint) nothrow

wintest32.d(358): Error: type of GRAYSTRINGPROC is:
extern (Windows) int function(void*, int, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int, int) nothrow

wintest32.d(361): Error: type of WNDENUMPROC is:
extern (Windows) int function(void*, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int) nothrow

wintest32.d(364): Error: type of HOOKPROC is:
extern (Windows) int function(int, uint, int) nothrow @nogc
but should be:
extern (Windows) int function(int, uint, int) nothrow

wintest32.d(367): Error: type of SENDASYNCPROC is:
extern (Windows) void function(void*, uint, uint, int) nothrow @nogc
but should be:
extern (Windows) void function(void*, uint, uint, int) nothrow

wintest32.d(370): Error: type of PROPENUMPROCA is:
extern (Windows) int function(void*, const(char)*, void*) nothrow @nogc
but should be:
extern (Windows) int function(void*, const(char)*, void*) nothrow

wintest32.d(373): Error: type of PROPENUMPROCW is:
extern (Windows) int function(void*, const(wchar)*, void*) nothrow @nogc
but should be:
extern (Windows) int function(void*, const(wchar)*, void*) nothrow

wintest32.d(376): Error: type of PROPENUMPROCEXA is:
extern (Windows) int function(void*, char*, void*, uint) nothrow @nogc
but should be:
extern (Windows) int function(void*, char*, void*, uint) nothrow

wintest32.d(379): Error: type of PROPENUMPROCEXW is:
extern (Windows) int function(void*, wchar*, void*, uint) nothrow @nogc
but should be:
extern (Windows) int function(void*, wchar*, void*, uint) nothrow

wintest32.d(382): Error: type of EDITWORDBREAKPROCA is:
extern (Windows) int function(char*, int, int, int) nothrow @nogc
but should be:
extern (Windows) int function(char*, int, int, int) nothrow

wintest32.d(385): Error: type of EDITWORDBREAKPROCW is:
extern (Windows) int function(wchar*, int, int, int) nothrow @nogc
but should be:
extern (Windows) int function(wchar*, int, int, int) nothrow

wintest32.d(388): Error: type of DRAWSTATEPROC is:
extern (Windows) int function(void*, int, uint, int, int) nothrow @nogc
but should be:
extern (Windows) int function(void*, int, uint, int, int) nothrow

wintest32.d(405): Error: type of FAILED is:
bool function(int Status) nothrow @nogc
but should be:
bool function(int) pure nothrow @nogc

wintest32.d(408): Error: type of SUCCEEDED is:
bool function(int Status) nothrow @nogc
but should be:
bool function(int) pure nothrow @nogc

wintest32.d(809): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(810): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(811): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(814): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(815): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(819): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(820):        while evaluating pragma(msg, format(Char, Args...)(in Char[] fmt, Args args) if (isSomeChar!Char)("%s(%d): Error: type of LIST_ENTRY is:\x0a%s\x0abut should be:\x0a%s\x0a", "wintest32.d", 820, "LIST_ENTRY", (_error_).stringof))
wintest32.d(821): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(822): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(828): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(834): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(835): Error: undefined identifier '_RTL_CRITICAL_SECTION' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION'?
wintest32.d(838): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(839): Error: undefined identifier '_LIST_ENTRY' in module 'core.sys.windows.windows', did you mean struct 'LIST_ENTRY'?
wintest32.d(844): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(850): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?
wintest32.d(856): Error: undefined identifier '_RTL_CRITICAL_SECTION_DEBUG' in module 'core.sys.windows.windows', did you mean struct 'RTL_CRITICAL_SECTION_DEBUG'?

There are many more (wintest32.d is 8374 lines long), but DMD stops compiling after 20 errors. These are all fixed in #1402.

I uploaded the generated test programs here: git clone git@gist.github.com:bd4bd187133b03c15e79.git

[adamdruppe]

@DmitryOlshansky let's hold off on a merge until we see how CyberShadow's turns out, he seems to making good speed on it and has a better test case than I do.

I agree with him that some of mine are declared @nogc when they shouldn't be and he found a few other bugs in there too (and I was already aware of some const correct issues, which have been fixed in the ~10 months since I forked the bindings to start this, and cybershadow's includes them). So, while mine finally passes the tests in Phobos, if his pass all tests too it is a better way to go.

[DmitryOlshansky]

@DmitryOlshansky let's hold off on a merge until we see how CyberShadow's turns out, he seems to making good speed on it and has a better test case than I do.

Agreed. Having WinAPI covered by tests is much better;)

[adamdruppe]

Closing this because CyberShadow's superior alternative got in! Yay, finally!