Issue 14474 - Internally use UTF-8 strings for Windows #4602

davispuh · 2015-04-20T22:19:52Z

This is a quick hack for Issue 14474
Decode UTF-8 to windows Wide string and then encode it to default windows ANSI code page.

Best should use Wide WinAPI functions and then wouldn't need to encode it to ANSI code page but that's quite a lot of work... Note that not all UTF-8 characters can be encoded to ANSI code page. Thus it's possible to have some paths/names that can't be accessed with ANSI functions.

MartinNowak · 2015-04-22T00:10:13Z

What if someone wrote that file by hand not using dub? Couldn't it already be in ACP encoding? Maybe we should use a file BOM.

davispuh · 2015-04-22T00:25:38Z

You're already writing D code in UTF-8 and thus using same editor you would also save this file in UTF-8. I think there isn't any good reason for it to be in ANSI code page, especially because it varies by windows locale and IMO would cause more problems.

WalterBright · 2015-04-22T05:51:13Z

src/root/response.c

@@ -134,6 +135,15 @@ bool response_expand(size_t *pargc, const char ***pargv)

            buffer = (char *)f.buffer;
            bufend = buffer + f.len;
+#if _WIN32


Need comment explaining what is happening here.

WalterBright · 2015-04-22T05:53:41Z

Could use a test case.

MartinNowak · 2015-04-22T09:44:40Z

You can get this in a shell script like this, we also run them on Windows.
https://github.com/D-Programming-Language/dmd/blob/a5bb5a092d8917776e3d5be5a70a3ddddee48326/test/runnable/link14198b.sh

davispuh · 2015-04-24T07:24:57Z

Created a test for this, but it would have failed if windows default locale (ANSI code page) couldn't support that charset.
So I looked into how to implemented proper support so that it would always pass no matter what's window's locale and I actually managed to do that :)
Weren't needed a lot changes like I thought at first, basically idea is that we simply always store UTF-8 strings in char * , then whenever interacting with external sources (such as WinAPI functions) need to encode/decode to/from Wide stings and use Unicode WinAPI. This way we've basically full Unicode support for path/file names and error messages while still can work with char * like it would be in ASCII.

It compiles and works fine with both MSVC and DMC on Windows. No idea about Linux. Also looks like need some extra library for ddmd linking but I'm not sure how magicport works.

In action

Whenever interacting with external sources (eg. WinAPI functions) then encode/decode to/from Wide strings Fix Issue 14474

yebblies · 2015-04-24T08:53:50Z

Also looks like need some extra library for ddmd linking but I'm not sure how magicport works.

What's the error?

davispuh · 2015-04-24T14:45:17Z

When linking

ddmd.obj(ddmd)
 Error 42: Symbol Undefined __d_assert
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __d_arraycopy
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D15TypeInfo_Struct6__vtblZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D6object10_xopEqualsFxPvxPvZb
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D14TypeInfo_Const6__vtblZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D16TypeInfo_Pointer6__vtblZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_a6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_m6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_i6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D13TypeInfo_Enum6__vtblZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_k6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __d_framehandler
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _stderr
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __d_local_unwind2
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __memset128
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __d_arraybounds
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _getErrno
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __wmkdir@4
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_e6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_l6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_h6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D10TypeInfo_b6__initZ
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __wputenv@4
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __wgetenv@4
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _stdout
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _setErrno
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _iswspace@4
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __wspawnv@12
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _ShellExecuteA@24
ddmd.obj(ddmd)
 Error 42: Symbol Undefined __memset16
ddmd.obj(ddmd)
 Error 42: Symbol Undefined _D14TypeInfo_Class6__vtblZ
OPTLINK : Warning 134: No Start Address

yebblies · 2015-04-25T07:59:15Z

Looks like it can't find druntime properly.

WalterBright · 2015-05-17T08:24:39Z

src/mars.c

@@ -15,6 +15,7 @@
 #include <assert.h>
 #include <limits.h>
 #include <string.h>
+#include <locale.h>


dear gawd, please, no locale.h! It's a giant bug.

WalterBright · 2015-05-17T08:29:00Z

If we're going to support Unicode in dmd source, we should do it right, not a quick hack. I suggest doing it the same way Phobos does for D - everything is UTF-8, and conversions to/from the Windows 'W' functions is done when those Windows functions are called.

WalterBright · 2015-05-17T08:31:44Z

Note that not all UTF-8 characters can be encoded to ANSI code page. Thus it's possible to have some paths/names that can't be accessed with ANSI functions.

I.e. we need to do this right or not bother, and we know how to do it right. Phobos does it right.

davispuh · 2015-05-17T12:48:57Z

That was written for first implementation, since then I've implemented it much better. Also this is about Unicode paths when specifying source filenames and not about it's content. Please review PR changeset, but there are still some issues, I'll work on them a bit later.

WalterBright · 2015-05-17T23:20:32Z

The code looks good, but it should be moved to port.h and port.c. dgetenv should be Port::getenv, for example. Port is where the reimplementations of missing or buggy platform features go.

ibuclaw · 2018-01-14T11:49:24Z

What's the status of this now that we've switch from C++ to D? Does declaring extern(C) int wmain() even work on Windows?

rainers · 2018-01-14T12:14:15Z

Does declaring extern(C) int wmain() even work on Windows?

Not sure about dmc, but msc should be fine. If we want utf8 command line arguments we should use D main instead IMO.

Most of the file i/o should already be fixed by the long-filename-fixes (but assumes ANSI), but sources of trouble for non-ANSI characters are sensible console output and invocation of the linker (MS link understands response files with BOM, but optlink is limited in that regard).

ibuclaw · 2018-01-14T12:21:33Z

OK, so speculatively closing then. @davispuh if you think this is still a problem, please rebase and reopen.

davispuh · 2018-01-14T19:23:49Z

Original issues are still present, several places use Windows ANSI API instead of Wide API. As well as getenv is used which doesn't support Unicode characters.

Some while ago I actually rebased this and rewrote some parts from C to D as it seemed easier. This way I changed char * to D string in a lot of places. I only need to rebase it again, split it in several commits and fully test.
For some reason, I haven't been able to run dmd tests locally on Linux. Not really sure how environment is supposed to be set up.

This is how that patch looks atm 38 files changed, 1497 insertions(+), 2044 deletions(-)

JinShil · 2018-01-14T22:52:03Z

For some reason, I haven't been able to run dmd tests locally on Linux. Not really sure how environment is supposed to be set up.

I suspect this is due to Linux now requiring "-fPIC" for builds. I run into this problem also. See #7420 and #7427. I currently apply #7420, and then I can build and run all but 2 or 3 tests. My understanding is @wilzbach is working on adding hardened Linux "machines" to our CIs (#7579), but I don't clearly understand what the roadmap is.

DmitryOlshansky · 2018-01-15T08:43:34Z

src/backend/os.c

@@ -710,7 +710,7 @@ char *file_8dot3name(const char *filename)
    char *buf;
    int i;

-    h = FindFirstFile(filename,&fileinfo);
+    h = FindFirstFileA(filename,&fileinfo);


*A versions are basically deprecated junk on Windows. Do not use unless absolutely needed.

P.S.: NT kernel is UTF-16 throughout.

WalterBright reviewed Apr 22, 2015
View reviewed changes

davispuh force-pushed the issue_14474 branch from 7b394d3 to 948ea93 Compare April 24, 2015 06:55

davispuh changed the title ~~Issue 14474 - In response_expand, decode file as UTF-8 for Windows~~ Issue 14474 - Internally use UTF-8 strings for Windows Apr 24, 2015

davispuh force-pushed the issue_14474 branch from 948ea93 to fa10900 Compare April 24, 2015 07:22

davispuh force-pushed the issue_14474 branch from fa10900 to f71c396 Compare April 24, 2015 07:34

Internally use UTF-8 strings for Windows

d08de44

Whenever interacting with external sources (eg. WinAPI functions) then encode/decode to/from Wide strings Fix Issue 14474

davispuh force-pushed the issue_14474 branch from f71c396 to d08de44 Compare April 24, 2015 07:37

WalterBright reviewed May 17, 2015
View reviewed changes

dlang-bot added Needs Rebase stalled labels Jan 1, 2018

ibuclaw self-assigned this Jan 14, 2018

dlang-bot removed the stalled label Jan 14, 2018

ibuclaw closed this Jan 14, 2018

dlang-bot added the Bug Fix label Jan 14, 2018

DmitryOlshansky reviewed Jan 15, 2018

View reviewed changes

ibuclaw removed their assignment Mar 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 14474 - Internally use UTF-8 strings for Windows #4602

Issue 14474 - Internally use UTF-8 strings for Windows #4602

davispuh commented Apr 20, 2015

MartinNowak commented Apr 22, 2015

davispuh commented Apr 22, 2015

WalterBright Apr 22, 2015

WalterBright commented Apr 22, 2015

MartinNowak commented Apr 22, 2015

davispuh commented Apr 24, 2015

yebblies commented Apr 24, 2015

davispuh commented Apr 24, 2015

yebblies commented Apr 25, 2015

WalterBright May 17, 2015

WalterBright commented May 17, 2015

WalterBright commented May 17, 2015

davispuh commented May 17, 2015

WalterBright commented May 17, 2015

ibuclaw commented Jan 14, 2018

rainers commented Jan 14, 2018

ibuclaw commented Jan 14, 2018

davispuh commented Jan 14, 2018

JinShil commented Jan 14, 2018

DmitryOlshansky Jan 15, 2018

DmitryOlshansky Jan 15, 2018

Issue 14474 - Internally use UTF-8 strings for Windows #4602

Issue 14474 - Internally use UTF-8 strings for Windows #4602

Conversation

davispuh commented Apr 20, 2015

MartinNowak commented Apr 22, 2015

davispuh commented Apr 22, 2015

WalterBright Apr 22, 2015

Choose a reason for hiding this comment

WalterBright commented Apr 22, 2015

MartinNowak commented Apr 22, 2015

davispuh commented Apr 24, 2015

yebblies commented Apr 24, 2015

davispuh commented Apr 24, 2015

yebblies commented Apr 25, 2015

WalterBright May 17, 2015

Choose a reason for hiding this comment

WalterBright commented May 17, 2015

WalterBright commented May 17, 2015

davispuh commented May 17, 2015

WalterBright commented May 17, 2015

ibuclaw commented Jan 14, 2018

rainers commented Jan 14, 2018

ibuclaw commented Jan 14, 2018

davispuh commented Jan 14, 2018

JinShil commented Jan 14, 2018

DmitryOlshansky Jan 15, 2018

Choose a reason for hiding this comment

DmitryOlshansky Jan 15, 2018

Choose a reason for hiding this comment