opening files with non-ANSI filenames on Windows

[wojdyr]

Thank you for maintaining PEGTL.
My understanding is that currently file_input and read_input don't work with Unicode filenames on Windows? Because I see that internal::file_reader uses fopen(_s) not _wfopen(_s).
If that's true, this is a feature request to add support for Unicode filenames.

[ColinH]

Are you saying that fopen on Windows only accepts ASCII, but not Unicode encoded as UTF-8? What does wfopen get as argument, a string or a wide string? What type and encoding?

[wojdyr]

fopen takes filename in Windows (ANSI) code page, which is ASCII + 128 extra characters.

_wfopen takes wchar_t * which is either UCS2 or UTF-16 (I don't know).

I don't have a lot of experience with it, but I came across one example of a library handling this issue: zlib has an extra function gzopen_w which does the same as gzopen but takes wchar_t* as an argument and exists only on Windows.

[skyrich62]

FYI:
https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx

FILE *fopen(
const char *filename,
const char *mode
);
FILE *_wfopen(
const wchar_t *filename,
const wchar_t *mode
);

Unicode Support

fopen supports Unicode file streams. To open a Unicode file, pass a ccs flag that specifies the desired encoding to fopen, as follows.

FILE *fp = fopen("newfile.txt", "rt+, ccs=encoding");

Allowed values of encoding are UNICODE, UTF-8, and UTF-16LE.

When a file is opened in Unicode mode, input functions translate the data that's read from the file into UTF-16 data stored as type wchar_t. Functions that write to a file opened in Unicode mode expect buffers that contain UTF-16 data stored as type wchar_t. If the file is encoded as UTF-8, then UTF-16 data is translated into UTF-8 when it is written, and the file's UTF-8-encoded content is translated into UTF-16 when it is read. An attempt to read or write an odd number of bytes in Unicode mode causes a parameter validation error. To read or write data that's stored in your program as UTF-8, use a text or binary file mode instead of a Unicode mode. You are responsible for any required encoding translation.

If the file already exists and is opened for reading or appending, the Byte Order Mark (BOM), if it present in the file, determines the encoding. The BOM encoding takes precedence over the encoding that is specified by the ccs flag. The ccs encoding is only used when no BOM is present or the file is a new file.

[ColinH]

Using _wfopen() instead of fopen() is easy, the problem is that we also store the filename as source, i.e. name-of-the-source-of-the-parsed-data.

This can currently be a const char *, or a std::string, so that we can construct the std::string in the position in the parse_error exception class.

The question is, (how) can we use _wfopen() without supporting wchar_t or std::wstring as source?

Can we convert the wide characters provided to _fopen() to a regular string - or do we have to bite the bullet and enable wide strings in parse_error etc...?

[ColinH]

@skyrich62 Thanks for the link; at the moment we are just looking at the encoding of the file name, not the content, but we'll keep the possibility of an on-the-fly conversion in mind.

[d-frey]

As a work-around for this: Would it be sufficient to add a ctor taking the FILE* directly and accept an additional "normal" std::string as the source? It makes usage slightly more verbose, but I think it fits better with the rest of the design.

@wojdyr Would that be OK with you or does it cause additional problems? Note that our class takes ownership of the FILE*, e.g. it closes it during destruction.

[wojdyr]

Yes, it would work fine for me.

[ColinH]

That makes things easier, read_input now has a constructor that takes a FILE* as first argument.

[wojdyr]

Thanks!