Crash on rfl::json::write in Windows, if std::filesystem::path (or std::string or std::wstring) contains non-latin symbols

[9241304]

1. As you know, native narrow encoding for Windows is not utf-8, so when I`m trying to serialize std::string in native Windows encoding (like cp1251) I'm getting crash. In fact, I can't use the same serialization code for Linux and Windows and I can't use rfl with WinAPI narrow functions easily. I think rfl::json::write should consider native narrow encoding.
2. Ok, I thought I can resolve my problems using std::wstring or std::filesystem::path (I got crash for path returned by SHGetFolderPathA). I tried to use SHGetFolderPathW and std::wstring or std::filesystem::path, but I also got crash! From other side, if I'm constructing std::filesystem::path from utf-8, then rfl::json::write works, but std::filesystem::path variable contains broken path. If I'm converting this path from utf-8 to std::wstring, then I'm getting crash again.

Version https://vcpkg.link/ports/reflectcpp/v/0.19.0/0
Windows any, Visual Studio any

Minimal code to reproduce:

std::ifstream infile("cp1251.txt", std::ios::binary);
std::string data;
std::getline(infile, data);
std::filesystem::path path{ data };
auto json{ rfl::json::write(path) };

or

std::wstring utf8_to_wstring(const std::string& str) 
{
	std::wstring_convert<std::codecvt_utf8<wchar_t>> converter;
	return converter.from_bytes(str);
}

...

std::ifstream infile("utf8.txt", std::ios::binary);
std::string data;
std::getline(infile, data);
auto ws{ utf8_to_wstring(data) };
auto json{ rfl::json::write(ws) };

cp1251.txt
utf8.txt

[liuzicheng1987]

Thanks for reporting.

[liuzicheng1987]

@9241304 could you show me the exact stack trace?

[9241304]

std::filesystem::path
Crash is similar to std::string case. Looks like path::string() was called.

 	test.exe!std::_Narrow_char_traits<char,int>::length(const char * const _First) Line 430	C++
 	test.exe!std::string::basic_string<char,std::char_traits<char>,std::allocator<char>>(const char * const _Ptr) Line 745	C++
>	test.exe!rfl::json::write<,std::filesystem::path>(const std::filesystem::path & _obj, const unsigned int _flag) Line 32	C++
 	test.exe!main(int argc, char * * argv) Line 281	C++
 	test.exe!invoke_main() Line 79	C++
 	test.exe!__scrt_common_main_seh() Line 288	C++
 	test.exe!__scrt_common_main() Line 331	C++
 	test.exe!mainCRTStartup(void * __formal) Line 17	C++
 	kernel32.dll!00007ff979e7e8d7()	Unknown
 	ntdll.dll!00007ff97aa314fc()	Unknown

String contains invalid utf-8 sequences, so after writing we get nullptr here and then crash in next line

[9241304]

std::wstring case.
stack is the same both for std::filesystem::path::wstring() and utf8_to_wstring()

>	test.exe!issue_debug_notification(const wchar_t * const message) Line 28	C++
 	test.exe!__acrt_report_runtime_error(const wchar_t * message) Line 154	C++
 	test.exe!abort() Line 61	C++
 	test.exe!terminate() Line 59	C++
 	test.exe!FindHandler<__FrameHandler4>(EHExceptionRecord * pExcept, unsigned __int64 * pRN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC, FH4::FuncInfo4 * pFuncInfo, unsigned char recursive, int CatchDepth, unsigned __int64 * pMarkerRN) Line 739	C++
 	test.exe!__InternalCxxFrameHandler<__FrameHandler4>(EHExceptionRecord * pExcept, unsigned __int64 * pRN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC, FH4::FuncInfo4 * pFuncInfo, int CatchDepth, unsigned __int64 * pMarkerRN, unsigned char recursive) Line 396	C++
 	test.exe!__InternalCxxFrameHandlerWrapper<__FrameHandler4>(EHExceptionRecord * pExcept, unsigned __int64 * pRN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC, FH4::FuncInfo4 * pFuncInfo, int CatchDepth, unsigned __int64 * pMarkerRN, unsigned char recursive) Line 236	C++
 	test.exe!__CxxFrameHandler4(EHExceptionRecord * pExcept, unsigned __int64 RN, _CONTEXT * pContext, _xDISPATCHER_CONTEXT * pDC) Line 304	C++
 	test.exe!__GSHandlerCheck_EH4(_EXCEPTION_RECORD * ExceptionRecord, void * EstablisherFrame, _CONTEXT * ContextRecord, _DISPATCHER_CONTEXT * DispatcherContext) Line 73	C++
 	ntdll.dll!00007ff97aae06ff()	Unknown
 	ntdll.dll!00007ff97a9ba328()	Unknown
 	ntdll.dll!00007ff97aae003e()	Unknown
 	KernelBase.dll!00007ff977ef933a()	Unknown
 	test.exe!_CxxThrowException(void * pExceptionObject, const _s__ThrowInfo * pThrowInfo) Line 79	C++
 	test.exe!std::_Xlength_error(const char * _Message) Line 19	C++
 	test.exe!std::_Xlen_string() Line 512	C++
 	test.exe!std::string::_Reallocate_grow_by<`std::string::append'::`2'::<lambda_1>,unsigned __int64,char>(const unsigned __int64 _Size_increase, std::string::append::__l2::<lambda_1> _Fn, unsigned __int64 <_Args_0>, char <_Args_1>) Line 2994	C++
 	test.exe!std::string::append(const unsigned __int64 _Count, const char _Ch) Line 1534	C++
 	test.exe!std::string::resize(const unsigned __int64 _New_size, const char _Ch) Line 2364	C++
 	test.exe!rfl::parsing::Parser<rfl::json::Reader,rfl::json::Writer,std::wstring,rfl::Processors<>>::write<rfl::parsing::Parent<rfl::json::Writer>::Root>(const rfl::json::Writer & _w, const std::wstring & _str, const rfl::parsing::Parent<rfl::json::Writer>::Root & _parent) Line 69	C++
 	test.exe!rfl::json::write<,std::wstring>(const std::wstring & _obj, const unsigned int _flag) Line 30	C++
 	test.exe!main(int argc, char * * argv) Line 281	C++
 	test.exe!invoke_main() Line 79	C++
 	test.exe!__scrt_common_main_seh() Line 288	C++
 	test.exe!__scrt_common_main() Line 331	C++
 	test.exe!mainCRTStartup(void * __formal) Line 17	C++
 	kernel32.dll!00007ff979e7e8d7()	Unknown
 	ntdll.dll!00007ff97aa314fc()	Unknown

I didn't investigate it

[9241304]

I tried to make workaround - Local8bitToUtf8Processor and vice versa. But this solution works only on write, because of on read processor is called before structure filled. So, I haven't good solution for this case

[AndriyOm]

Hi. I had also im my struct to serialize the std::wstring with cyrillic, japan chars in wstring. Crashing on writing json file.
For now I turned to std::string and using UTF-8 encoding to save.
Reflect CPP version from appr,. 26.04.2025

P.S.
Latest VS and Win10

Remark; I am not about filepath. I am about wstring to seriallize.
Created issue: #422

[liuzicheng1987]

This seems to be an issue with YYJSON. Maybe there is a flag for this.

[9241304]

There is no flag for this. Data should be encoded as utf-8. So, it should be converted before calling yyjson

[liuzicheng1987]

@9241304 Unfortunately we cannot do this on our side. It is not possible to automatically determine the encoding. (If it were, encodings wouldn't be such a pain.)

The only thing we can do is to document this more clearly.

[9241304]

I don`t agree

1. We shouldn`t determine encoding for std::wstring and std::filesystem::path. These cases are crashing now, but they both are UTF-16
2. For std::string we also shoudnt determine anything, we should just follow platform rules. On Linux its a UTF-8, on Windows - codepage. The STL follows this rule. If we want UTF-8 anywhere, we should use std::utf8string.
   As for converting std::string to utf-8, we can use the following simple code:

std::string acpToUtf8(const std::string& input)
{
	int wsize = MultiByteToWideChar(CP_ACP, 0, input.c_str(), -1, nullptr, 0);
	std::wstring wstr(wsize, L'\0');
	MultiByteToWideChar(CP_ACP, 0, input.c_str(), -1, &wstr[0], wsize);

	int utf8size = WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), -1, nullptr, 0, nullptr, nullptr);
	std::string utf8str(utf8size, '\0');
	WideCharToMultiByte(CP_UTF8, 0, wstr.c_str(), -1, &utf8str[0], utf8size, nullptr, nullptr);

	return utf8str;
}

Vice versa

std::string utf8ToAcp(const std::string& input)
{
	int wsize = MultiByteToWideChar(CP_UTF8, 0, input.c_str(), -1, nullptr, 0);
	std::wstring wstr(wsize, L'\0');
	MultiByteToWideChar(CP_UTF8, 0, input.c_str(), -1, &wstr[0], wsize);

	int utf8size = WideCharToMultiByte(CP_ACP, 0, wstr.c_str(), -1, nullptr, 0, nullptr, nullptr);
	std::string utf8str(utf8size, '\0');
	WideCharToMultiByte(CP_ACP, 0, wstr.c_str(), -1, &utf8str[0], utf8size, nullptr, nullptr);

	return utf8str;
}

If you don`t want do this, please make possibility to convert this on the fly. Now I can do this only on json::write, but not on json::read
So, @liuzicheng1987, lib is unusable for std::filesystem::path and std::wstring on Windows and requires additional hacks for std::string, but both cases may be fixed