from_json<std::wstring> is treated as an array on latest MSVC

[Convery]

What is the issue you have?

When using the get method with a std::wstring/std::u8string etc. the from_json(const BasicJsonType& j, ConstructibleArrayType& arr) instantiation is selected. As the basic_json value is correctly identified as a string, the from_json will throw an error. std::string still works as expected though.

Can you provide a small but working code example?

void Foo()
{
    const auto Test = nlohmann::json::object({ {"String", "String"} });
    assert(Test["String"].is_string());

    try { Test["String"].get<std::wstring>(); }
    catch (const std::exception &e)
    {
        // [json.exception.type_error.302] type must be array, but is string
        printf("%s\n", e.what());
    }
}

Which compiler and operating system are you using?

• Compiler: MSVC 16.7.7
• Operating system: Windows 7 x64

Which version of the library did you use?

• latest release version 3.9.1
• other release - please state the version: ___
• the develop branch

[nlohmann]

std::wstring is not really supported - for this, we would need code to internally convert a std::string to std::wstring. PRs welcome!

std::u8string is C++20, whereas this library targets C++11.

[Convery]

Ah, sorry about that. Assumed that since #656 there was general support for basic_string<T> conversions. Would the correct way to deal with UTF8 be A or B in this example?

auto J = nlohmann::json::object( { { "String", u8"€" } });
auto A = J["String"].get<std::string>();
auto B = J["string"].get<std::u8string>();

[nlohmann]

The library stores strings internally as std::string and assumes UTF-8 encoding. Anything you put in will be converted to std::string - the encoding is not changed. Anything you get out will just be cast. So (A) would be fine, and (B) would be something comparable to static_cast<std::string>(...).

[Teles1]

I also have the same problem. My Json is in wstring for compatibility with Portuguese letters é, Á, ã, ç, etc. It gets automatically casted to std::string and I couldn't find a way around this issue.
I tried converting my Json to the following => [ "\u00E9", "Teles", "key_value" ] but still the same issue. The library converts it internally to std::string

[nlohmann]

@Teles1 Can you share a complete example?

[RyanShear]

@nlohmann, I wrote a test to show @Teles1 issue (since I stumbled upon the same thing yesterday):

        std::wstring wideStringTest = L"Test of wide string";
        std::string stringTest = "Test of regular std string";
        nlohmann::json outgoingJson = nlohmann::json::object(
        {
            {"TestData",
            {
                {"std::string", stringTest},
                {"std::wstring", wideStringTest}
            }}
        });

        std::cout << outgoingJson.dump(4);

and the output is:

{
    "TestData": {
        "std::string": "Test of regular std string",
        "std::wstring": [
            84,
            101,
            115,
            116,
            32,
            111,
            102,
            32,
            119,
            105,
            100,
            101,
            32,
            115,
            116,
            114,
            105,
            110,
            103
        ]
    }

[nlohmann]

Yes, this behavior is known.

[nlohmann]

I added an FAQ entry: https://json.nlohmann.me/home/faq/#wide-string-handling

[UltraEngine]

Why not just get/set a wide string as an integer array? Conversion to Utf8 is unreliable. I have seen this fail. Wide strings are used in C++ programs because they can be used with string parsing and modification functions, whereas Utf8 does not work well for this.

We already have a JsonSetWString(nlohmann::json, std::wstring) function but it's awkward and I hate to explain it to the end users. Why not just build this into the library, have it 100% working correctly, and be done with it?

std::wstring JSONGetWString(const nlohmann::json& j3)
{
        if (!j3.is_array()) return L"";
        std::wstring s;
        s.reserve(j3.size());
        for (int n = 0; n < j3.size(); ++n)
        {
                if (!j3[n].is_number_integer()) return L"";
                s.push_back(j3[n]);
        }
        return s;
}

[UltraEngine]

I solved the problem by adding a custom constructor for our String and WString class:

	String::String(nlohmann::json& j3)
	{
		if (j3.is_string())
		{
			assign(j3);
			return;
		}
		else if (j3.is_number_float())
		{
			double f = j3;
			(*this).assign(String(f));
		}
		else if (j3.is_number_unsigned())
		{
			uint64_t i = j3;
			(*this).assign(String(i));
		}
		else if (j3.is_number_integer())
		{
			int64_t i = j3;
			(*this).assign(String(i));
		}
	}

        WString::WString(nlohmann::json& j3)
        {
	        if (j3.is_array())
	        {
		        for (int n = 0; n < j3.size(); ++n)
		        {
			        if (not j3[n].is_number_unsigned())
			        {
				        clear();
				        return;
			        }
			        (*this) += WChr(j3[n]);
		        }
	        }
	        else
	        {
		        String s = String(j3);
		        if (not s.empty()) assign(WString(s));
	        }
        }

It appears that std::wstring gets treated as an array when written to a json object, and it is possible to convert the wide string to UTF-8 if saving a string is preferred, I'm satisfied with the behavior now.

[UtkuBulkan]

        WString::WString(nlohmann::json& j3)
        {
	        if (j3.is_array())
	        {
		        for (int n = 0; n < j3.size(); ++n)
		        {
			        if (not j3[n].is_number_unsigned())
			        {
				        clear();
				        return;
			        }
			        (*this) += WChr(j3[n]);
		        }
	        }
	        else
	        {
		        String s = String(j3);
		        if (not s.empty()) assign(WString(s));
	        }
        }

Where shall we exactly put this ?

For instance, this is where my code is failing :

   ` try{ txt_strct->data.text_info->text = new std::wstring(txt["context"].get<std::string>()); } catch(json::type_error& te){ VI_FATAL("{}, {}, {}", te.what(), __FILE__, __LINE__); exit(0); }`

With this error :

   ` [10:57:59] APP: [json.exception.type_error.302] type must be array, but is string, /app/src/coms/json_parser.cpp, 160`

[gregmarr]

Are you sure that's the line that's causing the error? That error doesn't make any sense there. type must be array, but is string means that it's actually a string but you're trying to get it as an array, which is not what .get<std::string>() does.

[UtkuBulkan]

This is the code that I have been using :

std::u16string* text;

try{ text = new std::u16string(txt["context"].get<std::u16string>()); } catch(json::type_error& te){ VI_FATAL("{}, {}, {}", te.what(), __FILE__, __LINE__); exit(0); }

And this is the error that I have gotten :
[14:51:47] APP: [json.exception.type_error.302] type must be array, but is string, /app/src/coms/json_parser.cpp, 160

And here is the json :

{
    "4bcfadb7-c345-474c-aaaf-65734b14f1e6": {
        "fps": 24,
        "text": {
            "4e50c16f-a944-4adc-9d78-74cca4ba1cb9": {
                "alpha": 1,
                "groups": "4e50c16f-a944-4adc-9d78-74cca4ba1cb9",
                "context": "A ŞĞÜİ B",
                
                ...

[gregmarr]

Okay, so that makes more sense than the previous code. The JSON contains a string, and getstd::u16string looks for an array of characters. I'm not sure if there is a way to make this work for JSON that contains strings with characters that are not UTF-8.

[UtkuBulkan]

So per my understanding,

Wth the current state of the library, it is impossible to read ŞĞÜİ letters while using the nlohmann json library.

Is this correct ?