Invalid UTF-8 string after upgrade macOS to 14.4

[patanachai]

Minetest version

Minetest 5.8.0 (OSX)
Using Irrlicht 1.9.0mt13
Using LuaJIT 2.1.1700008891
BUILD_TYPE=Release
RUN_IN_PLACE=0
USE_CURL=1
USE_GETTEXT=1
USE_SOUND=1
STATIC_SHAREDIR="minetest.app/Contents/Resources"
STATIC_LOCALEDIR="minetest.app/Contents/Resources/locale"

Irrlicht device

No response

Operating system and version

macOS Sonoma 14.4

CPU model

Apple M1 Pro

GPU model

No response

Active renderer

No response

Summary

The issue happens after upgrade macOS from 14.3 to 14.4.
I tried the zip file installation and brew install minetest

Steps to reproduce

• Upgrade macOS to version 14.4
• install minetest
• launch minetest

[SmallJoker]

Relevant code:

minetest/src/util/string.cpp

Lines 73 to 135 in d5d6e36

	#elif defined(__NetBSD__) || defined(__OpenBSD__) || defined(__FreeBSD__)
	// NetBSD does not allow "WCHAR_T" as a charset input to iconv.
	#include <sys/endian.h>
	#if BYTE_ORDER == BIG_ENDIAN
	const char *DEFAULT_ENCODING = "UTF-32BE";
	#else
	const char *DEFAULT_ENCODING = "UTF-32LE";
	#endif
	#else
	const char *DEFAULT_ENCODING = "WCHAR_T";
	#endif
	std::wstring utf8_to_wide(std::string_view input)
	{
	const size_t inbuf_size = input.length();
	// maximum possible size, every character is sizeof(wchar_t) bytes
	size_t outbuf_size = input.length() * sizeof(wchar_t);
	char *inbuf = new char[inbuf_size]; // intentionally NOT null-terminated
	memcpy(inbuf, input.data(), inbuf_size);
	std::wstring out;
	out.resize(outbuf_size / sizeof(wchar_t));
	#if defined(__ANDROID__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__FreeBSD__)
	static_assert(sizeof(wchar_t) == 4, "Unexpected wide char size");
	#endif
	char *outbuf = reinterpret_cast<char*>(&out[0]);
	if (!convert(DEFAULT_ENCODING, "UTF-8", outbuf, &outbuf_size, inbuf, inbuf_size)) {
	infostream << "Couldn't convert UTF-8 string 0x" << hex_encode(input)
	<< " into wstring" << std::endl;
	delete[] inbuf;
	return L"<invalid UTF-8 string>";
	}
	delete[] inbuf;
	out.resize(outbuf_size / sizeof(wchar_t));
	return out;
	}
	std::string wide_to_utf8(std::wstring_view input)
	{
	const size_t inbuf_size = input.length() * sizeof(wchar_t);
	// maximum possible size: utf-8 encodes codepoints using 1 up to 4 bytes
	size_t outbuf_size = input.length() * 4;
	char *inbuf = new char[inbuf_size]; // intentionally NOT null-terminated
	memcpy(inbuf, input.data(), inbuf_size);
	std::string out;
	out.resize(outbuf_size);
	if (!convert("UTF-8", DEFAULT_ENCODING, &out[0], &outbuf_size, inbuf, inbuf_size)) {
	infostream << "Couldn't convert wstring 0x" << hex_encode(inbuf, inbuf_size)
	<< " into UTF-8 string" << std::endl;
	delete[] inbuf;
	return "<invalid wide string>";
	}
	delete[] inbuf;
	out.resize(outbuf_size);
	return out;
	}

I find it strange that a macOS upgrade would change the behaviour of iconv (for encoding conversion). I'd assume DEFAULT_ENCODING = "WCHAR_T" is no longer accepted. Does the test TestUtilities::testUTF8 in minetest --run-unittests fail too?

[sfan5]

Can you test the artifacts from here and here and tell us if either works?

[sfan5]

(reply here -> #14464 (comment))

[prg3]

PR from 14464 worked for me to fix this problem

[TechDudie]

can confirm issue
updated to 14.4 today