Expose convertCharset convenience function to controllers

[christophehenry]

Allow to convert from UTF-8 to whatever encoding the device supports

[Swiftb0y]

I think design-wise, it would make most sense to just return a QByteArray (which is automatically a JS TypedArray), since if you're converting into a different format, you'll likely want to send out the bytes soon after without manipulating them more (at least in terms of characters). Also putting it back into a QString would either convert it back to UTF-16 or create a (likely invalid, UB causing) string.

[christophehenry]

it would make most sense to just return a QByteArray

Isn't what I'm doing here?

Also putting it back into a QString

I don't understand… Where am I doing that?

[Swiftb0y]

I don't understand… Where am I doing that?

whoops, yeah, that assumption was based on the ucnv_fromUChars call that currently has its arguments swapped.

[daschuer]

Qt has a wrapper for all this. I think it should be:

Suggested change

	icu::UnicodeString unicodeString;
	UErrorCode errorCode;
	UConverter* latin9Converter = ucnv_open(targetCharset.toLocal8Bit().data(), &errorCode);
	if (!U_FAILURE(errorCode)) {
	ucnv_close(latin9Converter);
	return QJSValue::UndefinedValue;
	}
	char* result = nullptr;
	ucnv_fromUChars(latin9Converter,
	result,
	0,
	&value.data()->unicode(),
	value.length(),
	&errorCode);
	ucnv_close(latin9Converter);
	QTextCodec* codec = QTextCodec::codecForName(targetCharset);
	if (!codec) {
	return QJSValue::UndefinedValue;
	}
	QByteArray result = codec->fromUnicode(value);

(Not tested)

[JoergAtGithub]

Unfortunately this nice API was replaced in Qt6 by QStringConverter, which supports much less codecs. The old API is still available in the "Qt 5 Core Compat module" for Qt6, but I guess this is nothing that should be used in new code.

[daschuer]

Mmm interesting.

In Qt6 there is a common base class for QStringConverterBase for QStringConverter and QTextCodec
https://github.com/qt/qtbase/blob/30d90b4ccad83ab1f23dab7cd72b7e228c299895/src/corelib/text/qstringconverter.cpp#L1772

And it is also using ucnv under the hood:
https://github.com/qt/qtbase/blob/30d90b4ccad83ab1f23dab7cd72b7e228c299895/src/corelib/text/qstringconverter.cpp#L2085

Suggested change

	icu::UnicodeString unicodeString;
	UErrorCode errorCode;
	UConverter* latin9Converter = ucnv_open(targetCharset.toLocal8Bit().data(), &errorCode);
	if (!U_FAILURE(errorCode)) {
	ucnv_close(latin9Converter);
	return QJSValue::UndefinedValue;
	}
	char* result = nullptr;
	ucnv_fromUChars(latin9Converter,
	result,
	0,
	&value.data()->unicode(),
	value.length(),
	&errorCode);
	ucnv_close(latin9Converter);
	QStringEncoder encoder = QStringEncoder(targetCharset);
	if (!encoder.isValid()) {
	return QJSValue::UndefinedValue;
	}
	QByteArray result = encoder.encode(value);

[daschuer]

Our QT vcpkg is build with "icu"
https://github.com/daschuer/vcpkg/blob/5d58718e04ee9196314dc77d7cf414f36be24b65/ports/qtbase/vcpkg.json#L62

[christophehenry]

@daschuer thank you for pointing this up. If I can't use QTextCodec, there's definitely some code I can copy here.

[christophehenry]

So if I'm not mistaken, this should look like on my last commit. This, however doesn't work as expected. From JS side, convertEncoding returns an object that, printed using console.log, display the original string. Not a TypedArray. I can't figure out what's happening. I'm leaving it ofr tonight.

[Swiftb0y]

thats weird. It should work. https://doc.qt.io/qt-6/qtqml-cppintegration-data.html#qbytearray-to-javascript-arraybuffer

[Swiftb0y]

Maybe do some type introspection (instanceof) to confirm if it is an ArrayBuffer.

[christophehenry]

Ah you're right. I expected a TypedArray and thus a .forEach(). So I'd need to do new Uint8Array(midi.convertEncoding("ISO-8859-15", "Thing to display")) from JS? Is there a way to directly return a TypedArray?

[Swiftb0y]

You can technically execute some JS code in C++ that essentially does that and return the resulting QJSValue, but thats quite ugly IMO. I think it should up to the caller how they want to interpret that buffer. If thats too much typing for you, you can always create a utility function in your mapping, but lets not bake that into the API. It will cause API inconsistencies and unnecessary overhead.

[christophehenry]

Ok, so I can confirm this code works.

[daschuer]

Some nit picks.

[JoergAtGithub]

Since the supported charsets depend on the Qt build options and the ICU version, we should only allow charsets from a positive list.
The charsets on this positive list can than be probed in the CMake configuration step, to ensure that nobody builds a Mixxx version that don't support this functionality of the mapping API:

# Add a custom command to compile the code
add_custom_command(
    OUTPUT list_codecs
    COMMAND ${CMAKE_CXX_COMPILER} -o list_codecs -xc++ - <<EOF
#include <QStringConverter>
#include <QStringList>
#include <QTextStream>

int main(int argc, char* argv[]) {
    if (argc != 2) {
        return 1; // Invalid number of arguments
    }
    QString charset = argv[1];
    QStringList codecs = QStringConverter::availableCodecs();
    if (codecs.contains(charset)) {
        return 0; // Charset is available
    } else {
        return 1; // Charset is not available
    }
}
EOF
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
    COMMENT "Compiling list_codecs"
)

# List of specified charsets to check
set(SPECIFIED_CHARSETS "UTF-8;ISO-8859-1;ISO-8859-15")

# Check for each specified charset
foreach(CHARSET IN LISTS SPECIFIED_CHARSETS)
    execute_process(
        COMMAND ${CMAKE_BINARY_DIR}/list_codecs ${CHARSET}
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
        RESULT_VARIABLE RETURN_CODE
    )
    if(${RETURN_CODE} EQUAL 0)
        message(STATUS "Charset ${CHARSET} is available.")
    else()
        message(FATAL_ERROR "Charset ${CHARSET} is not available.")
    endif()
endforeach()

[Swiftb0y]

alternatively we just let the mapping handle it. Not sure whats better.

[christophehenry]

Meh. I'm not in favor of that. The idea is to allow controller developers to use charsets if available without having to open a PR to make it available. I can, however check against QStringConverter::availableCodecs() and QTextCodec::availableCodecs().

[JoergAtGithub]

Every Mixxx installation must support each mapping. The mapping API must be stable accross installations. There is no problem, to add a long list of charsets to the positive list.

[Swiftb0y]

not sure if packagers are very happy about this though. Letting the mapping handle it (disable the corresponding feature) may make more sense. It entirely depends on the list.

[daschuer]

How likely is the case that QT is build without ICU? It is a core feature of every Linux distribution and for Window an macOs we provide our own solution.

In the unlikely case a certain codec is not available we get an empty string. This is much better than a totally failed mapping.

So I guess we can be relaxed here and use an easy solution like a simple isEmpt() sanity check or such.

[christophehenry]

In the unlikely case a certain codec is not available we get an empty string

No. Current implementation returns undefined and I documented that it will return undefined if charset is unavailable. This lets mapping developers implement an alternative behavior.

[christophehenry]

Should I expose QStringConverter::availableCodecs()/QTextCodec::availableCodecs() to JS too, so that developers can chek what is available on their environment?

[JoergAtGithub]

Should I expose QStringConverter::availableCodecs()/QTextCodec::availableCodecs() to JS too, so that developers can chek what is available on their environment?

No, the API must guarantee to support the same charsets on all platforms and builds of Mixxx.

[Swiftb0y]

No, the API must guarantee to support the same charsets on all platforms and builds of Mixxx.

That'll limit us to a somewhat small common subset though. Is that a price we're willing to pay? Do we want to tell a contributor "no we can not accept your mapping because one niche feature doesn't work on one particular platform"?

[JoergAtGithub]

That'll limit us to a somewhat small common subset though.

I don't think so, if you use a Qt build with the same ICU version, you will get always the same charsets. We just need to ensure, that the configuration of the Mixxx build fails, if someone uses an inproper Qt build.

[christophehenry]

Plus, it's not as if it poses the risk of a crash. At worst, the function would return undefined and the mapping can propose an alternative behavior. If the charset becomes mainstream, we can always add it to the well-known charsets later.

[christophehenry]

Looks good for a merge to me.

[Swiftb0y]

addressing @JoergAtGithub now since this is his code.

[christophehenry]

@Swiftb0y @JoergAtGithub I'm really sorry, the ongoing discussion is way above my skills. So just tell me what to do when you agree on a correct implementation.

[Swiftb0y]

No worries. I'll create a PR to your branch. I think we settled on an implementation.

[JoergAtGithub]

No worries. I'll create a PR to your branch. I think we settled on an implementation.

@Swiftb0y Wait, I'm already on it. That's why the questions came up.

[christophehenry]

Awesome. I'll be glad if you could explain me the changes when you're done. I like to learn.

[Swiftb0y]

this would be more robust if instead of relying on an out-of-band CO, we would simply evaluate the code and use the expression result. At least this current approach seems very convoluted to me.

[christophehenry]

Dammit, this was supposed to be a simple feature. @JoergAtGithub feel free to take this to your own repo.

[Swiftb0y]

I agree. This as sprawled into an overengineered mess I'm not comfortable merging. If we want something like this, we need to go back to the drawing board.

[christophehenry]

Closing in favor of #13935