Moving to a pure Unicode text setup

[asarium]

A clarification up front: No, the old tables will of course not be deprecated. I plan on enabling Unicode mode with a mod table switch or enabling it by default if a recent enough FSO version is targetted.

This was prompted by the discussion in the forum thread about special characters in strings. While I fixed the bug reported in the thread I realized that the current mix of TTF and Bitmap glyphs simply doesn't work. The text rendering functions already support rendering UTF-8 encoded Unicode strings since NanoVG supports it but since we need to support the old tables which don't use UTF-8 we can't use that yet.

So, to fix that we would "simply" need to convert all our table text reading to support UTF-8 encoded strings and then pass that to the text rendering functions. Fortunately, most tables and mission files are already properly encoded since the use ASCII encoding. Only the strings and tstrings tables currently use the special FSO text encoding for special characters so modders would only need to convert those files.

Thoughts?

[Goober5000]

I have a branch where I'm working on converting parselo.cpp to use SCP_string and string iterators. It's between 75% - 85% complete. I can push that for others to take a look if they want. I started working on it because I figured that it would be a simple matter to enable parsing of Unicode characters once that step was complete.

[asarium]

Using SCP_strings in the parsing could is obviously a good idea but I don't think it's required for Unicode support since normal char strings can handle UTF-8 data just fine.