Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

ofTrueTypeFont support for iso8859-1 on utf-8 encoded files #1539

Merged
merged 8 commits into from

6 participants

@arturoc
Owner

ofTrueTypeFont as it is right now has support for all the characters in iso8859-1 if the flag bFullCharacterSet is set to true when loading, which can render all the extended characters in the latin alphabet like ñàäáâç...

the problem is that most IDEs use utf-8 encoding by default so when writing this characters in strings in the code or text files they fail to render because the encoding freetype is expecting is iso8859-1.

this converts utf-8 to iso8859-1 which solves the problem of rendering special characters for the latin alphabet. might be good to test in code from other ide's in case the default encoding is not utf-8

@damiannz

i have an app where i force latin-1 encoding in a source file (Xcode) using \xyyy format. i suspect this is going to break that...

@damiannz

although i could be wrong. it's very welcome to have oF be more aware of character encoding in any case! nice work.

@arturoc
Owner

it shouldn't break except for  and A with a ~ on top, but still this is kind of a hack, perhaps we should be able to specify the default encoding for strings and set it to utf-8? having a method like:

font.setEncoding(OF_ENCODING_UTF-8)

and by now we can support utf-8 and iso-8859-1 (latin 1)

@damiannz

+1 being able to properly set encoding.

actually when i was working with ofTrueTypeFont earlier this year i thought that there should be ofTrueTypeFont::drawLatin1String and ofTrueTypeFont::drawUTF8String

@arturoc arturoc ofTrueTypeFont: add getEncoding/setEncoding
ofConstants: add encoding enum
ofUTF8ToISO88591: faster
154874f
libs/openFrameworks/utils/ofUtils.cpp
@@ -467,6 +467,22 @@ string ofBinaryToString(const string& value) {
}
//--------------------------------------------------
+string ofUTF8ToISO8859_1(const string & utf8){
@bakercp Collaborator
bakercp added a note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@bakercp
Collaborator

I did a bunch of work on this in ofxUnicode / ofxFont to wrap Poco's text converters. Snippets below. I ended up moving away from it in favor of ICU based unicode support -- mainly because poco does not support bidirectional or UTF8 iterators, and more complex encoding support. You can find more on the ICU branch here: https://github.com/bakercp/ofxUnicode/

Also you might check out the ofCharacterSet integration into ofxFont. Anyway, all of this is good progress in the right direction!

enum ofTextEncoding {
    OF_TEXT_ENCODING_UTF8 = 0        , // variable width encoding (http://en.wikipedia.org/wiki/UTF-8) backward compatible  w/ ASCII
    OF_TEXT_ENCODING_UTF16           , // 16-bit multi-byte (http://en.wikipedia.org/wiki/UTF-16)
    OF_TEXT_ENCODING_ASCII           , // 7-bit ASCII text encoding (http://en.wikipedia.org/wiki/ASCII)
    OF_TEXT_ENCODING_LATIN_1         , // 8-bit single-byte - (http://en.wikipedia.org/wiki/ISO/IEC_8859-1)
    OF_TEXT_ENCODING_LATIN_9         , // 8-bit single-byte - (http://en.wikipedia.org/wiki/ISO/IEC_8859-15), western chars + €
    OF_TEXT_ENCODING_WINDOWS_1252      // Superset of Latin 1 (ISO 8859-1) http://en.wikipedia.org/wiki/Windows-1252
};
//------------------------------------------------------------------
string ofTextConverter::convert(const ofBuffer& buffer, ofTextEncoding inputEncoding, ofTextEncoding outputEncoding) {
    return convert(buffer.getBinaryBuffer(), buffer.size(), inputEncoding, outputEncoding);
}

//------------------------------------------------------------------
string ofTextConverter::convert(const void* source, int length,  ofTextEncoding inputEncoding, ofTextEncoding outputEncoding) {
    string output;
    Poco::TextEncoding& ie = getTextEncoding(inputEncoding);
    Poco::TextEncoding& oe = getTextEncoding(outputEncoding);
    Poco::TextConverter converter(ie,oe);
    converter.convert(source, length, output);
    return output;
}

//------------------------------------------------------------------
string ofTextConverter::convert(const string& input, ofTextEncoding inputEncoding, ofTextEncoding outputEncoding) {
    // this returns a std::string -- which is just a sequence of bytes.
    // the way those bytes is interpreted is up to the user.

    // pass through
    if(inputEncoding == outputEncoding) return input;

    string output;
    Poco::TextEncoding& ie = getTextEncoding(inputEncoding);
    Poco::TextEncoding& oe = getTextEncoding(outputEncoding);
    Poco::TextConverter converter(ie,oe);
    converter.convert(input, output);
    return output;
}

//------------------------------------------------------------------
Poco::TextEncoding& ofTextConverter::getTextEncoding(ofTextEncoding enc) {
    if(enc == OF_TEXT_ENCODING_UTF8) {
        static Poco::UTF8Encoding utf8_enc;
        return utf8_enc;
    } else if(enc == OF_TEXT_ENCODING_UTF16) {
        static Poco::UTF16Encoding utf16_enc;
        return utf16_enc;
    } else if(enc == OF_TEXT_ENCODING_ASCII) {
        static Poco::ASCIIEncoding ascii_enc;
        return ascii_enc;
    } else if(enc == OF_TEXT_ENCODING_LATIN_1) {
        static Poco::Latin1Encoding latin1_enc;
        return latin1_enc;
    } else if(enc == OF_TEXT_ENCODING_LATIN_9) {
        static Poco::Latin9Encoding latin9_enc;
        return latin9_enc;
    } else if(enc == OF_TEXT_ENCODING_WINDOWS_1252) {
        static Poco::Windows1252Encoding windows1252_enc;
        return windows1252_enc;
    } else {
        ofLogWarning("ofTextConverter") << "getEncoding - unknown encoding, returning utf8.");
        static Poco::UTF8Encoding utf8_enc;
        return utf8_enc;
    }
}
@arturoc
Owner

wow, thanks, i've moved it to use poco internally in ofTrueTypeFont, but those functions seem really useful. actually freetype uses utf-32 which poco lacks of so if we want to give support for other alphabets will need to find some other library or add that conversion to poco

@bakercp
Collaborator

Yes, my mod of ofFont is utf-32 aware (it iterates through UTF8 strings) but it is not currently a drop-in replacement for ofTrueTypeFont. I'm currently separating out text formatting / layout from the ofFont class and building from (ofBaseFont)[https://github.com/bakercp/ofxFont/blob/master/src/ofBaseFont.h]. There are other, more flexible functions that do layout. That UTF8 iteration process is possible with POCO::TextIterator, but as I mentioned earlier, it is limited to one direction w/o random access, reverse iteration, etc. Anyway, I'm continuing work on ofFont -- it's kind of a ground-up reengineer. Keep an eye on it! It's coming soon.

@bakercp
Collaborator

Nice mods! Eventually I think oF should have a suite of charset conversions in some sort of ofTextUtilities file (it's happening here currently - https://github.com/bakercp/ofxUnicode/blob/ICU/src/ofTextUtilities.cpp), so this seems like a really good move for the time being.

@thiagohersan

fwiw, I'm using this in a project to render tweets in spanish.

so far so good.

@bakercp
Collaborator

Unless there are any disputes, I'll merge this in the next day or two. As mentioned previously in the thread, many of these features will be brought into the upcoming ofFont updates, but for now this seems to be a solid, pragmatic solution.

@kylemcdonald

i'm going to check out the conflicts here and try to merge this.

@kylemcdonald kylemcdonald merged commit d9caec4 into from
@yty

Can support Asian encoded do? I tested not. .

@bakercp
Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
39 libs/openFrameworks/graphics/ofTrueTypeFont.cpp
@@ -11,6 +11,10 @@
#include "ofUtils.h"
#include "ofGraphics.h"
+#include "Poco/TextConverter.h"
+#include "Poco/UTF8Encoding.h"
+#include "Poco/Latin1Encoding.h"
+#include "Poco/Latin9Encoding.h"
static bool printVectorInfo = false;
static int ttfGlobalDpi = 96;
@@ -205,6 +209,7 @@ ofTrueTypeFont::ofTrueTypeFont(){
//visibleBorder = 2;
stringQuads.setMode(OF_PRIMITIVE_TRIANGLES);
binded = false;
+ encoding = OF_ENCODING_UTF8;
}
//------------------------------------------------------------------
@@ -278,6 +283,7 @@ bool ofTrueTypeFont::loadFont(string _filename, int _fontSize, bool _bAntiAliase
return false;
}
+
FT_Set_Char_Size( face, fontSize << 6, fontSize << 6, dpi, dpi);
lineHeight = fontSize * 1.43f;
@@ -286,7 +292,7 @@ bool ofTrueTypeFont::loadFont(string _filename, int _fontSize, bool _bAntiAliase
//ofLog(OF_LOG_NOTICE,"FT_HAS_KERNING ? %i", FT_HAS_KERNING(face));
//------------------------------------------------------
- nCharacters = bFullCharacterSet ? 256 : 128 - NUM_CHARACTER_TO_START;
+ nCharacters = (bFullCharacterSet ? 256 : 128) - NUM_CHARACTER_TO_START;
//--------------- initialize character info and textures
cps.resize(nCharacters);
@@ -300,11 +306,14 @@ bool ofTrueTypeFont::loadFont(string _filename, int _fontSize, bool _bAntiAliase
long areaSum=0;
+
//--------------------- load each char -----------------------
for (int i = 0 ; i < nCharacters; i++){
//------------------------------------------ anti aliased or not:
- err = FT_Load_Glyph( face, FT_Get_Char_Index( face, (unsigned char)(i+NUM_CHARACTER_TO_START) ), FT_LOAD_DEFAULT );
+ int glyph = (unsigned char)(i+NUM_CHARACTER_TO_START);
+ if (glyph == 0xA4) glyph = 0x20AC; // hack to load the euro sign, all codes in 8859-15 match with utf-32 except for this one
+ err = FT_Load_Glyph( face, FT_Get_Char_Index( face, glyph ), FT_LOAD_DEFAULT );
if(err){
ofLog(OF_LOG_ERROR,"ofTrueTypeFont::loadFont - Error with FT_Load_Glyph %i: FT_Error = %d", i, err);
@@ -500,6 +509,14 @@ bool ofTrueTypeFont::loadFont(string _filename, int _fontSize, bool _bAntiAliase
return true;
}
+ofTextEncoding ofTrueTypeFont::getEncoding() const {
+ return encoding;
+}
+
+void ofTrueTypeFont::setEncoding(ofTextEncoding _encoding) {
+ encoding = _encoding;
+}
+
//-----------------------------------------------------------
bool ofTrueTypeFont::isLoaded() {
return bLoadedOk;
@@ -606,6 +623,12 @@ void ofTrueTypeFont::drawChar(int c, float x, float y) {
//-----------------------------------------------------------
vector<ofTTFCharacter> ofTrueTypeFont::getStringAsPoints(string str){
+ if(bFullCharacterSet && encoding==OF_ENCODING_UTF8){
+ string o;
+ Poco::TextConverter(Poco::UTF8Encoding(),Poco::Latin9Encoding()).convert(str,o);
+ str=o;
+ }
+
vector<ofTTFCharacter> shapes;
if (!bLoadedOk){
@@ -752,6 +775,12 @@ void ofTrueTypeFont::drawString(string c, float x, float y) {
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
texAtlas.draw(0,0);*/
+ if(bFullCharacterSet && encoding==OF_ENCODING_UTF8){
+ string o;
+ Poco::TextConverter(Poco::UTF8Encoding(),Poco::Latin9Encoding()).convert(c,o);
+ c=o;
+ }
+
if (!bLoadedOk){
ofLog(OF_LOG_ERROR,"ofTrueTypeFont::drawString - Error : font not allocated -- line %d in %s", __LINE__,__FILE__);
return;
@@ -859,6 +888,12 @@ void ofTrueTypeFont::drawStringAsShapes(string c, float x, float y) {
return;
}
+ if(bFullCharacterSet && encoding==OF_ENCODING_UTF8){
+ string o;
+ Poco::TextConverter(Poco::UTF8Encoding(),Poco::Latin9Encoding()).convert(c,o);
+ c=o;
+ }
+
GLint index = 0;
GLfloat X = 0;
GLfloat Y = 0;
View
4 libs/openFrameworks/graphics/ofTrueTypeFont.h
@@ -70,6 +70,9 @@ class ofTrueTypeFont{
void bind();
void unbind();
+ ofTextEncoding getEncoding() const;
+ void setEncoding(ofTextEncoding encoding);
+
protected:
bool bLoadedOk;
bool bAntiAliased;
@@ -109,6 +112,7 @@ class ofTrueTypeFont{
GLboolean blend_enabled;
GLboolean texture_2d_enabled;
#endif
+ ofTextEncoding encoding;
void unloadTextures();
void reloadTextures();
};
View
5 libs/openFrameworks/utils/ofConstants.h
@@ -541,3 +541,8 @@ enum ofDrawBitmapMode{
OF_BITMAPMODE_MODEL,
OF_BITMAPMODE_MODEL_BILLBOARD
};
+
+enum ofTextEncoding{
+ OF_ENCODING_UTF8,
+ OF_ENCODING_ISO_8859_15
+};
View
1  libs/openFrameworks/utils/ofUtils.h
@@ -173,6 +173,7 @@ char ofBinaryToChar(const string& value);
float ofBinaryToFloat(const string& value);
string ofBinaryToString(const string& value);
+
string ofGetVersionInfo();
void ofSaveScreen(string filename);
Something went wrong with that request. Please try again.