Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

* Fix JSON unicode handling #45

Merged
merged 2 commits into from

2 participants

Rangel Reale Aleksandar Fabijanic
Aleksandar Fabijanic
Owner

Please add a test case for this change and verify/confirm that it does not break any existing unit tests.

Rangel Reale
Collaborator

I'm not an Unicode expert, please review if the test case I made is correct.
There were no errors on the JSON tests.

Aleksandar Fabijanic aleks-f merged commit dcf92dc into from
Aleksandar Fabijanic aleks-f referenced this pull request from a commit
Aleksandar Fabijanic aleks-f Revert "Merge pull request #45 from RangelReale/jsonunicode"
This reverts commit dcf92dc, reversing
changes made to 79838aa.
52867ed
Aleksandar Fabijanic
Owner

I reverted this. The pull tests do pass but it's somewhat of a half-solution because it does not work for the OP case - the string ends up being internally converted into ???. It probably has to do with the current system code page but I have no time right now, maybe Franky can review it and come up with a more comprehensive set of tests.

Aleksandar Fabijanic aleks-f referenced this pull request from a commit
Aleksandar Fabijanic aleks-f Revert "Revert "Revert "Merge pull request #45 from RangelReale/jsonu…
…nicode"""

This reverts commit 86647bd.
26a480d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
26 JSON/src/Parser.cpp
View
@@ -38,6 +38,7 @@
#include "Poco/JSON/JSONException.h"
#include "Poco/Ascii.h"
#include "Poco/Token.h"
+#include "Poco/UnicodeConverter.h"
#undef min
#undef max
#include <limits>
@@ -140,28 +141,28 @@ class StringToken: public Token
switch(c)
{
case '"' :
- c = '"';
+ _value += '"';
break;
case '\\' :
- c = '\\';
+ _value += '\\';
break;
case '/' :
- c = '/';
+ _value += '/';
break;
case 'b' :
- c = '\b';
+ _value += '\b';
break;
case 'f' :
- c = '\f';
+ _value += '\f';
break;
case 'n' :
- c = '\n';
+ _value += '\n';
break;
case 'r' :
- c = '\r';
+ _value += '\r';
break;
case 't' :
- c = '\t';
+ _value += '\t';
break;
case 'u' : // Unicode
{
@@ -196,7 +197,11 @@ class StringToken: public Token
{
throw JSONException("Invalid unicode");
}
- c = unicode;
+ //unicode to utf8
+ std::string utf8;
+ UnicodeConverter::toUTF8((const UTF32Char*)&unicode,1,utf8);
+ _value += utf8;
+
break;
}
default:
@@ -204,8 +209,9 @@ class StringToken: public Token
throw JSONException(format("Invalid escape '%c' character used", (char) c));
}
}
+ }else{
+ _value += c;
}
- _value += c;
c = istr.get();
}
30 JSON/testsuite/src/JSONTest.cpp
View
@@ -859,6 +859,35 @@ void JSONTest::testTemplate()
}
+void JSONTest::testUnicode()
+{
+ const unsigned char supp[] = {0x61, 0xE1, 0xE9, 0x78, 0xED, 0xF3, 0xFA, 0x0};
+ std::string text((const char*) supp);
+
+ std::string json = "{ \"test\" : \"a\u00E1\u00E9x\u00ED\u00F3\u00FA\" }";
+ Parser parser;
+
+ Var result;
+ try
+ {
+ DefaultHandler handler;
+ parser.setHandler(&handler);
+ parser.parse(json);
+ result = handler.result();
+ }
+ catch(JSONException& jsone)
+ {
+ std::cout << jsone.message() << std::endl;
+ assert(false);
+ }
+ assert(result.type() == typeid(Object::Ptr));
+
+ Object::Ptr object = result.extract<Object::Ptr>();
+ Var test = object->get("test");
+ assert(test.convert<std::string>() == text);
+}
+
+
std::string JSONTest::getTestFilesPath(const std::string& type)
{
std::ostringstream ostr;
@@ -918,6 +947,7 @@ CppUnit::Test* JSONTest::suite()
CppUnit_addTest(pSuite, JSONTest, testValidJanssonFiles);
CppUnit_addTest(pSuite, JSONTest, testInvalidJanssonFiles);
CppUnit_addTest(pSuite, JSONTest, testTemplate);
+ CppUnit_addTest(pSuite, JSONTest, testUnicode);
return pSuite;
}
1  JSON/testsuite/src/JSONTest.h
View
@@ -76,6 +76,7 @@ class JSONTest: public CppUnit::TestCase
void testInvalidJanssonFiles();
void testTemplate();
void testItunes();
+ void testUnicode();
void setUp();
void tearDown();
Something went wrong with that request. Please try again.