json_spirit: use utf8 intenally when parsing \uHHHH

[tserong]

When the python CLI is given non-ASCII characters, it converts them to
\uHHHH escapes in JSON. json_spirit parses these internally into 16 bit
characters, which could only work if json_spirit were built to use
std::wstring, which it isn't; it's using std::string, so the high byte
ends up being zero'd, leaving the low byte which is effectively garbage.

This hack^H^H^H^H change makes json_spirit convert to utf8 internally
instead, which can be stored just fine inside a std::string.

Note that this implementation still assumes \uHHHH escapes are four hex
digits, so it'll only cope with characters in the Basic Multilingual
Plane. Still, that's rather a lot more characters than it could cope
with before ;)

(For characters outside the BMP, Python seems to generate escapes in the
form \uHHHHHHHH, i.e. 8 hex digits, which the current implementation
doesn't expect to see)

Fixes: #7387

Signed-off-by: Tim Serong tserong@suse.com

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/4905/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/4905/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/4905/

[tchaikov]

@tserong looks good, but can we have a test case which reproduces the funny test in http://pastebin.com/KzWab33X ?

[tserong]

@tchaikov, sure, will write a test case and see about the cleanup you suggested above.

[tserong]

@tchaikov done. Assuming the above is OK, are you happy with it as three commits, or would you prefer I squash it back to one?

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/4989/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/4989/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/4989/

[tchaikov]

thanks @tserong =)

yeah, i'd prefer we have a single commit for this change, could you do that?

[tserong]

No problem, squashed.

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/5004/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/5004/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/5004/

[gregsfortytwo]

Looks like this busted things up a bit? http://tracker.ceph.com/issues/11574

[tchaikov]

@gregsfortytwo ack.

[tserong]

This might fix it:

diff --git a/src/json_spirit/json_spirit_reader_template.h b/src/json_spirit/json_spirit_reader_template.h
index 2eaf743..c50f885 100644
--- a/src/json_spirit/json_spirit_reader_template.h
+++ b/src/json_spirit/json_spirit_reader_template.h
@@ -79,7 +79,7 @@ namespace json_spirit
     template<>
     std::string unicode_str_to_utf8( std::string::const_iterator & begin )
     {
-        typedef typename std::string::value_type Char_type;
+        typedef std::string::value_type Char_type;

         const Char_type c1( *( ++begin ) );
         const Char_type c2( *( ++begin ) );

Although I can't say for sure, as I never had that error ("using 'typename' outside of template") in my test builds -- for me it builds fine with or without typename, actually.

[tchaikov]

@tserong yes. i just pushed the same patch to wip-11574-fix-FTBFS, seems the build on centos 6.5 is happy, see http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-rpm-rhel6-5-amd64-basic/#origin/wip-11574-fix-FTBFS .

[tchaikov]

#4614 is posted to address this FTBFS.