Convert entire source tree from json_spirit to UniValue #6121

jonasschnelli · 2015-05-11T08:19:02Z

Takes up and supersedes #4738.

This PR keeps the JSON Spirit "backward-compatibility" and passes all tests. Therefore there is a new UniValue number type VREAL to distinct between int and double with the current JSON spirit precision of 8 digits after point.
Because after rfc4627 (and UniValue honors that) some basic types/strings like "null" or "0" and "string" are not valid JSON strings, i had to slightly change some unit tests (will comment below directly in the code).

jonasschnelli · 2015-05-11T08:20:05Z

src/bitcoin-cli.cpp

@@ -142,8 +142,8 @@ Object CallRPC(const string& strMethod, const Array& params)
        throw runtime_error("no response from server");

    // Parse reply
-    Value valReply;
-    if (!read_string(strReply, valReply))
+    Value valReply(UniValue::VSTR);


UniValue required explicit type definition during constructing.
Value should be replaced with UniValue soon.

jonasschnelli · 2015-05-11T08:21:15Z

I did use merge instead of rebase to clearly see my adaptations from #4738.

jonasschnelli · 2015-05-11T08:25:45Z

src/rpcclient.cpp

+                if (!jVal.read(strVal) || (jVal.isNull() && strVal.size() > 0))
+                    if(!jVal.setNumStr(strVal) || jVal.isNull())
+                        throw runtime_error(string("Error parsing JSON:")+strVal);
+            }


I had to extend this to gain compatibility between JSON Spirit and UniValue. UniValue is more strict in case of JSON encoding.

jgarzik · 2015-05-11T15:15:30Z

ut ACK - reviewed the whole thing

Will do some testing later this week.

Thanks for moving this forward!

laanwj · 2015-05-12T12:48:22Z

Thanks for doing this work.

One remark: I see AmountFromValue and ValueFromAmount still make the value pass through a double. My prime gripe with JSON spirit is that monetary values still had to be converted from and to floating point which can cause deviations (see See #3759 and https://bitcoin.stackexchange.com/questions/22716/bitcoind-sendfrom-round-amount-error ).

Ideally a new JSON implementation will avoid this and can interpret and generate Numbers from/to fixed-point or decimal directly. Similar to Python's

postdata = json.dumps(list(rpc_call_list), default=EncodeDecimal)
response = json.loads(responsedata, parse_float=decimal.Decimal)

jgarzik · 2015-05-12T14:09:46Z

Agree w/ your gripe @laanwj. However, I think of that as a separate logical step:

Convert tree with as much equivalence as possible, to prove everything works as it always did.
Improve the tree beyond json-spirit equivalence.

Separated thusly it is easier to test the conversion before moving on to more ambitious cleanups.

jonasschnelli · 2015-05-18T12:08:39Z

This is now complete.

Just added another commit (2a3126c) which does completely remove JSON Spirit (also the UniValue wrapper) from the sources and the compile process.

Passes all tests.

If the diff size makes this PR to risky, there would also be the chance to pull this in without 2a3126c or / and separate 2a3126c into another PR.

I think it would be good to remove JSON Spirit in this PR while not breaking the RPC API. After this has been done, we could focus on optimizing the monetary values.

jonasschnelli · 2015-05-18T16:21:52Z

Gitian was also happy with this: http://builds.jonasschnelli.ch/pulls/6121/

jgarzik · 2015-05-18T16:51:29Z

Yes, I agree JSON-spirit should be removed in this PR.

Reviewing & testing right now...

jonasschnelli · 2015-06-01T13:36:29Z

Rebased (puhh).
Thanks in advance for reviewing this (maybe @jgarzik and @theuni) and finally bring this forward.

laanwj · 2015-06-02T07:34:28Z

(apparantly) spurious travis failure with the comparison tool, retriggering

laanwj · 2015-06-02T08:10:14Z

Looks good to me. Tested and reviewed ACK.
Would like to merge this asap now that 0.11 is branched.

laanwj · 2015-06-02T08:43:23Z

This breaks linearize-hashes.py. Likely, this is caused by JSON-RPC batching no longer working:

./linearize-hashes.py  linearize.cfg 
Traceback (most recent call last):
  File "./linearize-hashes.py", line 112, in <module>
    get_block_hashes(settings)
  File "./linearize-hashes.py", line 68, in get_block_hashes
    for x,resp_obj in enumerate(reply):
TypeError: 'NoneType' object is not iterable

(expected output: a list of block hashes)

jonasschnelli · 2015-06-02T09:20:49Z

Thanks for testing. Nice catch!
Will have a close look at the batching feature within the next hours.

jonasschnelli · 2015-06-02T09:41:28Z

RPC batching issue is fixed.

44c7474 univalue: add type check unit tests (Jonas Schnelli) c023092 univalue: add strict type checking (Wladimir J. van der Laan) 7e98a3c util: Add ParseInt64 and ParseDouble functions (Wladimir J. van der Laan) 043df2b Simplify RPCclient, adapt json_parse_error test (Wladimir J. van der Laan) 519eede fix univalue json parse tests (Jonas Schnelli) c7fbbc7 fix missing univalue types during constructing (Jonas Schnelli) 8f7e4ab fix rpc batching univalue issue (Jonas Schnelli) 9a8897f Remove JSON Spirit wrapper, remove JSON Spirit leftovers (Jonas Schnelli) 3df0411 remove JSON Spirit UniValue wrapper (Jonas Schnelli) 1f263c8 fix rpc unit test, plain numbers are not JSON compatible object (Jonas Schnelli) e04d9c2 univalue: correct bool support (Jonas Schnelli) 0c5b2cf univalue: add support for real, fix percision and make it json_spirit compatible (Jonas Schnelli) 21c10de special threatment for null,true,false because they are non valid json (Jonas Schnelli) 6c7bee0 expicit set UniValue type to avoid empty values (Jonas Schnelli) 53b4671 extend conversion to UniValue (Jonas Schnelli) 15982a8 Convert tree to using univalue. Eliminate all json_spirit uses. (Jeff Garzik) 5e3060c UniValue: export NullUniValue global constant (Jeff Garzik) efc7883 UniValue: prefer .size() to .count(), to harmonize w/ existing tree (Jeff Garzik)

sdaftuar · 2015-06-04T19:01:19Z

I'm getting an error now when running qa/rpc-tests/getblocktemplate_proposals.py:
JSONRPC error: Missing data String key for proposal

Looks like it's here:

File "./getblocktemplate_proposals.py", line 86, in assert_template
    rsp = node.getblocktemplate({'data':template_to_hex(tmpl, txlist),'mode':'proposal'})

jonasschnelli · 2015-06-04T19:45:24Z

Meh. This bug crept in during multiple transitions from json spirit objects to UniValue.
Fixed with #6234.

Thanks @sdaftuar for reporting!

jgarzik · 2015-06-04T20:10:33Z

Odd that I missed that in my testing. Anyway, re-ACK! Tested.

theuni · 2015-06-05T00:08:32Z

@jonasschnelli Sorry for not getting to a final review in time. I'll still look over it again in master for good measure. Great work!

was introduced with bitcoin#6121

strict checking of the precision of real numbers. See: https://bitcoin.org/en/release/v0.12.0#rpc-low-level-api-changes and: bitcoin/bitcoin#6121

…=<try> Convert entire source tree from json_spirit to UniValue This PR cherry-picks bitcoin/bitcoin#6121 and then migrates the Zcash-specific code to UniValue. Also cherry-picks: - bitcoin/bitcoin#6241 - bitcoin/bitcoin#6234 Closes #1985.

…=str4d Convert entire source tree from json_spirit to UniValue This PR cherry-picks bitcoin/bitcoin#6121 and then migrates the Zcash-specific code to UniValue. Also cherry-picks: - bitcoin/bitcoin#6241 - bitcoin/bitcoin#6234 Closes #1985.

was introduced with #6121

Change `read_string` to fail when not the entire input has been consumed. This avoids unexpected, even dangerous behavior (fixes bitcoin#6223). The new JSON parser adapted in bitcoin#6121 also solves this problem so in master this is a temporary fix, but should be backported to older releases. Also adds tests for the new behavior. Github-Pull: bitcoin#6226 Rebased-From: 4e157fc (cherry picked from commit 181771b)

jonasschnelli reviewed May 11, 2015
View reviewed changes

jonasschnelli force-pushed the 2015/05/unival branch from 0d57ecc to 95c9b49 Compare May 11, 2015 09:42

laanwj added RPC/REST/ZMQ REST labels May 12, 2015

laanwj mentioned this pull request May 12, 2015

Convert entire source tree from json_spirit to UniValue. #4738

Closed

laanwj mentioned this pull request May 15, 2015

libevent-based http server #5677

Merged

5 tasks

jonasschnelli force-pushed the 2015/05/unival branch from eee8acb to 2a3126c Compare May 18, 2015 12:04

jonasschnelli force-pushed the 2015/05/unival branch from 2a3126c to 2f292e2 Compare May 18, 2015 13:09

jonasschnelli mentioned this pull request May 27, 2015

rest: remove JSON input format for getutxos to avoid potential DoS problems #6192

Closed

jonasschnelli force-pushed the 2015/05/unival branch 2 times, most recently from d8f0bd1 to 8412abd Compare June 1, 2015 13:35

laanwj mentioned this pull request Jun 2, 2015

use const references where appropriate #6206

Merged

univalue: add strict type checking

c023092

jonasschnelli force-pushed the 2015/05/unival branch from 7f442e7 to c023092 Compare June 4, 2015 11:22

univalue: add type check unit tests

44c7474

laanwj merged commit 44c7474 into bitcoin:master Jun 4, 2015

jonasschnelli mentioned this pull request Jun 4, 2015

fix rpcmining/getblocktemplate univalue transition logic error #6234

Merged

jonasschnelli added a commit to jonasschnelli/bitcoin that referenced this pull request Jun 5, 2015

fix util_tests.cpp clang warnings

0fd0194

was introduced with bitcoin#6121

jonasschnelli mentioned this pull request Jun 5, 2015

fix util_tests.cpp clang warnings #6241

Merged

jonasschnelli added a commit to jonasschnelli/bitcoin that referenced this pull request Jun 6, 2015

fix util_tests.cpp clang warnings

c946ebe

was introduced with bitcoin#6121

l0rdicon mentioned this pull request Oct 10, 2015

Replace json_spirit with UniValue nochowderforyou/clams#236

Merged

coinspect mentioned this pull request Aug 26, 2016

Test Bitcoin's JSON deserialization with large amounts of data zcash/zcash#1122

Closed

dagurval mentioned this pull request Nov 12, 2016

Convert entire source tree from json_spirit to UniValue bitcoinxt/bitcoinxt#171

Merged

This was referenced Jan 5, 2017

Merge UniValue refactoring from Bitcoin zcash/zcash#1985

Closed

Convert entire source tree from json_spirit to UniValue zcash/zcash#1990

Merged

AllanDoensen referenced this pull request in AllanDoensen/BitcoinUnlimited Apr 23, 2017

fix util_tests.cpp clang warnings

2b81a4a

was introduced with #6121

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert entire source tree from json_spirit to UniValue #6121

Convert entire source tree from json_spirit to UniValue #6121

jonasschnelli commented May 11, 2015

jonasschnelli May 11, 2015

jonasschnelli commented May 11, 2015

jonasschnelli May 11, 2015

jgarzik commented May 11, 2015

laanwj commented May 12, 2015

jgarzik commented May 12, 2015

jonasschnelli commented May 18, 2015

jonasschnelli commented May 18, 2015

jgarzik commented May 18, 2015

jonasschnelli commented Jun 1, 2015

laanwj commented Jun 2, 2015

laanwj commented Jun 2, 2015

laanwj commented Jun 2, 2015

jonasschnelli commented Jun 2, 2015

jonasschnelli commented Jun 2, 2015

sdaftuar commented Jun 4, 2015

jonasschnelli commented Jun 4, 2015

jgarzik commented Jun 4, 2015

theuni commented Jun 5, 2015

Convert entire source tree from json_spirit to UniValue #6121

Convert entire source tree from json_spirit to UniValue #6121

Conversation

jonasschnelli commented May 11, 2015

jonasschnelli May 11, 2015

Choose a reason for hiding this comment

jonasschnelli commented May 11, 2015

jonasschnelli May 11, 2015

Choose a reason for hiding this comment

jgarzik commented May 11, 2015

laanwj commented May 12, 2015

jgarzik commented May 12, 2015

jonasschnelli commented May 18, 2015

jonasschnelli commented May 18, 2015

jgarzik commented May 18, 2015

jonasschnelli commented Jun 1, 2015

laanwj commented Jun 2, 2015

laanwj commented Jun 2, 2015

laanwj commented Jun 2, 2015

jonasschnelli commented Jun 2, 2015

jonasschnelli commented Jun 2, 2015

sdaftuar commented Jun 4, 2015

jonasschnelli commented Jun 4, 2015

jgarzik commented Jun 4, 2015

theuni commented Jun 5, 2015