Error: JSON-RPC error: JSON decode error: invalid type: map, expected a sequence

[jkugler]

Howdy! Just found your project. Looking forward to using it.

I set up bitcoind, and it's running. The following curl command works:

$ curl --user user --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getblockchaininfo", "params": [] }'  -H 'content-type: text/plain;' http://127.0.0.1:8332/
Enter host password for user 'user':
{"result":{"chain":"main","blocks":618983,"headers":618983,"bestblockhash":"0000000000000000000a9aae670c69c9874d6006065edf73a2c26af7c878602b","difficulty":15486913440292.87,"mediantime":1582661642,"verificationprogress":0.9999986054933675,"initialblockdownload":false,"chainwork":"00000000000000000000000000000000000000000d076aae823919dd1c7f5b68","size_on_disk":300418549537,"pruned":false,"softforks":{"bip34":{"type":"buried","active":true,"height":227931},"bip66":{"type":"buried","active":true,"height":363725},"bip65":{"type":"buried","active":true,"height":388381},"csv":{"type":"buried","active":true,"height":419328},"segwit":{"type":"buried","active":true,"height":481824}},"warnings":""},"error":null,"id":"curltest"}

My .env looks like this:

DATABASE_URL=postgres://jkugler:PASS@localhost/bitcoin-indexer
NODE_RPC_URL=http://user:PASS@localhost:8332

But when I run cargo run --release --bin bitcoin-indexer it fails with:

    Finished release [optimized] target(s) in 0.13s
     Running `target/release/bitcoin-indexer`
Error: JSON-RPC error: JSON decode error: invalid type: map, expected a sequence

Any suggestions?

I am using bitcoin-core 0.19.0.1. What version was used with this project? I'm really looking forward to using this...just trying to get it up and running.

[jkugler]

I've tried digging through the source to find the source of this error. The backtrace isn't helpful either. The error doesn't print out which definition in rust-bitcoincore-rpc/json/src/lib.rs failed, nor the text it was trying to deserialize, so I'm quite at a loss.

[jkugler]

I downgraded to bitcoin-core 0.18 and it appears to be running.

[j1warren]

@jkugler did you find out how to fix this?
Downgrading is ok as a workaround, but it is still a bug.

[jkugler]

@j1warren I did not fix this. I assume the structure of the JSON reply changed between versions and the indexer has not been updated to use that.

[dpc]

RPC communication is handled by bitcoincore-rpc crate. I'm aware that there were multiple fixes in the latest/next version of it. If anyone is interested, you can just bump the version and see if it helps. I'm happy to accept a PR. Otherwise it will have to wait for me to get to it.

[dpc]

That would mean main bitcoin dependency needs to be upgraded as well, so the bitcoin_hashes common dependency is at the same version. :)

[dpc]

I've rebased everything and pushed devel branch. I have not tested it though, but it compiles so it probably works. :)

[j1warren]

Seems to be working.
@dpc thank you so much for spending your time!

[dpc]

@j1warren Not a problem. You will have to pay it back. Let me know after a while if everything went OK. Then I'll land it to master. :)

[j1warren]

@dpc
Ok, seems like the most of the work is done, but I'm still stuck.

[2020-05-16T12:10:30Z INFO bitcoin_indexer::db::pg] Entering normal mode: all indices
[2020-05-16T12:10:34Z INFO bitcoin_indexer::db::pg] Block 630429H fully indexed and commited; 11block/s; 23011tx/s
...
[2020-05-16T12:10:58Z INFO bitcoin_indexer::db::pg] Block 630547H fully indexed and commited; 3block/s; 9448tx/s
[2020-05-16T12:10:58Z INFO bitcoin_indexer::db::pg] Adjusting schema to mode: normal

So I guess block 630547H was the Height when the indexer started, so after it downloaded it, it changed the DB mode from bulk to normal and postgres started indexing.

[2020-05-17T11:22:31Z INFO bitcoin_indexer::util] Bottleneck: database
Block 630548H: 00000000000000000010e914c3eaa4f3a33762d13cda3d5c2af10f92b244c5a4
[2020-05-17T11:41:07Z INFO bitcoin_indexer::db::pg] Block 630549H fully indexed and commited; 0block/s; 5tx/s

And here it has completed indexing, probably saw that there are new blocks already and started to process them in normal mode now. A couple of questions:

1. what happens if I now restart indexer? will it continue to work from where it was left in the normal DB mode with this low speed?
2. when the address_balances table is gonna be created?

[dpc]

Yes, initial building indices on my machine was taking ~5 hours, after indexing itself which only took 4 hours. :)

The initial indexing is optimized for performance, after that the performance is not as important anymore. Inserting data alone is cheap, updating all the indices is much more costly. Though on my box indexing afterwards is still quite snappy. 5tx/s looks really low. I was getting such numbers only when I was testing with the db stored on my SMR HDD. I suspect the SMR part was causing it since it causes huge write-amplification on small writes (like b-tree node updates). And afaik most modern consumer HDDs are SMRs, which really sucks for this type of applications. So if you're trying it on HDD - it might not fly. On a non-SMR HDD, I hope updates will be fast enough to keep up with the chain as it grows. But I never tried.

You could try disabling some indices, which would give some linear speedup, but that will make a lot of useful queries impractical.

You can restart indexer with no fear. Everything is using db transactions, so db takes care of consistency. When runninig indexer will always be trying to catch up.

address_balances is actually a unmaterialized VIEW, that sums all the outputs, and all inputs of an address and then substracts one from the other. It should already be there.

[j1warren]

I've tried to use views, restarted indexer, etc (didn't try mempool scrubber), it works.

My HDD despite being non-SMR (and trying to enhance shared_memory and other values in psql settings) still gives only 4-8 tx/s, so for every 1 new block it processes 2 blocks, slowly catching up.
But I think if you turn indexer off for 1 week+, then it's makes more sense to wipe the DB and start all over in bulk mode.

@dpc how do you think, with current schema is it feasible to search the addresses by their "value"?

[dpc]

@j1warren Thanks for all that info.

So it's an interesting data-point. On my SMR-HDD I remember that it took about 2 blocks (20 minutes) to index one block, so I gave up completely at this point.

There is a lot of file-system level and db-level tuning that you could potentially do, especially if you're OK with just rebuilding everything if it ever crashes. Some examples:

• disable Postgresql WAL: https://blog.4xxi.com/sacrificing-resilience-for-performance-with-postgresql-wal-unlogged-tables-d8db7253160b?gi=6bd7129d13b9 "Various sources (1, 2) suggest that expecting 15x performance boost is not unrealistic." YMMV
• similar tweaks on the file-system level, like disabling write-barriers ( https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/writebarrieronoff ); make sure you're not running on any CoW filesystem

Right now indexer does not store balances - only UTXO events. Which is fast if you ask about one address, but requires quite a bit of work if you want to do it on many addresses. I did big aggregated queries on whole UTXO set and after about an hour I'd get an answer (that's on fast SSDs, though!). If you're thinking about doing many such queries, you could build an index to speed this up, or some form of a cache, or modify the indexer itself, or write an additional application that follows events produced by indexer and recalculates things and keeps them in additional table(s).

[dpc]

I'm closing since there's nothing to do, but feel free to ask questions / post comments.