[v2] use binary fields in coin_store table

[arvidn]

This patch updates the v2 database schema to use binary fields instead if hex in the coin_store table. Since this table contains mostly hashes, this provides material space savings and a modest speedup.

These changes are best reviewed hiding white space changes.

test	v1	v2	v2 / v1
table size	638 MB	354 MB	55.5%
fast computer, fast SSD	245.8s	196.2s	79.8%
Xeon, HDD	4892.9s	3269.9	66.8%
RPi, flash card	7492s	6082s	81.2%

Raw benchmark output

fast computer on fast SSD:

version 1
Building database 
53.8711s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
8.8726s, MOSTLY REMOVALS additions: 600 removals: 140000
78.0423s, FULLBLOCKS additions: 400400 removals: 400000
1.4273s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
77.8722s, GET RECORDS BY NAMES without spent 200 lookups found 20641 coins in total
25.7557s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 245.8411s
database size: 638.099 MB
version 2
Building database 
46.2741s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
7.8080s, MOSTLY REMOVALS additions: 600 removals: 140000
64.6801s, FULLBLOCKS additions: 400400 removals: 400000
1.3728s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
52.2265s, GET RECORDS BY NAMES without spent 200 lookups found 20693 coins in total
23.8446s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 196.2061s
database size: 353.669 MB

Xeon on HDD:

version 1
Building database
1594.8638s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
597.7385s, MOSTLY REMOVALS additions: 600 removals: 140000
2605.9138s, FULLBLOCKS additions: 400400 removals: 400000
1.1940s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
70.6391s, GET RECORDS BY NAMES without spent 200 lookups found 20641 coins in total
22.5559s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 4892.9050s
database size: 638.095 MB
version 2
Building database
993.7195s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
541.6172s, MOSTLY REMOVALS additions: 600 removals: 140000
1659.5738s, FULLBLOCKS additions: 400400 removals: 400000
2.1022s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
50.8519s, GET RECORDS BY NAMES without spent 200 lookups found 20693 coins in total
22.0893s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 3269.9540s
database size: 353.669 MB

RPi on flash card:

version 1
Building database
2570.0886s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
525.1631s, MOSTLY REMOVALS additions: 600 removals: 140000
3964.7256s, FULLBLOCKS additions: 400400 removals: 400000
6.5235s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
337.6326s, GET RECORDS BY NAMES without spent 200 lookups found 20641 coins in total
88.0736s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 7492.2069s
database size: 638.099 MB
version 2
Building database 
1961.0997s, MOSTLY ADDITIONS additions: 400400.0 removals: 20000.0
483.3735s, MOSTLY REMOVALS additions: 600 removals: 140000
3310.9383s, FULLBLOCKS additions: 400400 removals: 400000
6.1662s, GET RECORDS BY NAMES with spent 200 lookups found 40000 coins in total
229.4063s, GET RECORDS BY NAMES without spent 200 lookups found 20693 coins in total
90.6036s, GET COINS REMOVED AT HEIGHT 800 blocks, found 580000 coins in total
all tests completed in 6081.5876s
database size: 353.620 MB

[mariano54]

I like the changes. V2 is not enabled yet right? So we can keep merging v2 updates until it's enabled?

[arvidn]

I like the changes. V2 is not enabled yet right? So we can keep merging v2 updates until it's enabled?

That's right. My intentions are to:

• keep updating it until we're happy with it
• write a transition tool
• test it thoroughly
• maybe enable it by default for new installs
• encourage existing users to transition