voting, rpc, gui: Implement demand loading of historical poll by poll id and AVW calculation

[jamescowens]

This PR implements a temporary workaround to demand load a poll with the specified (poll transaction) id and the corresponding votes in the chain for the duration of the poll. This allows pulling up old polls in the wallet that are older than the 180 day contract lookback window for contract scanning during wallet init.

This workaround is activated by calling getpollresults poll_id with a valid id that is older than the lookback (i.e. not already in the poll registry). Note that the transaction id containing the poll must be known in advance as this will not work for a poll specified via title.

The old behavior was simply to return poll not found for any poll older than the lookback.

This workaround is very convenient for someone wanting to check a historical poll's information from a historical poll on gidcoinstats.eu for example.

Once getpollresults is called on a valid historical poll, it will be fully loaded, and then will also appear in the GUI, because it has now been loaded in the poll registry.

Note this is intended to be temporary and should be replaced when the improved poll/vote state machine and caching is implemented.

This PR also implements the calculation of GetActiveVoteWeight() in an expired or active poll. To maximize the efficiency of the calculation, the starting and ending block index pointer for the poll is determined (ending being the current pindexBest if the poll is active), and a single scan through the index is done.

It has been instrumented with a timer, and based on testing, it takes on average 50 - 1000 msec depending on the length of the poll, so the overhead is not too bad.

AVW and the Vote Weight % of AVW (a very important metric for poll validation) have been added to the voting GUI.

Closes #2015.
Closes #2209.

[RoboticMind]

Question for future scripting use: does this work for only v2 polls or v1 and v2 polls? Either way this is still really useful

[jamescowens]

I just tweaked it. It works for both, although, you cannot scan in a historical v1 poll/results by title. You must know the txid of the transaction containing the legacy poll.

[jamescowens]

I have also implemented starting and ending (if applicable) block height fields for poll results, because it would appear to be useful for external scripts using the information.

getpollresults df3af5f55268bd71b7498da4690d36b9ba7eb1f403162f72fce5d3cd950e04d9
{
"poll_id": "df3af5f55268bd71b7498da4690d36b9ba7eb1f403162f72fce5d3cd950e04d9",
"poll_title": "technical_poll:_should_we_implement_fixed_pos_block_rewards?",
"poll_expired": true,
"starting_block_height": 1018333,
"ending_block_height": 1037461,
"votes": 170,
"invalid_votes": 19,
"total_weight": 58370964.63848168,
"top_choice_id": 0,
"top_choice": "Yes",
"responses": [
{
"choice": "Yes",
"id": 0,
"weight": 54752273.04727800,
"votes": 122
},
{
"choice": "No",
"id": 1,
"weight": 2989924.69892376,
"votes": 16
},
{
"choice": "Abstain",
"id": 2,
"weight": 628766.89227992,
"votes": 13
}
]
}

[jamescowens]

I am going to do some rearranging of the new code I added to refine it. I think I can also take a stab at AVW.

[jamescowens]

Ok. Rearranging done. I will think about AVW tomorrow.

[jamescowens]

AVW is done.

Here are two examples:

v2 poll:
getpollresults d3c5a547312acb4b97ae37594cadfc98bc72d174dfc9c5892fa4b4a3e9c1f25b
{
"poll_id": "d3c5a547312acb4b97ae37594cadfc98bc72d174dfc9c5892fa4b4a3e9c1f25b",
"poll_title": "Should Gridcoin whitelist the BOINC project SiDock@home?",
"poll_expired": true,
"starting_block_height": 2272529,
"ending_block_height": 2291393,
"votes": 357,
"invalid_votes": 42,
"total_weight": 108434615.37915860,
"active_vote_weight": 228512571.82289608,
"vote_percent_avw": 47.45236313002446,
"top_choice_id": 0,
"top_choice": "Yes",
"responses": [
{
"choice": "Yes",
"id": 0,
"weight": 105234494.34476914,
"votes": 294
},
{
"choice": "No",
"id": 1,
"weight": 1670072.55222291,
"votes": 12
},
{
"choice": "Abstain",
"id": 2,
"weight": 1530048.48216655,
"votes": 9
}
]
}

v1 poll:

getpollresults 22c2ee5e0c049ce93acc6f40d0430f6335367da1c3f61c66d211863cb346600d
{
"poll_id": "22c2ee5e0c049ce93acc6f40d0430f6335367da1c3f61c66d211863cb346600d",
"poll_title": "constant_block_reward_(cbr)_proposal_and_poll",
"poll_expired": true,
"starting_block_height": 1215850,
"ending_block_height": 1267726,
"votes": 412,
"invalid_votes": 81,
"total_weight": 128895044.12683587,
"active_vote_weight": 87902296.08747266,
"vote_percent_avw": 146.6344451327765,
"top_choice_id": 2,
"top_choice": "75/25_with_a_10_GRC_block_reward",
"responses": [
{
"choice": "65/35_with_a_16_GRC_block_reward",
"id": 0,
"weight": 3154376.42412304,
"votes": 25.33333333333333
},
{
"choice": "70/30_with_a_13_GRC_block_reward",
"id": 1,
"weight": 3793019.66460565,
"votes": 11.16666666666667
},
{
"choice": "75/25_with_a_10_GRC_block_reward",
"id": 2,
"weight": 83638309.31465761,
"votes": 166.3333333333334
},
{
"choice": "80/20_with_a_7_GRC_block_reward",
"id": 3,
"weight": 1121641.64235769,
"votes": 27
},
{
"choice": "85/15_with_a_5_GRC_block_reward",
"id": 4,
"weight": 35674085.01625876,
"votes": 77.16666666666666
},
{
"choice": "Reject_Proposal",
"id": 5,
"weight": 1104032.09673936,
"votes": 12
},
{
"choice": "Abstain",
"id": 6,
"weight": 409579.96809376,
"votes": 12
}
]
}

[RoboticMind]

Compared these against the Python script I wrote which matches these numbers very closely (within various averaging differences):

% of AVW comparison:

Poll ID	Python Script	getpollresults
d3c5a547312acb4b97ae37594cadfc98bc72d174dfc9c5892fa4b4a3e9c1f25b	47.447200759708934	47.45236313002446
c8a5d3417f3bacd086fd1cb1be84c00f659fcdddd945d404a421b30e287a4211	51.650724871714246	51.48743989360435
22c2ee5e0c049ce93acc6f40d0430f6335367da1c3f61c66d211863cb346600d	144.2369534918115	146.6344451327765

These numbers differences are likely all accounted for by the endpoint average method in my script for money supply vs the block by block in this code

Full Python Script Output

d3c5a547312acb4b97ae37594cadfc98bc72d174dfc9c5892fa4b4a3e9c1f25b

Looking at the poll with title: "Should Gridcoin whitelist the BOINC project SiDock@home?"

Poll started at block 2272529 and has ended. Finding when it ended | step 1/5
Poll ended at block 2291393

Finding the average money supply | step 2/5
Average money supply is 447791697

Finding the average pool magnitudes (this may take a minute) | step 3/5
Average magnitude is 27402.047655649498

Finding average difficulty (this may take a minute) | step 4/5
Average difficulty is 17.64181382450703

Finding poll voteweight (this may take a minute) | step 5/5
voteweight is 108434615.3791586

AVW is 228537434.54395473

----------
% of AVW is 47.447200759708934
----------

c8a5d3417f3bacd086fd1cb1be84c00f659fcdddd945d404a421b30e287a4211

Looking at the poll with title: "Dispersing Funds for Past Development Work"

Poll started at block 2241781 and has ended. Finding when it ended | step 1/5
Poll ended at block 2260703

Finding the average money supply | step 2/5
Average money supply is 446555291

Finding the average pool magnitudes (this may take a minute) | step 3/5
Average magnitude is 27412.729281767955

Finding average difficulty (this may take a minute) | step 4/5
Average difficulty is 17.30340773651324

Finding poll voteweight (this may take a minute) | step 5/5
voteweight is 116283417.28822023

AVW is 225134143.9583573

----------
% of AVW is 51.650724871714246
----------

22c2ee5e0c049ce93acc6f40d0430f6335367da1c3f61c66d211863cb346600d

Looking at the poll with title: "Constant_Block_Reward_(CBR)_Proposal_and_Poll"

Poll started at block 1215850 and has ended. Finding when it ended | step 1/5
Poll ended at block 1267725

Finding the average money supply | step 2/5
Average money supply is 404148794

Finding the average pool magnitudes (this may take a minute) | step 3/5
Average magnitude is 32359.677272727273

Finding average difficulty (this may take a minute) | step 4/5
Average difficulty is 3.996271125301249

Finding poll voteweight (this may take a minute) | step 5/5
voteweight is 128895109.01299503

AVW is 89363409.08714221

----------
% of AVW is 144.23700967730954
----------

[jamescowens]

Thanks @RoboticMind with a complicated calculation like AVW, it is good to get a completely independent verification of the results. :)

[jamescowens]

Some handwritten notes on vote weight...

[jamescowens]

Ah, what the heck. I have a little more energy tonight. Let's see if I can do a commit to get AVW in the GUI.

[jamescowens]

This shows the "card" style with an older (v1) poll demand loaded via getpollresults 22c2ee5e0c049ce93acc6f40d0430f6335367da1c3f61c66d211863cb346600d after a wallet start and then refresh the voting GUI.

Here is the table view, sorted by % of Active Vote Weight

scrolled over...

[RoboticMind]

Tested this locally and it looks pretty good. For some reason in your screenshot the arrow is overlapping the text on the table, but it doesn't for me when I tested. The only other really minor thing I noticed is that the kerning between the W in AVW and the : is a bit off (looks too close to it), but not sure how easy that is to change

[denravonska]

utACK with minor style suggestions.

[jamescowens]

Ok. Here is the first test on testnet using a node with a build prior to this PR to create a poll with the same title only differing by the case of the first letter.
Once the poll was activated, here is what it looks like on that node:

Here is what the node with this PR sees:

[jamescowens]

As expected based on my review of the consensus code, this doesn't cause a consensus issue per se, because the node with this PR simply doesn't insert the second poll with the same title but a lowercase P.

[jamescowens]

Let's try and insert a poll with the same title on the node with the PR, but with the case differing.

[jamescowens]

Ok. Found another bug with this. The creation was successful according to the UI, but it is not showing in either list. This is because on the post PR node, it won't pass the emplace line (which uses the to_lower title), and the same on the pre-PR node, because there is already a node with the small p there.

I need to add a condition to the poll insert in the GUI to do to_lower on the poll title and also check for existing polls with the same title (if not already done).

[jamescowens]

Note. And this is an important concept. The polls are primarily keyed in v2 by txid, not title; HOWEVER, it still does not make sense to allow two polls to exist with the same title (even if they differ by case only), because that results in a misleading situation.

Prior to Fern, this was a way to override old polls with new ones of the same name. Allowing two polls with the same title but indexed by txid presents a different, less serious, but still confusing issue, which is that two polls of the same title would be present in the UI.

[jamescowens]

Ok. I put a duplicate title check in CWalletTx PollBuilder::BuildContractTx(CWallet* const pwallet). Now an attempt to create a poll contract with the same (case-insensitive) title results in...

[jamescowens]

Duplicate check works properly from rpc too...

addpoll "Poll title case test" 7 "Is P in poll capitalized (try 4)?" "" 2 1 "http://foo.com"
Poll with that title already exists. Please choose another title. (code -1)

[jamescowens]

@div72 please take a final look at this.

[div72]

ACK 7aae8db.