Improve hash rate reporting

[JayDDee]

The hash rate reported reported by miners is calculated by counting the number of hash
iterations over a period of time. This represents how fast the miner is. This hash rate tends
to be consistent and not affected by luck or factors other than the performance of the miner.

The pool calculates hash rate based on the share frequency and difficulty. This hash rate
represents the actual success of the miner in finding and submitting valid has, which
includes a lot of luck.

The questions to be ansered are:

1. Can the miner replicate the way pools calculate hash rate?

2. What other useful information does the miner have to give users a bigger picture of
   the mining environment.

The answer to the first question is yes and the answer to the second question is quite a lot.

Without delay here is a sample of the kind of information available. Everything will be
explained below.

`[2019-06-23 01:46:41] Share submitted.
[2019-06-23 01:46:41] Accepted 72/72 (100%), diff 3.01e-05, 1013.17 H/s, 60C
[2019-06-23 01:46:41] Share hash rate: 596.612 H/s, time: 27.068 secs.
[2019-06-23 01:46:41] Share of block: 0.00262, Block time 2.868 hours.
[2019-06-23 01:46:41] Block height 417421, net hash rate 2.28e+05
[2019-06-23 01:46:41] Averages: hash rate 942.3420 H/s, share 0.238 %
[2019-06-23 01:46:41] Average block time 2.283 hours.

[2019-06-23 01:46:58] Share submitted.
[2019-06-23 01:46:59] Accepted 73/73 (100%), diff 9.19e-06, 1013.81 H/s, 60C
[2019-06-23 01:46:59] Share hash rate: 279.633 H/s, time: 17.638 secs.
[2019-06-23 01:46:59] Share of block: 0.000801, Block time 6.118 hours.
[2019-06-23 01:46:59] Block height 417421, net hash rate 3.49e+05
[2019-06-23 01:46:59] Averages: hash rate 934.1547 H/s, share 0.236 %
[2019-06-23 01:46:59] Average block time 2.301 hours.

[2019-06-23 01:47:42] Share submitted.
[2019-06-23 01:47:42] Accepted 74/74 (100%), diff 1.51e-05, 1015.35 H/s, 60C
[2019-06-23 01:47:42] Share hash rate: 186.654 H/s, time: 43.415 secs.
[2019-06-23 01:47:42] Share of block: 0.00132, Block time 9.166 hours.
[2019-06-23 01:47:42] Block height 417421, net hash rate 1.42e+05
[2019-06-23 01:47:42] Averages: hash rate 912.0947 H/s, share 0.235 %
[2019-06-23 01:47:42] Average block time 2.353 hours.

[2019-06-23 01:47:52] Share submitted.
[2019-06-23 01:47:52] Accepted 75/75 (100%), diff 8.38e-06, 1009.21 H/s, 60C
[2019-06-23 01:47:52] Share hash rate: 440.825 H/s, time: 10.207 secs.
[2019-06-23 01:47:52] Share of block: 0.000731, Block time 3.881 hours.
[2019-06-23 01:47:52] Block height 417421, net hash rate 6.03e+05
[2019-06-23 01:47:52] Averages: hash rate 908.8474 H/s, share 0.233 %
[2019-06-23 01:47:52] Average block time 2.359 hours.
`
Share submitted: This message is displayed when a hash is found and submitted to the
pool.

Accepted: the response from the pool, inclusing the share difficulty and the iterative hash rate.

Share hash rate: based on share diffficulty and time.

Time: time since last submit.

Share of block: How much of a block does the share represent.

Block time: The estimated time to find a block at reported share hash rate.

Block height: Can be used to identify which coin received the share in a multipool.

net hash rate: garbage ignore it.

Averages are for entire session. It usually takes around 100 shares for convergence.

Average hash rate tends to track the pool's reported hashrate much better than
the iterative hash rate.

Block time is a function of the coin being mined so average block time doesn't make sense
in a multipool.

Block time means nothing to the pool or it's TTF, it is an estimate of solo mining expectations.

Network hash rate would be nice but I don't think I have the required variables to
calculate it. Another would be nice is % share of a block but that would require knowing
when a block is found.

This is a very rough first cut but is very encouraging. I can make it work for 2 algos
bt it requires some hard coding, which leads to what's next.

I need to find the magic recipe that works for every algo, hard coded values for each algo
is a show stopper.

Verify accuracy of derived data. Share hash rate looks very good, block time is difficult
to prove.

Change the cummulative averages to 5 minute windowed averages like the pools do.

Report averages at 5 minute intervals insteasd of with each share.

Include other information in 5 minute report: number of shares submitted, accumulated
share difficulty...

Any thoughts? suggestions? additions? deletions? concerns? questions? comments?

[JayDDee]

Some updates.

This will only work for stratum pools at first. Will consider something similar for GBT and getwork
later.

It is my intent to provide API access to the share stats and summary stats.

I still haven't found a magic constant to make it work for all algos. At this time it will be
necessary to engineer each algo individually using trial and error. Also this technique may
work it is clumsy and stupid because I still wouldn't understand "how" it works.
It seems impossible for the miner to determine the net hash rate, and similarly impossible
to determine TTF at the pool. That information will have to be obtained elsewhere such as
the pool API.

Some stats can't be averaged because they could include stats for different coins that use the
same algo but with very different mining specifications. Block time (TTF) is one of those stats.

The summary report will be seperated from the share report and changed to a 5 minute window
to match most pools. How the windowing will be implemented is TBD.

The summary report will include share submission rate, average share hash rate and
average share value (% of block per share).

Block time (TTF) will be removed from the share report. It is mostly a curiosity, is very volatile
and can be calculated from other available data.

Using the block height to determine coin id in a multipool has some promise for automation
by mining manager programs that also collect data from the pool's API.

It is unlikely that 2 coins using the same algo will have the same block height. Well maybe not
so unlikely after seeing the muktiple bitcoin forks. Anyway I'm going with it.

Users can use the block height to match up with a coin in the multipool with the same block height.
Mining manager's can do the same. Use the block height from the share report to do a lookup
at the pool to identify the coin being credited with each share. This would enable comprehensive
per-coin mining stats. Creative mining manager program developpers will find many uses.

Creative mining manager program developpers may find other uses.

Findng a way to determine net hash rate seems futil for a miner

[JayDDee]

The magic constant was an illusion. I didn't wait long enough for the hash rate to converge and
drew the wrong conclusion. All algos should work.

The format is firming up. A new 5 minute summary report has been created providing
average hash rates that more closely match pool reporting.

[2019-06-24` 11:14:01] Summary: 27 shares submitted in 5 mins 18 secs.
[2019-06-24 11:14:01] Avg hashrate 3402.76 kH/s, avg latency 23 ms, temp 76C.

The share submission report is divided into 2 parts. Only the accepted/rejected line is output when
-q option is used.

default:

[2019-06-24 11:20:43] Share 2 submitted by thread 9, lane 1.
[2019-06-24 11:20:43] Accepted, diff 0.00731, 2.740 secs, A/R/S: 2/0/0.
[2019-06-24 11:20:43] Miner 2679.44 kH/s, Share 2863.49 kH/s, Latency 20 ms.
[2019-06-24 11:20:43] Block height 1162141, Block share 0.00003%.

-q:

[2019-06-24 11:10:08] Accepted, diff 0.0121, 3.467 secs, A/R/S: 11/0/0.
[2019-06-24 11:10:08] Accepted, diff 0.0972, 0.302 secs, A/R/S: 12/0/0.
[2019-06-24 11:10:24] Accepted, diff 0.191, 15.670 secs, A/R/S: 13/0/0.
[2019-06-24 11:10:40] Accepted, diff 0.0627, 15.502 secs, A/R/S: 14/0/0.

[JayDDee]

Included in this issue is a change to supress the per-thread hash rates reports by default. A new
command line option "--hash-meter" will enable display of per-thread hash rates.

[JayDDee]

The final report formats apppears to be set and will be included in the next release.
Here is how to interpret the information.

[2019-06-24 14:28:03] Share 26 submitted by thread 7, lane 6.
[2019-06-24 14:28:03] Accepted, diff 0.011, 28.244 secs, A/R/S: 26/0/0.
[2019-06-24 14:28:03] Miner 2681.93 kH/s, Share 418.74 kH/s, Latency 22 ms.
[2019-06-24 14:28:03] Block height 1162225, Block share 0.00005%.

Line 1: Displayed when a share is submitted to the pool. Not displayed with -q option.

Line 2: Displayed when response is received from the pool.

[BLOCK SOLVED,Accepted,Rejected]: the result of the share.

diff; the share's difficulty.

[secs]: number of seconds to find the share.

[A/R/S]: the number share that were accepted, rejected and that solved blocks.

Lines 2 & 3 are displayed with more info about the accepted share. This info is
only displayed for stratum mining and only if the share was accepted by the pool.
It is not displayed with the -q option.

Miner: the hashrate calculated by the miner by counting hash iterations over time.

Share: the hashrate calculated using the share diffficulty. This is more representative
of how pools report hash rate. The rate is very volatile and can report hashrates orders
of magnitude higher or lower than the miner rate.

latency: the round trip time from submitting the share to receiving the response.

Block height: The block number of the coin the share contributed to. In a multipool the block
height can be used to identify which coin is crediting the user with the share by finding the
coin with a current block height in the same range.

Block share: a percentage of the block credited to the user for that share.

---

[2019-06-24 14:28:07] Summary: 26 shares submitted in 5 mins 20 secs.
[2019-06-24 14:28:07] Avg hashrate 5519.04 kH/s, avg latency 21 ms, temp 78C.

The summary report in displayed approximately every 5 minutes.

Line 1: the number of shares submitted during the time and the precise time interval.

Line2:

Avg hashrate: the average share hashrate (see definition above) of the submitted shares.

avg latency: the average network latency.

temp: the CPU temperature measured at the time of the report. This is displayed on Linux only.

[JayDDee]

Colour coding has been added to the diff field of the accept message and the block share
percentage in the summary report.

Magenta: grade A+, solved a block, "BLOCK SOLVED" replaces "Accept",
over 90% average block share in summary.
Green: grade A, over 1% block share or average block share.
Blue: grade B, over 0.1% block share or average block share.
Yellow: grade C. above average hash rate or block share.
White: grade D, below average, statistically half the shares.

[JayDDee]

Final format of the new share submission report and 5 minute summary report.
All assertions below are statistical and are subject to luck.

For each share submitted:

[2019-06-26 10:07:04] Share 1 submitted by thread 12.
[2019-06-26 10:07:04] Accepted, diff 1.12, 21.106 secs, A/R/B: 1/0/0.
[2019-06-26 10:07:04] Miner 64.50 MH/s, Share 56.81 MH/s, Latency 20 ms.
[2019-06-26 10:07:04] Height 116082, Block share 0.01931%.

Line 1 is displayed when a share is submitted unless -q option is set.
Line 1 has 3 formats:

Thread and lane: Parallel multiway hashing.
Thread only: One way hashing.
No thread or lane: One way old style share submission.

Lines 2 is displayed when a response to a share submission has been received.

Result: (green) "Acccepted" a valid share was accepted by the pool.
(red) "Rejected" share was rejected by the pool, the reason usually follows.
(purple) "BLOCK SOLVED" a valid share solved a block.

diff: the difficulty of the submitted share. Colour coded, see previous comment for details.

time: The time since the last share was submitted.

A/R/B: A running total of shares accepted, rejected and blocks solved. Note a solved block
also counts as an accepted share.

Lines 3 & 4 are displayed for accepted shares unless -q option is set.

Miner: the hashrate calculated by the miner by counting hash iterations over the share time.

Share: the hashrate calculated using the share diffficulty and time. This is more representative
of how pools report hash rate. The rate is very volatile and can report hashrates orders
of magnitude higher or lower than the miner rate.

latency: the round trip time from submitting the share to receiving the response.

Block height: The block number of the coin the share contributed to. In a multipool the block
height can be used to identify which coin is crediting the user with the share by finding the
coin with a current block height in the same range.

Block share: The percentage of a block the accepted share represents if the share hashrate
was maintained for the duration of the current estimated block time.

---

[2019-06-26 10:17:48] Summary: 13 submits in 5m15s, block share 0.09825%.
[2019-06-26 10:17:48] Share hashrate 247.23 MH/s, latency 20 ms, temp 76C.

The summary report in displayed approximately every 5 minutes.

Line 1: the number of shares submitted during the time and the precise time interval.

Block share: The percentage share of a block the shares accepted during the period represent
if the average share hash rate was maintained for the duration of the current estimated block time. Colour coded, see previous comment for details.

Line2:

Share hashrate: the average share hashrate (see definition above) of the submitted shares.

latency: the average time it takes to receive a response to a submitted share.

temp: the CPU temperature measured at the time of the report. This is displayed on Linux only.

[hermangroup]

Very well done .
Just a small thing.. for some reason I had API bind error port problem so I think it's good to add " --api-bind=0 " to readme for people who get same annoying message.

Everything else works good.. tested with SSE2 , AES / AVX / AVX2

[JayDDee]

Thanks for the input.

I don't think that's a good workaround. It's easier just to ignore it and it will go away.

I'm making few changes to the share report and summary. I'm reversing the colours white and
yellow. In the next release white means average and yellow means below average. I've also
tweaked the colour thresholds.

Something not so visible is a problem if a new share is submitted before the previous share's
response is received by the pool. It can mess up the stats pretty bad. The next release will have a fix.

[JayDDee]

cpuminer-opt-3.9.5.1 has some aditional changes.

The biggest change is the use of the colour yellow. from now on yellow will only be used to
warn of abnormal events or conditions. This results in a reversal of the colour coded share
grading system, yellow now means below average and white means average.

Normal protocol messages will be blue. Stratum difficulty changes and share submissions will
now be blue.

Job id has been added to many existing messages as well as one new message. This is to
heklp troubleshoot invalid job id rejects seen intermittantly at zergpool.

The colour grade thresholds have been tweaked to better reflect the exponential nature
of some statistics.

There is also a fix to handle rapid share submissions and slow acknowledgments. Share
submissions can be stacked 4 deep pending a response from the pool. The stack can be increased
if necessary but chronic pending responses is a symtom of other problems.

[JayDDee]

A note about the API bind error, that usually means another miner, usuallly ccminer, is running
and already using the port. If you want to use the API for both just specify a different port,
otherwise the warnoings will stop after a few failed attempts.

I could look at making some changes such as choosing a different default port to avoid clashing
with ccminer, but that's a very low priority.

[JayDDee]

Another refoinement is necessary to manage the stats circular buffer in case it overflows (too many
un-acked shares) or shares that are never acked. Both would seem to be handled by submit_work
discarding stats when the buffer is full and share_result checking for valid stats. When share_result
gets caught up it will encounter an ack with no stats data. That's allows it to get back in sync
with submit_work.

If an ack is never received the buffer pointers will be out of sync with a permanent pending share.
This will result in the stats being out of sync with the un-acked share being counted and the last
share's stats remaining in the buffer uncounted. There's no way to detect that situation from
a legitimate un-acked share. The net result is a slight statistical error for the remainder of the
session.

If the pool just stops responding all bets are off, the buffer will overflow and never get emptied.

[JayDDee]

This issue has run its course. V3.9.5.3 includes some tweaks, specifically an 8 slot circular
buffer to handle rapid share submission. The miner hash rate was added to the summary report for
direct comparison with share rate.