[DO NOT MERGE] Difficulty adjustment algorithm using LWMA

[thaerkh]

I've overridden my original Weighted-weigthed Harmonic Mean (WWHM) implementation (kyuupichan/difficulty#28) with a modified version of the newest Linearly Weighted Moving Average (LWMA) implementation (zawy12/difficulty-algorithms#3 (comment)).

• Documented in the links above are mentions of a median timespan exploit - the fix for this doesn't need a hard fork, and a pull request exists at Use median timestamp if current time renders a block invalid #3731
  -- a hard fork would be needed if we include this FTL minimum bound masari-project/masari@0ad730e
• There is also a bug fix in the PR, and there's another PR for it that could be merged independently Ensure m_timestamps has the correct number for computing difficulty #3732
• The WWHM implementation for mitigating negative timestamps was using "Neil's method" (as defined by zawy), and the latest LWMA version allows for negative timestamps, which is better addressing future time limit exploits.
• The code submitted is a mixture reference of Wownero's code with added modifications in consultation with @zawy12 . This includes a higher max on a timespan for better responsiveness to high drops in hash rate, and higher minimums defined by the blocks future time limit (FTL) to be symmetric against potential FTL exploits (coupled with allowing negatives).

All connected copyrights have been retained.

[zawy12]

There is a potential exploit in the above algorithm. Use this instead:

#  Weighted-Weighted Harmonic Mean (WWHM) difficulty algorithm
# Original idea from Tom Harding (Deger8) called "WT-144"  
# No limits in rise or fall rate should be employed.
# MTP should not be used.
# ST = solvetime, T=target solvetime
# height = most recently solved block

# set constants
# N=60 # See Masari coin for live data with N=60
# T=600 # coin's Target solvetime. If this changes, nothing else needs to be changed.
# adjust=0.99 # 0.98 for N=30, 0.99 for N=60
#  k = (N+1)/2 *adjust * T # there is not a missing N. 

# algorithm
d=0, t=0, j=0
previous_max=timestamp[height - N] 
for ( i = height-N+1; i < height+1; i++) {  # (N most recent blocks)
    max_timestamp=max(timestamp[i], previous_max)
    solvetime = max_timestamp - previous_max
    solvetime=1 if solvetime < 1
    # for N=60, 10*T solvetimes drives difficulty too far down, so:
     solvetime = 10*T if solvetime > 10*T 
     previous_max=max_timestamp
    j++
    t +=  solvetime * j 
    d += D[i] # sum the difficulties this is how WWHM is different from Tom's WT.
}
t=T*N if t < T*N  # in case of startup weirdness, keep t reasonable
next_D = d * k / t 

The following EMA is just as good and pretty much the same when N of EMA is 1/2 the N of WWHM. So the N=70 selected below will be a lot slower to respond than the N=60 above, but will be a lot smoother. A lot smoother invites hash attacks a lot less (because the price/difficulty ratio looks attractive when difficulty accidentally goes down). So you can't win for losing when selecting N. The the past I've wanted smaller N, but BCH DAA N=144 success is causing me to go to larger N. The WWHM N=60 on Masari coin is doing great, so it might be best for small coins.

# Jacob Eliosoff's EMA (exponential moving average)
# https://en.wikipedia.org/wiki/Moving_average#Application_to_measuring_computer_performance
# ST = solvetime, T=target solvetime
# height = most recently solved block
# N=70
# MTP should not be used

for (i=height - 10 to height) {  # check timestamps starting 10 blocks into past)
   previous_max = max
   max=timestamp[i] if timestamp[i] > max
}
ST = max - previous_max
ST = max(T/100,min(T*10, ST))
next_D = previous_D * ( T/ST + e^(-ST/T/N) * (1-T/ST) )

HOWEVER, I am not yet sure what is the best N for either of the two algorithms above. If I make a change to the above I'll modify this blog post.

[thaerkh]

I've made adjustments to the implementation as recommended above - thanks!

[thaerkh]

Target solvetime/timespan has been 120 seconds since inception for Masari - it might look different in v2's algo compared to Sumo's because they have 4 minute blocks (WHM algorithm above is part of v3 hard fork for Masari)

[zawy12]

We've developed a new algorithm that's the best of the best and allows integer math on the target.

zawy12/difficulty-algorithms#17

[thaerkh]

Commit 9d0c0b4 is a bit ugly but resolves overflow issue for when the Monero network hits 40x+ the current difficulty if this implementation is considered when compared to competing algorithms.

[thaerkh]

I'm told by @zawy12 that this is "almost as good", but is a much simpler approximation of EMA and may be more preferred for integrating into Monero - zawy12/difficulty-algorithms#21

[thaerkh]

This LWMA implementation is a better than my implementation https://github.com/wowario/wownero/blob/master/src/cryptonote_basic/difficulty.cpp#L192

references:
zawy12/difficulty-algorithms#3 (comment)

[thaerkh]

I've updated the description and revised the pull request to include LWMA as it is a better algorithm than WWHM.

[thaerkh]

I've just included an FTL bound in Masari, which bounds a chain's timestamps by it's top block timestamp - FTL (masari-project/masari@0ad730e), which should work well with the -FTL minimum in the DAA proposed here.
I'll see how to include it, since it needs the patch from #3731 and it's possible to merge the two

[zawy12]

If the -FTL is not enforced in addition to the MTP for block validation and template creation, then the following line must be removed from the LWMA proposed:
solveTime = std::min<int64_t>((T * 10), std::max<int64_t>(solveTime, -FTL));
as gabetron pointed out to us that -FTL and +10xT limits in the code are a milder form of using
if solvetime < 0 then solvetime =0
which allowed a catastrophic exploit (on the part of -FTL and a reverse problem on the part of +10xT, i.e. difficulty could be forced to rise).