New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Difficulty adjustment algorithm using LWMA #2887

Open
wants to merge 6 commits into
base: master
from

Conversation

2 participants
@thaerkh
Contributor

thaerkh commented Dec 5, 2017

I've overridden my original Weighted-weigthed Harmonic Mean (WWHM) implementation (kyuupichan/difficulty#28) with a modified version of the newest Linearly Weighted Moving Average (LWMA) implementation (zawy12/difficulty-algorithms#3 (comment)).

  • Documented in the links above are mentions of a median timespan exploit - the fix for this doesn't need a hard fork, and a pull request exists at #3731
    -- a hard fork would be needed if we include this FTL minimum bound masari-project/masari@0ad730e
  • There is also a bug fix in the PR, and there's another PR for it that could be merged independently #3732
  • The WWHM implementation for mitigating negative timestamps was using "Neil's method" (as defined by zawy), and the latest LWMA version allows for negative timestamps, which is better addressing future time limit exploits.
  • The code submitted is a mixture reference of Wownero's code with added modifications in consultation with @zawy12 . This includes a higher max on a timespan for better responsiveness to high drops in hash rate, and higher minimums defined by the blocks future time limit (FTL) to be symmetric against potential FTL exploits (coupled with allowing negatives).

All connected copyrights have been retained.

@thaerkh thaerkh changed the title from Difficulty adjustment algorithm using a weighted harmonic mean to WIP: Difficulty adjustment algorithm using a weighted harmonic mean Dec 6, 2017

@zawy12

This comment has been minimized.

zawy12 commented Dec 6, 2017

There is a potential exploit in the above algorithm. Use this instead:

#  Weighted-Weighted Harmonic Mean (WWHM) difficulty algorithm
# Original idea from Tom Harding (Deger8) called "WT-144"  
# No limits in rise or fall rate should be employed.
# MTP should not be used.
# ST = solvetime, T=target solvetime
# height = most recently solved block

# set constants
# N=60 # See Masari coin for live data with N=60
# T=600 # coin's Target solvetime. If this changes, nothing else needs to be changed.
# adjust=0.99 # 0.98 for N=30, 0.99 for N=60
#  k = (N+1)/2 *adjust * T # there is not a missing N. 

# algorithm
d=0, t=0, j=0
previous_max=timestamp[height - N] 
for ( i = height-N+1; i < height+1; i++) {  # (N most recent blocks)
    max_timestamp=max(timestamp[i], previous_max)
    solvetime = max_timestamp - previous_max
    solvetime=1 if solvetime < 1
    # for N=60, 10*T solvetimes drives difficulty too far down, so:
     solvetime = 10*T if solvetime > 10*T 
     previous_max=max_timestamp
    j++
    t +=  solvetime * j 
    d += D[i] # sum the difficulties this is how WWHM is different from Tom's WT.
}
t=T*N if t < T*N  # in case of startup weirdness, keep t reasonable
next_D = d * k / t 

The following EMA is just as good and pretty much the same when N of EMA is 1/2 the N of WWHM. So the N=70 selected below will be a lot slower to respond than the N=60 above, but will be a lot smoother. A lot smoother invites hash attacks a lot less (because the price/difficulty ratio looks attractive when difficulty accidentally goes down). So you can't win for losing when selecting N. The the past I've wanted smaller N, but BCH DAA N=144 success is causing me to go to larger N. The WWHM N=60 on Masari coin is doing great, so it might be best for small coins.

# Jacob Eliosoff's EMA (exponential moving average)
# https://en.wikipedia.org/wiki/Moving_average#Application_to_measuring_computer_performance
# ST = solvetime, T=target solvetime
# height = most recently solved block
# N=70
# MTP should not be used

for (i=height - 10 to height) {  # check timestamps starting 10 blocks into past)
   previous_max = max
   max=timestamp[i] if timestamp[i] > max
}
ST = max - previous_max
ST = max(T/100,min(T*10, ST))
next_D = previous_D * ( T/ST + e^(-ST/T/N) * (1-T/ST) )

HOWEVER, I am not yet sure what is the best N for either of the two algorithms above. If I make a change to the above I'll modify this blog post.

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented Dec 6, 2017

I've made adjustments to the implementation as recommended above - thanks!

Add WHM difficulty adjustment algorithm.
Algorithm design by @zawy12 kyuupichan/difficulty#28

Adjustments:
- rolling window N=60
- Block future limit of 20 minutes
- Timestamp check window of 12 blocks
@zawy12

This comment has been minimized.

zawy12 commented on ecfe57f Dec 6, 2017

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented Dec 6, 2017

Target solvetime/timespan has been 120 seconds since inception for Masari - it might look different in v2's algo compared to Sumo's because they have 4 minute blocks (WHM algorithm above is part of v3 hard fork for Masari)

@thaerkh thaerkh changed the title from WIP: Difficulty adjustment algorithm using a weighted harmonic mean to Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018

@thaerkh thaerkh changed the title from Difficulty adjustment algorithm using a weighted harmonic mean to WIP: Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018

@zawy12

This comment has been minimized.

zawy12 commented Jan 6, 2018

We've developed a new algorithm that's the best of the best and allows integer math on the target.

zawy12/difficulty-algorithms#17

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented Jan 8, 2018

Commit 9d0c0b4 is a bit ugly but resolves overflow issue for when the Monero network hits 40x+ the current difficulty if this implementation is considered when compared to competing algorithms.

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented Jan 15, 2018

I'm told by @zawy12 that this is "almost as good", but is a much simpler approximation of EMA and may be more preferred for integrating into Monero - zawy12/difficulty-algorithms#21

@valiant1x valiant1x referenced this pull request Mar 28, 2018

Closed

New difficulty algorithm #53

@thaerkh thaerkh changed the title from WIP: Difficulty adjustment algorithm using a weighted harmonic mean to [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean Apr 22, 2018

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented May 2, 2018

@thaerkh thaerkh changed the title from [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean to Difficulty adjustment algorithm using LWMA May 3, 2018

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented May 3, 2018

I've updated the description and revised the pull request to include LWMA as it is a better algorithm than WWHM.

@thaerkh

This comment has been minimized.

Contributor

thaerkh commented May 3, 2018

I've just included an FTL bound in Masari, which bounds a chain's timestamps by it's top block timestamp - FTL (masari-project/masari@0ad730e), which should work well with the -FTL minimum in the DAA proposed here.
I'll see how to include it, since it needs the patch from #3731 and it's possible to merge the two

@zawy12

This comment has been minimized.

zawy12 commented May 3, 2018

If the -FTL is not enforced in addition to the MTP for block validation and template creation, then the following line must be removed from the LWMA proposed:
solveTime = std::min<int64_t>((T * 10), std::max<int64_t>(solveTime, -FTL));
as gabetron pointed out to us that -FTL and +10xT limits in the code are a milder form of using
if solvetime < 0 then solvetime =0
which allowed a catastrophic exploit (on the part of -FTL and a reverse problem on the part of +10xT, i.e. difficulty could be forced to rise).

@thaerkh thaerkh changed the title from Difficulty adjustment algorithm using LWMA to [DO NOT MERGE] Difficulty adjustment algorithm using LWMA May 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment