Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Difficulty adjustment algorithm using LWMA #2887

Open
wants to merge 6 commits into
base: master
from

Conversation

@thaerkh
Copy link
Contributor

@thaerkh thaerkh commented Dec 5, 2017

I've overridden my original Weighted-weigthed Harmonic Mean (WWHM) implementation (kyuupichan/difficulty#28) with a modified version of the newest Linearly Weighted Moving Average (LWMA) implementation (zawy12/difficulty-algorithms#3 (comment)).

  • Documented in the links above are mentions of a median timespan exploit - the fix for this doesn't need a hard fork, and a pull request exists at #3731
    -- a hard fork would be needed if we include this FTL minimum bound masari-project/masari@0ad730e
  • There is also a bug fix in the PR, and there's another PR for it that could be merged independently #3732
  • The WWHM implementation for mitigating negative timestamps was using "Neil's method" (as defined by zawy), and the latest LWMA version allows for negative timestamps, which is better addressing future time limit exploits.
  • The code submitted is a mixture reference of Wownero's code with added modifications in consultation with @zawy12 . This includes a higher max on a timespan for better responsiveness to high drops in hash rate, and higher minimums defined by the blocks future time limit (FTL) to be symmetric against potential FTL exploits (coupled with allowing negatives).

All connected copyrights have been retained.

@thaerkh thaerkh changed the title Difficulty adjustment algorithm using a weighted harmonic mean WIP: Difficulty adjustment algorithm using a weighted harmonic mean Dec 6, 2017
@zawy12
Copy link

@zawy12 zawy12 commented Dec 6, 2017

There is a potential exploit in the above algorithm. Use this instead:

#  Weighted-Weighted Harmonic Mean (WWHM) difficulty algorithm
# Original idea from Tom Harding (Deger8) called "WT-144"  
# No limits in rise or fall rate should be employed.
# MTP should not be used.
# ST = solvetime, T=target solvetime
# height = most recently solved block

# set constants
# N=60 # See Masari coin for live data with N=60
# T=600 # coin's Target solvetime. If this changes, nothing else needs to be changed.
# adjust=0.99 # 0.98 for N=30, 0.99 for N=60
#  k = (N+1)/2 *adjust * T # there is not a missing N. 

# algorithm
d=0, t=0, j=0
previous_max=timestamp[height - N] 
for ( i = height-N+1; i < height+1; i++) {  # (N most recent blocks)
    max_timestamp=max(timestamp[i], previous_max)
    solvetime = max_timestamp - previous_max
    solvetime=1 if solvetime < 1
    # for N=60, 10*T solvetimes drives difficulty too far down, so:
     solvetime = 10*T if solvetime > 10*T 
     previous_max=max_timestamp
    j++
    t +=  solvetime * j 
    d += D[i] # sum the difficulties this is how WWHM is different from Tom's WT.
}
t=T*N if t < T*N  # in case of startup weirdness, keep t reasonable
next_D = d * k / t 

The following EMA is just as good and pretty much the same when N of EMA is 1/2 the N of WWHM. So the N=70 selected below will be a lot slower to respond than the N=60 above, but will be a lot smoother. A lot smoother invites hash attacks a lot less (because the price/difficulty ratio looks attractive when difficulty accidentally goes down). So you can't win for losing when selecting N. The the past I've wanted smaller N, but BCH DAA N=144 success is causing me to go to larger N. The WWHM N=60 on Masari coin is doing great, so it might be best for small coins.

# Jacob Eliosoff's EMA (exponential moving average)
# https://en.wikipedia.org/wiki/Moving_average#Application_to_measuring_computer_performance
# ST = solvetime, T=target solvetime
# height = most recently solved block
# N=70
# MTP should not be used

for (i=height - 10 to height) {  # check timestamps starting 10 blocks into past)
   previous_max = max
   max=timestamp[i] if timestamp[i] > max
}
ST = max - previous_max
ST = max(T/100,min(T*10, ST))
next_D = previous_D * ( T/ST + e^(-ST/T/N) * (1-T/ST) )

HOWEVER, I am not yet sure what is the best N for either of the two algorithms above. If I make a change to the above I'll modify this blog post.

@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch 2 times, most recently from 9a80015 to 1d706e0 Dec 6, 2017
@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented Dec 6, 2017

I've made adjustments to the implementation as recommended above - thanks!

Algorithm design by @zawy12 kyuupichan/difficulty#28

Adjustments:
- rolling window N=60
- Block future limit of 20 minutes
- Timestamp check window of 12 blocks
@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch from 1d706e0 to ecfe57f Dec 6, 2017
@zawy12

This comment has been minimized.

Copy link

@zawy12 zawy12 commented on ecfe57f Dec 6, 2017

@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented Dec 6, 2017

Target solvetime/timespan has been 120 seconds since inception for Masari - it might look different in v2's algo compared to Sumo's because they have 4 minute blocks (WHM algorithm above is part of v3 hard fork for Masari)

@thaerkh thaerkh changed the title WIP: Difficulty adjustment algorithm using a weighted harmonic mean Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018
@thaerkh thaerkh changed the title Difficulty adjustment algorithm using a weighted harmonic mean WIP: Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018
@zawy12
Copy link

@zawy12 zawy12 commented Jan 6, 2018

We've developed a new algorithm that's the best of the best and allows integer math on the target.

zawy12/difficulty-algorithms#17

@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch 2 times, most recently from 1078136 to ecfe57f Jan 8, 2018
@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch from 1078136 to 9d0c0b4 Jan 8, 2018
@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented Jan 8, 2018

Commit 9d0c0b4 is a bit ugly but resolves overflow issue for when the Monero network hits 40x+ the current difficulty if this implementation is considered when compared to competing algorithms.

@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented Jan 15, 2018

I'm told by @zawy12 that this is "almost as good", but is a much simpler approximation of EMA and may be more preferred for integrating into Monero - zawy12/difficulty-algorithms#21

@thaerkh thaerkh changed the title WIP: Difficulty adjustment algorithm using a weighted harmonic mean [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean Apr 22, 2018
@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented May 2, 2018

@thaerkh thaerkh changed the title [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean Difficulty adjustment algorithm using LWMA May 3, 2018
@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented May 3, 2018

I've updated the description and revised the pull request to include LWMA as it is a better algorithm than WWHM.

@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch from cc89d3d to 6e5de32 May 3, 2018
@thaerkh
Copy link
Contributor Author

@thaerkh thaerkh commented May 3, 2018

I've just included an FTL bound in Masari, which bounds a chain's timestamps by it's top block timestamp - FTL (masari-project/masari@0ad730e), which should work well with the -FTL minimum in the DAA proposed here.
I'll see how to include it, since it needs the patch from #3731 and it's possible to merge the two

@zawy12
Copy link

@zawy12 zawy12 commented May 3, 2018

If the -FTL is not enforced in addition to the MTP for block validation and template creation, then the following line must be removed from the LWMA proposed:
solveTime = std::min<int64_t>((T * 10), std::max<int64_t>(solveTime, -FTL));
as gabetron pointed out to us that -FTL and +10xT limits in the code are a milder form of using
if solvetime < 0 then solvetime =0
which allowed a catastrophic exploit (on the part of -FTL and a reverse problem on the part of +10xT, i.e. difficulty could be forced to rise).

@thaerkh thaerkh changed the title Difficulty adjustment algorithm using LWMA [DO NOT MERGE] Difficulty adjustment algorithm using LWMA May 3, 2018
@thaerkh thaerkh force-pushed the thaerkh:WHM-DAA branch from b1ebe95 to 535b47c May 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants