Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Difficulty adjustment algorithm using LWMA #2887

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

thaerkh
Copy link
Contributor

@thaerkh thaerkh commented Dec 5, 2017

I've overridden my original Weighted-weigthed Harmonic Mean (WWHM) implementation (kyuupichan/difficulty#28) with a modified version of the newest Linearly Weighted Moving Average (LWMA) implementation (zawy12/difficulty-algorithms#3 (comment)).

  • Documented in the links above are mentions of a median timespan exploit - the fix for this doesn't need a hard fork, and a pull request exists at Use median timestamp if current time renders a block invalid #3731
    -- a hard fork would be needed if we include this FTL minimum bound masari-project/masari@0ad730e
  • There is also a bug fix in the PR, and there's another PR for it that could be merged independently Ensure m_timestamps has the correct number for computing difficulty #3732
  • The WWHM implementation for mitigating negative timestamps was using "Neil's method" (as defined by zawy), and the latest LWMA version allows for negative timestamps, which is better addressing future time limit exploits.
  • The code submitted is a mixture reference of Wownero's code with added modifications in consultation with @zawy12 . This includes a higher max on a timespan for better responsiveness to high drops in hash rate, and higher minimums defined by the blocks future time limit (FTL) to be symmetric against potential FTL exploits (coupled with allowing negatives).

All connected copyrights have been retained.

@thaerkh thaerkh changed the title Difficulty adjustment algorithm using a weighted harmonic mean WIP: Difficulty adjustment algorithm using a weighted harmonic mean Dec 6, 2017
@zawy12
Copy link

zawy12 commented Dec 6, 2017

There is a potential exploit in the above algorithm. Use this instead:

#  Weighted-Weighted Harmonic Mean (WWHM) difficulty algorithm
# Original idea from Tom Harding (Deger8) called "WT-144"  
# No limits in rise or fall rate should be employed.
# MTP should not be used.
# ST = solvetime, T=target solvetime
# height = most recently solved block

# set constants
# N=60 # See Masari coin for live data with N=60
# T=600 # coin's Target solvetime. If this changes, nothing else needs to be changed.
# adjust=0.99 # 0.98 for N=30, 0.99 for N=60
#  k = (N+1)/2 *adjust * T # there is not a missing N. 

# algorithm
d=0, t=0, j=0
previous_max=timestamp[height - N] 
for ( i = height-N+1; i < height+1; i++) {  # (N most recent blocks)
    max_timestamp=max(timestamp[i], previous_max)
    solvetime = max_timestamp - previous_max
    solvetime=1 if solvetime < 1
    # for N=60, 10*T solvetimes drives difficulty too far down, so:
     solvetime = 10*T if solvetime > 10*T 
     previous_max=max_timestamp
    j++
    t +=  solvetime * j 
    d += D[i] # sum the difficulties this is how WWHM is different from Tom's WT.
}
t=T*N if t < T*N  # in case of startup weirdness, keep t reasonable
next_D = d * k / t 

The following EMA is just as good and pretty much the same when N of EMA is 1/2 the N of WWHM. So the N=70 selected below will be a lot slower to respond than the N=60 above, but will be a lot smoother. A lot smoother invites hash attacks a lot less (because the price/difficulty ratio looks attractive when difficulty accidentally goes down). So you can't win for losing when selecting N. The the past I've wanted smaller N, but BCH DAA N=144 success is causing me to go to larger N. The WWHM N=60 on Masari coin is doing great, so it might be best for small coins.

# Jacob Eliosoff's EMA (exponential moving average)
# https://en.wikipedia.org/wiki/Moving_average#Application_to_measuring_computer_performance
# ST = solvetime, T=target solvetime
# height = most recently solved block
# N=70
# MTP should not be used

for (i=height - 10 to height) {  # check timestamps starting 10 blocks into past)
   previous_max = max
   max=timestamp[i] if timestamp[i] > max
}
ST = max - previous_max
ST = max(T/100,min(T*10, ST))
next_D = previous_D * ( T/ST + e^(-ST/T/N) * (1-T/ST) )

HOWEVER, I am not yet sure what is the best N for either of the two algorithms above. If I make a change to the above I'll modify this blog post.

@thaerkh thaerkh force-pushed the WHM-DAA branch 2 times, most recently from 9a80015 to 1d706e0 Compare December 6, 2017 09:03
@thaerkh
Copy link
Contributor Author

thaerkh commented Dec 6, 2017

I've made adjustments to the implementation as recommended above - thanks!

Algorithm design by @zawy12 kyuupichan/difficulty#28

Adjustments:
- rolling window N=60
- Block future limit of 20 minutes
- Timestamp check window of 12 blocks
@thaerkh
Copy link
Contributor Author

thaerkh commented Dec 6, 2017

Target solvetime/timespan has been 120 seconds since inception for Masari - it might look different in v2's algo compared to Sumo's because they have 4 minute blocks (WHM algorithm above is part of v3 hard fork for Masari)

@thaerkh thaerkh changed the title WIP: Difficulty adjustment algorithm using a weighted harmonic mean Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018
@thaerkh thaerkh changed the title Difficulty adjustment algorithm using a weighted harmonic mean WIP: Difficulty adjustment algorithm using a weighted harmonic mean Jan 2, 2018
@zawy12
Copy link

zawy12 commented Jan 6, 2018

We've developed a new algorithm that's the best of the best and allows integer math on the target.

zawy12/difficulty-algorithms#17

@thaerkh thaerkh force-pushed the WHM-DAA branch 2 times, most recently from 1078136 to ecfe57f Compare January 8, 2018 06:24
@thaerkh
Copy link
Contributor Author

thaerkh commented Jan 8, 2018

Commit 9d0c0b4 is a bit ugly but resolves overflow issue for when the Monero network hits 40x+ the current difficulty if this implementation is considered when compared to competing algorithms.

@thaerkh
Copy link
Contributor Author

thaerkh commented Jan 15, 2018

I'm told by @zawy12 that this is "almost as good", but is a much simpler approximation of EMA and may be more preferred for integrating into Monero - zawy12/difficulty-algorithms#21

@thaerkh thaerkh changed the title WIP: Difficulty adjustment algorithm using a weighted harmonic mean [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean Apr 22, 2018
@thaerkh
Copy link
Contributor Author

thaerkh commented May 2, 2018

This LWMA implementation is a better than my implementation https://github.com/wowario/wownero/blob/master/src/cryptonote_basic/difficulty.cpp#L192

references:
zawy12/difficulty-algorithms#3 (comment)

@thaerkh thaerkh changed the title [DO NOT MERGE] WIP: Difficulty adjustment algorithm using a weighted harmonic mean Difficulty adjustment algorithm using LWMA May 3, 2018
@thaerkh
Copy link
Contributor Author

thaerkh commented May 3, 2018

I've updated the description and revised the pull request to include LWMA as it is a better algorithm than WWHM.

@thaerkh
Copy link
Contributor Author

thaerkh commented May 3, 2018

I've just included an FTL bound in Masari, which bounds a chain's timestamps by it's top block timestamp - FTL (masari-project/masari@0ad730e), which should work well with the -FTL minimum in the DAA proposed here.
I'll see how to include it, since it needs the patch from #3731 and it's possible to merge the two

@zawy12
Copy link

zawy12 commented May 3, 2018

If the -FTL is not enforced in addition to the MTP for block validation and template creation, then the following line must be removed from the LWMA proposed:
solveTime = std::min<int64_t>((T * 10), std::max<int64_t>(solveTime, -FTL));
as gabetron pointed out to us that -FTL and +10xT limits in the code are a milder form of using
if solvetime < 0 then solvetime =0
which allowed a catastrophic exploit (on the part of -FTL and a reverse problem on the part of +10xT, i.e. difficulty could be forced to rise).

@thaerkh thaerkh changed the title Difficulty adjustment algorithm using LWMA [DO NOT MERGE] Difficulty adjustment algorithm using LWMA May 3, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants