New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

select difficulty adjustment algorithm #147

Closed
nathan-at-least opened this Issue Mar 26, 2015 · 105 comments

Comments

@nathan-at-least
Contributor

nathan-at-least commented Mar 26, 2015

Are there improvements to availability if difficulty is adjusted each block? Is there any larger vulnerability (eg: DOSing a nodes peer connections to starve it of blocks lowering it's local belief about difficulty)?

Are there any known advantages or risks which clearly make this worthwhile to spend time on or avoid?

We suspect this may be important in the long term, because PoW mining may eventually lead to cascading exit of miners when their margins go <= 0 (Brad Templeton Death Spiral, #95), or during sudden-withdrawal infanticide, or during sudden withdrawal due to side-chain knock-on effects (see #146). This may help with availability during those kinds of events (although it won't mitigate those events directly).

Is the implementation complexity low? (It would seem like changing a single constant K to 1 in "calculate difficulty every K blocks".)

@zookoatleastauthoritycom

This comment has been minimized.

Show comment
Hide comment
@zookoatleastauthoritycom

zookoatleastauthoritycom Mar 26, 2015

I'd like to hear @amiller's opinion of this.

zookoatleastauthoritycom commented Mar 26, 2015

I'd like to hear @amiller's opinion of this.

@zookoatleastauthoritycom

This comment has been minimized.

Show comment
Hide comment
@zookoatleastauthoritycom

zookoatleastauthoritycom May 9, 2015

I would also be interested in evaluation of this from any of the 7S.

zookoatleastauthoritycom commented May 9, 2015

I would also be interested in evaluation of this from any of the 7S.

@nathan-at-least

This comment has been minimized.

Show comment
Hide comment
@nathan-at-least

nathan-at-least Aug 20, 2015

Contributor

Hey @amiller: Are you already aware of any research on this issue? If you aren't and don't intend to look into this, please let us know soon.

Contributor

nathan-at-least commented Aug 20, 2015

Hey @amiller: Are you already aware of any research on this issue? If you aren't and don't intend to look into this, please let us know soon.

@nathan-at-least nathan-at-least added this to the Zcoin 1.0 milestone Aug 20, 2015

@daira daira added the SECURITY label Feb 3, 2016

@daira daira changed the title from Evaluate security/incentives of adjusting difficulty per block. to Evaluate security/incentives of adjusting difficulty more quickly than Bitcoin Mar 8, 2016

@zookozcash

This comment has been minimized.

Show comment
Hide comment
@zookozcash

zookozcash Mar 8, 2016

A related change is that we should use median instead of mean for these sorts of calculations, so that a few outliers can't affect the result.

zookozcash commented Mar 8, 2016

A related change is that we should use median instead of mean for these sorts of calculations, so that a few outliers can't affect the result.

@zookozcash

This comment has been minimized.

Show comment
Hide comment
@zookozcash

zookozcash Mar 9, 2016

In addition to the benefits Nathan summarized in #147 (comment), an additional benefit to this would be to reduce the effect of "fast-mining", which generates suspicion among those people who didn't get in on the first few hours/days of mining, when the difficulty was too low to keep the blocks at the intended rate. See #758 (comment)

zookozcash commented Mar 9, 2016

In addition to the benefits Nathan summarized in #147 (comment), an additional benefit to this would be to reduce the effect of "fast-mining", which generates suspicion among those people who didn't get in on the first few hours/days of mining, when the difficulty was too low to keep the blocks at the intended rate. See #758 (comment)

@zookozcash zookozcash changed the title from Evaluate security/incentives of adjusting difficulty more quickly than Bitcoin to make difficulty adjust more quickly than Bitcoin Mar 9, 2016

@ebfull ebfull modified the milestones: DEPRECATED - Zcoin 1.0, 1.0 Mar 15, 2016

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 5, 2016

Contributor

We can't just rinse and repeat. Every change to this is a hard-forkng consensus change, and we MUST stop making those, ideally with beta 1.

Contributor

daira commented Sep 5, 2016

We can't just rinse and repeat. Every change to this is a hard-forkng consensus change, and we MUST stop making those, ideally with beta 1.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

Here it is against the debug.log data from past 2 days. I was previously using my own log file. Change from N=2 to N=3 on the > 2.5 line. A 10x attack like the one below would result in a 10x delay on a single post-attack block (25 minutes). It needs a change to prevent an attack that would be 5 blocks on, 5 blocks off repeatedly. They seem to be capable of and planning 5-block attacks. In words: revert to 1st spike up averaging period period if a second spike up occurs within 24 blocks of the 1st one. A "icing on cake" change that's too late is to stop the gentle cycles that occur with simple averaging, at 4x the averaging period. You can easily see the cycles below which is not a network effect nor random walk of a Poisson process. With that removed, you can see the difficulty will be a real reflection of the network hash rate.


# zawy difficulty algorithm  v0.12
$N=3 if $ActualTimespan[3] / $TargetTimespan[3] > 2.5;
$N=2 if $ActualTimespan[2] / $TargetTimespan[2] < 0.18;
$difficulty = $SumPastDifficulties[$N] * $TargetInterval / $ActualTimespan[$N]; 
$N++ if $N < 25; 

zawyv0 12

Edit: algo as displayed in this post was corrected to remove the overt error

zawy12 commented Sep 5, 2016

Here it is against the debug.log data from past 2 days. I was previously using my own log file. Change from N=2 to N=3 on the > 2.5 line. A 10x attack like the one below would result in a 10x delay on a single post-attack block (25 minutes). It needs a change to prevent an attack that would be 5 blocks on, 5 blocks off repeatedly. They seem to be capable of and planning 5-block attacks. In words: revert to 1st spike up averaging period period if a second spike up occurs within 24 blocks of the 1st one. A "icing on cake" change that's too late is to stop the gentle cycles that occur with simple averaging, at 4x the averaging period. You can easily see the cycles below which is not a network effect nor random walk of a Poisson process. With that removed, you can see the difficulty will be a real reflection of the network hash rate.


# zawy difficulty algorithm  v0.12
$N=3 if $ActualTimespan[3] / $TargetTimespan[3] > 2.5;
$N=2 if $ActualTimespan[2] / $TargetTimespan[2] < 0.18;
$difficulty = $SumPastDifficulties[$N] * $TargetInterval / $ActualTimespan[$N]; 
$N++ if $N < 25; 

zawyv0 12

Edit: algo as displayed in this post was corrected to remove the overt error

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Sep 5, 2016

Contributor

@daira I agree we need to define this ASAP, but we also need to get this particular aspect as stable as we can, or we will end up pushing out a hard fork two weeks after launch. As friendly as we are to hard forks, I'd prefer to try and leverage whatever data we can get to make it less likely, and I think we will have a much better idea of whether or not we need to make another minor tweak once beta 1 is out.

I'll defer to @nathan-at-least on this, but I'd like to reserve the option to at least tweak the difficulty adjustment bounds before 1.0 if beta 1 shows it to be necessary, since while they are indeed a consensus change, they are very simple to implement, and they do not affect third parties trying to leverage zcashd in any way.

Contributor

str4d commented Sep 5, 2016

@daira I agree we need to define this ASAP, but we also need to get this particular aspect as stable as we can, or we will end up pushing out a hard fork two weeks after launch. As friendly as we are to hard forks, I'd prefer to try and leverage whatever data we can get to make it less likely, and I think we will have a much better idea of whether or not we need to make another minor tweak once beta 1 is out.

I'll defer to @nathan-at-least on this, but I'd like to reserve the option to at least tweak the difficulty adjustment bounds before 1.0 if beta 1 shows it to be necessary, since while they are indeed a consensus change, they are very simple to implement, and they do not affect third parties trying to leverage zcashd in any way.

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Sep 5, 2016

Contributor

@zawy12 it's not clear to me whether your algorithms (and resulting figures) are using the median block time or not. We use median block times in order to limit attacks caused by adversaries selecting whatever timestamps benefit them the most.

Contributor

str4d commented Sep 5, 2016

@zawy12 it's not clear to me whether your algorithms (and resulting figures) are using the median block time or not. We use median block times in order to limit attacks caused by adversaries selecting whatever timestamps benefit them the most.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

$ActualTimespan[3] / $TargetTimespan[3] for the above sequence of 308, 21, 39 seconds for the last 3 blocks would be
(308 + 21 +39) / (3 x 150)
When 150 / (308 + 21 +39) is multiplied by the sum of past 3 block difficulties, the difficulty and the Actual Time for them are thereby both medians since the N cancels. TargetInterval = 150.

zawy12 commented Sep 5, 2016

$ActualTimespan[3] / $TargetTimespan[3] for the above sequence of 308, 21, 39 seconds for the last 3 blocks would be
(308 + 21 +39) / (3 x 150)
When 150 / (308 + 21 +39) is multiplied by the sum of past 3 block difficulties, the difficulty and the Actual Time for them are thereby both medians since the N cancels. TargetInterval = 150.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 5, 2016

Contributor

^ Isn't that the mean, not the median?

Contributor

daira commented Sep 5, 2016

^ Isn't that the mean, not the median?

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

Oops yeah.

zawy12 commented Sep 5, 2016

Oops yeah.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

Here it is with median ActualTimeSpan on the difficulty calculation.
screenshot from 2016-09-05 08-46-36


# zawy difficulty algorithm  v0.13
$N=3 if $ActualTimespan[3] / $TargetTimespan[3] > 2.5;
$N=2 if $ActualTimespan[2] / $TargetTimespan[2] < 0.18;
$difficulty = average($SumPastDifficulties[$N]) * $TargetInterval / median($ActualTimespan[$N]); 
$N++ if $N < 25; 

I do not know off hand what it uses for a median on N=2. Is an average for say N<6 OK?

zawy12 commented Sep 5, 2016

Here it is with median ActualTimeSpan on the difficulty calculation.
screenshot from 2016-09-05 08-46-36


# zawy difficulty algorithm  v0.13
$N=3 if $ActualTimespan[3] / $TargetTimespan[3] > 2.5;
$N=2 if $ActualTimespan[2] / $TargetTimespan[2] < 0.18;
$difficulty = average($SumPastDifficulties[$N]) * $TargetInterval / median($ActualTimespan[$N]); 
$N++ if $N < 25; 

I do not know off hand what it uses for a median on N=2. Is an average for say N<6 OK?

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

I believe timestamps on past blocks are fixed, so I don't understand what you mean by adversaries selecting a difference between timestamps. Or why choosing the median can prevent something that int(mean) can't.

edit:
If a miner solves early by adjusting his timestamp it would allow him to adjust the average for an easier solve on the next block that he has a head start on? Instead of median maybe the seconds could be a multiple of 10, keeping accuracy but restricting possibilities to be used for cheating.

zawy12 commented Sep 5, 2016

I believe timestamps on past blocks are fixed, so I don't understand what you mean by adversaries selecting a difference between timestamps. Or why choosing the median can prevent something that int(mean) can't.

edit:
If a miner solves early by adjusting his timestamp it would allow him to adjust the average for an easier solve on the next block that he has a head start on? Instead of median maybe the seconds could be a multiple of 10, keeping accuracy but restricting possibilities to be used for cheating.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

D=1 is now correct in z9? The "attack" above was just when it's D=1. Why are no solve times more than 330 seconds? It should be more than 20%.

Edit:

Maybe it is an attack and they are turning it on when solve time is taking more than 6 minutes, forcing all of them to be solved in less than 330 seconds, as well as getting what appears to be "1" solved quickly. They know D=1 will be the next block, so they turn on to get it. If that's the case, they've been doing it since block 1500.

zawy12 commented Sep 5, 2016

D=1 is now correct in z9? The "attack" above was just when it's D=1. Why are no solve times more than 330 seconds? It should be more than 20%.

Edit:

Maybe it is an attack and they are turning it on when solve time is taking more than 6 minutes, forcing all of them to be solved in less than 330 seconds, as well as getting what appears to be "1" solved quickly. They know D=1 will be the next block, so they turn on to get it. If that's the case, they've been doing it since block 1500.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 5, 2016

It looks like you're multiplying by a Sum of Past N difficulties based on the previous N ~ 10 to 24, not the current N=2 when it is triggered. So the peaks are 5 to 10x higher than they should be. Is that program usable by someone who can't program in C? Please try v0.12

zawy12 commented Sep 5, 2016

It looks like you're multiplying by a Sum of Past N difficulties based on the previous N ~ 10 to 24, not the current N=2 when it is triggered. So the peaks are 5 to 10x higher than they should be. Is that program usable by someone who can't program in C? Please try v0.12

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Sep 6, 2016

Contributor

@zawy12 the code above is JavaScript, and it definitely is using the correct N (this.nAveragingInterval in the code): see how it first checks the last 2 blocks and sets this.nAveragingInterval = 2 if they are outside the bounds, and after that it continues back up to this.nAveragingInterval blocks to sum past difficulties and get nActualTimespan.

Contributor

str4d commented Sep 6, 2016

@zawy12 the code above is JavaScript, and it definitely is using the correct N (this.nAveragingInterval in the code): see how it first checks the last 2 blocks and sets this.nAveragingInterval = 2 if they are outside the bounds, and after that it continues back up to this.nAveragingInterval blocks to sum past difficulties and get nActualTimespan.

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Sep 6, 2016

Contributor

Put another way, if the limits are triggered, then this.nAveragingInterval gets set to 2, and so the second for loop will do nothing and exit immediately, leaving sumDifficulties and nActualTimespan being calculated for N = 2.

Contributor

str4d commented Sep 6, 2016

Put another way, if the limits are triggered, then this.nAveragingInterval gets set to 2, and so the second for loop will do nothing and exit immediately, leaving sumDifficulties and nActualTimespan being calculated for N = 2.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 6, 2016

[edit: OK, I'm looking into it more to see why we're getting something different ]

zawy12 commented Sep 6, 2016

[edit: OK, I'm looking into it more to see why we're getting something different ]

@str4d

This comment has been minimized.

Show comment
Hide comment
@str4d

str4d Sep 6, 2016

Contributor

@zawy12, in my simulations, red is the difficulty, green is the block interval averaged over the short interval (10 blocks), and blue is the block interval averaged over the long interval (100 blocks). The total hash power is stable, apart from a single entity controlling 35% of total hash power, which turns on and off every 500 blocks (starting on at block 0).

Contributor

str4d commented Sep 6, 2016

@zawy12, in my simulations, red is the difficulty, green is the block interval averaged over the short interval (10 blocks), and blue is the block interval averaged over the long interval (100 blocks). The total hash power is stable, apart from a single entity controlling 35% of total hash power, which turns on and off every 500 blocks (starting on at block 0).

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 6, 2016

OK, yes, your chart concurs with my math for a Poisson process with a stable hash rate and my settings. There should be about 5 jumps up and 5 jumps down per 100 blocks. It may be that by judging it by observation instead of theory (using my charts), my algo is useless. But it needs to be "investigated" why this is so (why z9 is not a Poisson process). In particular, there are no block times since block > 1500 that was more than 330 seconds ( so Actual/Target > 2.5 never occurred on me unless it was post attack). And the minimum is not a Poisson process since only an attack can make it < 0.18 when it should have been occurring on me also at 5%.

It may be I have to abandon hope of using N=2 or 3 minimums as a result of not being able to tweak on an active network. Even if my charts are correct for z9, any future change to the solve routine that makes it more or less like a Poisson process would need an unavailable tweaking of my difficulty settings. It needs to be more robust in regards to solve routines.

Concerning your previous comment "real block times aren't themselves sufficient for describing the effect of changes, since changing the algorithm would completely alter those times" I have been saying my solve time would have been (actual solve time) x (my difficulty) / (actual difficulty) which I can't see from theory would not be perfectly accurate, at least on average.

zawy12 commented Sep 6, 2016

OK, yes, your chart concurs with my math for a Poisson process with a stable hash rate and my settings. There should be about 5 jumps up and 5 jumps down per 100 blocks. It may be that by judging it by observation instead of theory (using my charts), my algo is useless. But it needs to be "investigated" why this is so (why z9 is not a Poisson process). In particular, there are no block times since block > 1500 that was more than 330 seconds ( so Actual/Target > 2.5 never occurred on me unless it was post attack). And the minimum is not a Poisson process since only an attack can make it < 0.18 when it should have been occurring on me also at 5%.

It may be I have to abandon hope of using N=2 or 3 minimums as a result of not being able to tweak on an active network. Even if my charts are correct for z9, any future change to the solve routine that makes it more or less like a Poisson process would need an unavailable tweaking of my difficulty settings. It needs to be more robust in regards to solve routines.

Concerning your previous comment "real block times aren't themselves sufficient for describing the effect of changes, since changing the algorithm would completely alter those times" I have been saying my solve time would have been (actual solve time) x (my difficulty) / (actual difficulty) which I can't see from theory would not be perfectly accurate, at least on average.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

@zawy12: the reason for the block times not being greater than 330 seconds is the testnet-specific minimum difficulty rule. It will be removed in beta 1, since it's obviously causing confusion and is an obstacle to understanding what the behaviour will be on mainnet.

Contributor

daira commented Sep 6, 2016

@zawy12: the reason for the block times not being greater than 330 seconds is the testnet-specific minimum difficulty rule. It will be removed in beta 1, since it's obviously causing confusion and is an obstacle to understanding what the behaviour will be on mainnet.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 6, 2016

@daira str4d said in the related issue the same as my thoughts: someone is turning a big miner on when they see > 300 seconds (5 minutes) which is why the "1" is solved quickly. My big confusion with "1" was the result of it never being actually set to "1" for what the miners had to solve. str4d has said that this is the case. It shows "1" but as far as the difficulty being solved it's set at the previous difficulty. This made Zcash difficulty algorithm function differently from Digisheild v3 because when it was "1" the 16% decrease was being blocked. So it stayed high long periods. I posted a comparison in the forum. The accident is arguably an improvement on Digisheild v3 as both have terrible oscillations.

In z8 d=1 averaged 200 second solve time instead of 30 to 60 seconds. And in z9, here is a typical sequence of block solve times before the ongoing "attack" started. Block 1364 below is when it was set to "1" instead of "200" like all the other blocks around it.

Block solve times should be shifted up 1 in this list.
1359 210
1360 270
1361 55 D =240
1362 107
1363 151
1364 318
1365 306 D=1
1366 143 D=1
1367 57 D=240
1368 60 D= 240
1369 111
1370 136
1371 241

zawy12 commented Sep 6, 2016

@daira str4d said in the related issue the same as my thoughts: someone is turning a big miner on when they see > 300 seconds (5 minutes) which is why the "1" is solved quickly. My big confusion with "1" was the result of it never being actually set to "1" for what the miners had to solve. str4d has said that this is the case. It shows "1" but as far as the difficulty being solved it's set at the previous difficulty. This made Zcash difficulty algorithm function differently from Digisheild v3 because when it was "1" the 16% decrease was being blocked. So it stayed high long periods. I posted a comparison in the forum. The accident is arguably an improvement on Digisheild v3 as both have terrible oscillations.

In z8 d=1 averaged 200 second solve time instead of 30 to 60 seconds. And in z9, here is a typical sequence of block solve times before the ongoing "attack" started. Block 1364 below is when it was set to "1" instead of "200" like all the other blocks around it.

Block solve times should be shifted up 1 in this list.
1359 210
1360 270
1361 55 D =240
1362 107
1363 151
1364 318
1365 306 D=1
1366 143 D=1
1367 57 D=240
1368 60 D= 240
1369 111
1370 136
1371 241

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

It's not necessary for there to be a big miner switching on in that case; the expected outcome is that whichever miner next finds an Equihash solution after the 5 minutes, will grab the block. It's not surprising at all that this would take less than 30 seconds.

Honestly, focusing on the minimum difficulty behaviour is not useful, because it's testnet-specific and is going away.

Contributor

daira commented Sep 6, 2016

It's not necessary for there to be a big miner switching on in that case; the expected outcome is that whichever miner next finds an Equihash solution after the 5 minutes, will grab the block. It's not surprising at all that this would take less than 30 seconds.

Honestly, focusing on the minimum difficulty behaviour is not useful, because it's testnet-specific and is going away.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 6, 2016

If a big miner is not turning on, how can it suddenly violate a Poisson distribution with a rigid limit? Both things for me are about trying to understand what the network is doing under testnet conditions so I can know what to expect from mainnet when the programmign differences are removed. D=1 took 40% longer than D=80 on z8. D=1 has generally been the same as D=150. But suddenly in z9 this changed after block 1500.

If D is really "1" like it says, and if big miners have not been turning on above 300 s in testnet, then main net will not be a Poisson distribution for a given hash rate. But by knowing D is not actually being set to "1", I am able to see there is a miner and that Poisson distribution is not violated.

It also enables me to see that testnet has not tested Digishield v3 as intended.

zawy12 commented Sep 6, 2016

If a big miner is not turning on, how can it suddenly violate a Poisson distribution with a rigid limit? Both things for me are about trying to understand what the network is doing under testnet conditions so I can know what to expect from mainnet when the programmign differences are removed. D=1 took 40% longer than D=80 on z8. D=1 has generally been the same as D=150. But suddenly in z9 this changed after block 1500.

If D is really "1" like it says, and if big miners have not been turning on above 300 s in testnet, then main net will not be a Poisson distribution for a given hash rate. But by knowing D is not actually being set to "1", I am able to see there is a miner and that Poisson distribution is not violated.

It also enables me to see that testnet has not tested Digishield v3 as intended.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

I'm not sure how much if anything can be inferred from behaviour when D = 1, but just to clarify the current behaviour on testnet: at the time of the difficulty switch 5 minutes after the previous block, all miners will already be some part of the way through a solution run. Whichever one gets there first will obtain the block. So the expected time to obtain a block after 5 minutes is essentially a "next bus arrival problem" (for multiple buses with different interarrival times depending on the performance of each miner) and in practice will be pretty quick whenever there are many miners.

Contributor

daira commented Sep 6, 2016

I'm not sure how much if anything can be inferred from behaviour when D = 1, but just to clarify the current behaviour on testnet: at the time of the difficulty switch 5 minutes after the previous block, all miners will already be some part of the way through a solution run. Whichever one gets there first will obtain the block. So the expected time to obtain a block after 5 minutes is essentially a "next bus arrival problem" (for multiple buses with different interarrival times depending on the performance of each miner) and in practice will be pretty quick whenever there are many miners.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

What may be confusing you @zawy12 is that although the difficulty for the current block is set to 1 after 5 minutes, the difficulty used for input to the adjustment algorithm for the next block is unchanged. And just to repeat, this behaviour is going away in beta 1.

Contributor

daira commented Sep 6, 2016

What may be confusing you @zawy12 is that although the difficulty for the current block is set to 1 after 5 minutes, the difficulty used for input to the adjustment algorithm for the next block is unchanged. And just to repeat, this behaviour is going away in beta 1.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

Also, the oscillation problem seems very likely to be fixed completely by @str4d's averaging change, as far as we can tell from simulations.

Contributor

daira commented Sep 6, 2016

Also, the oscillation problem seems very likely to be fixed completely by @str4d's averaging change, as far as we can tell from simulations.

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Sep 6, 2016

Contributor

Incidentally, the way we're graphing block times does tend to exaggerate the variability. When we have time we might try to find a visualisation that properly takes into account the expected variability of the solution process and teases apart the effect of the difficulty adjustment on that. Maybe we should hire a statistician? (Serious suggestion.)

Contributor

daira commented Sep 6, 2016

Incidentally, the way we're graphing block times does tend to exaggerate the variability. When we have time we might try to find a visualisation that properly takes into account the expected variability of the solution process and teases apart the effect of the difficulty adjustment on that. Maybe we should hire a statistician? (Serious suggestion.)

@mcelrath

This comment has been minimized.

Show comment
Hide comment
@mcelrath

mcelrath Sep 6, 2016

I'll reiterate a previous comment: All behavior of the difficulty adjustment algorithm can be understood in terms of a damped harmonic oscillator: https://en.wikipedia.org/wiki/Harmonic_oscillator#Damped_harmonic_oscillator

Median time is the correct thing to do as str4d points out, and all other parameters can be understood as fooling with the damping factor and averaging window. But! There's a well-known optimal solution for the damping factor ("critical damping"). In this case you know the period of the driving force (it's the block time -- hashrate that changes faster than this is unobservable), so you know the critical damping factor too.

I know you guys didn't want to mess with this close to release (and I don't blame you), but the conversation I see above seems to be re-creating a bad approximation to a critically damped harmonic oscillator... You're simply not going to do any better than that solution, and at best will recreate it using a bunch of other opaque parameters.

The fundamental parameters are:

  1. Damping timescale (can be removed by choosing "critical damping" -- setting it equal to the block time)
  2. averaging window for measuring (hashrate), d(hashrate)/dt, and d^2(hashrate)/dt^2

With these in hand, you can write a simple 2nd order differential equation for critical damping with only one parameter (the block time). The necessity of averaging introduces a second timescale.

I think reducing this problem to have one parameter (the averaging window, at fixed block time) significantly simplifies and clarifies the problem.

mcelrath commented Sep 6, 2016

I'll reiterate a previous comment: All behavior of the difficulty adjustment algorithm can be understood in terms of a damped harmonic oscillator: https://en.wikipedia.org/wiki/Harmonic_oscillator#Damped_harmonic_oscillator

Median time is the correct thing to do as str4d points out, and all other parameters can be understood as fooling with the damping factor and averaging window. But! There's a well-known optimal solution for the damping factor ("critical damping"). In this case you know the period of the driving force (it's the block time -- hashrate that changes faster than this is unobservable), so you know the critical damping factor too.

I know you guys didn't want to mess with this close to release (and I don't blame you), but the conversation I see above seems to be re-creating a bad approximation to a critically damped harmonic oscillator... You're simply not going to do any better than that solution, and at best will recreate it using a bunch of other opaque parameters.

The fundamental parameters are:

  1. Damping timescale (can be removed by choosing "critical damping" -- setting it equal to the block time)
  2. averaging window for measuring (hashrate), d(hashrate)/dt, and d^2(hashrate)/dt^2

With these in hand, you can write a simple 2nd order differential equation for critical damping with only one parameter (the block time). The necessity of averaging introduces a second timescale.

I think reducing this problem to have one parameter (the averaging window, at fixed block time) significantly simplifies and clarifies the problem.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 6, 2016

OK, thanks to daira's comment I understand it (finally). I had no idea it was possible to change the difficulty in the middle of a block that was being solved. This greatly affects the testing of difficulty algorithms. The bus analogy should be: 1st bus is on a 2.5 hour Poisson distribution schedule, and another is on a 15 minute Poisson schedule, but the 2nd bus is dispatched to replace the 1st bus if the 1st bus did not make it in 5 hours.

Because of this, the current difficulty algorithm has not been tested because testnet is not a Poisson process, and the algorithm itself was modified away from Digisheild v3.

Since you were not seeing oscillations in the theoretical implementation, I suspect no simple averaging method would have oscillations in that implementation. I suspect the oscillations are the result of either the hard upper limit, Digishield v3, if it is not a Poisson effect.

@mcelrath To an extent you're correct and it's what I'm trying to do, but there are problems with saying it is a straight-forward controller problem. 1) it's not a system with a stable forcing function. Both its frequency and magnitude are changed after you've designed your controller in order to cheat the controller's design. Big miners will make it go into oscillation if you underdamp. They'll get blocks on the down side. If you overdamp, they will mine quickly to pick up 20 blocks and make everyone else wait an hour to get a few more. This is what the adjustment method is going to do. Digishield v3 actually uses postive feedback in a way that cause BOTH to occur. I suspect it's not obvious to everyone on a real network because miners competing to cheat may have reduced the visibility of the effect, actually helping everyone to avoid the problems. 2) You can't easily program the controller to change itself in response to a new forcing function because you can only look at past blocks to determine the new function and the the Poisson process prevents you from being able to see it quickly and clearly. Your inability to measure and respond is where a cheater's profit is. It's hard to not prevent wide variation when someone is secretly trying to force wide variation.

The problem is so simple in its characterization that there should be a standard solution. The median problem is the only thing that I can see would modify it. The variables for input to the solution are average blocks/time setpoint and what max time to solution is allowable X% of the time. The solution should pop out without any theoretical improvements possible. It will also tell you how many blocks per time can be cheated based on your selection of X. If you want minimal coins lost at all cost, then I know the ideal solution: next difficulty = (last difficulty) x (set point) / (last solve time). That's it. It's perfect. It keeps the average. There's no way to cheat it for more than 1 block in a sequence. Miners faced with this might choose a constant hash rate, and simply allow cheaters to get every other block free. But what if there are two cheaters? Three? 400? Would it not balance out based on market forces and perfectly fair mining? Suppose a pool cycles through pools trying to get 1 block at a time. Would that be profitable?

Only partially joking, that's my new recommendation for beta:
(next difficulty) = (last difficulty) x (150 seconds) / (last solve time)

Consequences: 10% of blocks next D = 10x longer than 2.5 minutes. But 25 minutes 57 times a day is not possible. It appears the simple ratio method needs adjusting based on the Poisson distribution being skewed on the low verses high side (asymmetrical). If there are inherent oscillations in the averaging methods, maybe that is the (fixable) cause. so it's not this bad, but something like 25 minutes 4 times a day.

zawy12 commented Sep 6, 2016

OK, thanks to daira's comment I understand it (finally). I had no idea it was possible to change the difficulty in the middle of a block that was being solved. This greatly affects the testing of difficulty algorithms. The bus analogy should be: 1st bus is on a 2.5 hour Poisson distribution schedule, and another is on a 15 minute Poisson schedule, but the 2nd bus is dispatched to replace the 1st bus if the 1st bus did not make it in 5 hours.

Because of this, the current difficulty algorithm has not been tested because testnet is not a Poisson process, and the algorithm itself was modified away from Digisheild v3.

Since you were not seeing oscillations in the theoretical implementation, I suspect no simple averaging method would have oscillations in that implementation. I suspect the oscillations are the result of either the hard upper limit, Digishield v3, if it is not a Poisson effect.

@mcelrath To an extent you're correct and it's what I'm trying to do, but there are problems with saying it is a straight-forward controller problem. 1) it's not a system with a stable forcing function. Both its frequency and magnitude are changed after you've designed your controller in order to cheat the controller's design. Big miners will make it go into oscillation if you underdamp. They'll get blocks on the down side. If you overdamp, they will mine quickly to pick up 20 blocks and make everyone else wait an hour to get a few more. This is what the adjustment method is going to do. Digishield v3 actually uses postive feedback in a way that cause BOTH to occur. I suspect it's not obvious to everyone on a real network because miners competing to cheat may have reduced the visibility of the effect, actually helping everyone to avoid the problems. 2) You can't easily program the controller to change itself in response to a new forcing function because you can only look at past blocks to determine the new function and the the Poisson process prevents you from being able to see it quickly and clearly. Your inability to measure and respond is where a cheater's profit is. It's hard to not prevent wide variation when someone is secretly trying to force wide variation.

The problem is so simple in its characterization that there should be a standard solution. The median problem is the only thing that I can see would modify it. The variables for input to the solution are average blocks/time setpoint and what max time to solution is allowable X% of the time. The solution should pop out without any theoretical improvements possible. It will also tell you how many blocks per time can be cheated based on your selection of X. If you want minimal coins lost at all cost, then I know the ideal solution: next difficulty = (last difficulty) x (set point) / (last solve time). That's it. It's perfect. It keeps the average. There's no way to cheat it for more than 1 block in a sequence. Miners faced with this might choose a constant hash rate, and simply allow cheaters to get every other block free. But what if there are two cheaters? Three? 400? Would it not balance out based on market forces and perfectly fair mining? Suppose a pool cycles through pools trying to get 1 block at a time. Would that be profitable?

Only partially joking, that's my new recommendation for beta:
(next difficulty) = (last difficulty) x (150 seconds) / (last solve time)

Consequences: 10% of blocks next D = 10x longer than 2.5 minutes. But 25 minutes 57 times a day is not possible. It appears the simple ratio method needs adjusting based on the Poisson distribution being skewed on the low verses high side (asymmetrical). If there are inherent oscillations in the averaging methods, maybe that is the (fixable) cause. so it's not this bad, but something like 25 minutes 4 times a day.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 7, 2016

I'm getting surprisingly good results with the following.It responds to attacks with correct difficulty and returns to normal in 3 blocks. The drawback is that it has maybe twenty 10 minute intervals per day. But as I recall 2.5 minutes is less of a requirement than even distribution of mining. There's no error introduced by a medium.

(next difficulty) = (last difficulty) x SQRT [ (150 seconds) / (last solve time) ]

edit: I made a Poisson tester like str4d's and it doesn't work at all on a Poisson distribution. It just works on the testnet data. But are the accidental 1.5 second solves a Poisson generator gives even a remote possibility on a solver that normally take 150 seconds? These never occur on testnet.

zawy12 commented Sep 7, 2016

I'm getting surprisingly good results with the following.It responds to attacks with correct difficulty and returns to normal in 3 blocks. The drawback is that it has maybe twenty 10 minute intervals per day. But as I recall 2.5 minutes is less of a requirement than even distribution of mining. There's no error introduced by a medium.

(next difficulty) = (last difficulty) x SQRT [ (150 seconds) / (last solve time) ]

edit: I made a Poisson tester like str4d's and it doesn't work at all on a Poisson distribution. It just works on the testnet data. But are the accidental 1.5 second solves a Poisson generator gives even a remote possibility on a solver that normally take 150 seconds? These never occur on testnet.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 7, 2016

A Poisson generator will give 1.5 second solves 1% of the time for a 150 second solver. This is what throws off my low N methods (1, 2, or 3 instead of 17) that work on testnet. Is there some way to prove a lower limit on the solve time in order to make a Poisson generator for this testing more realistic?

zawy12 commented Sep 7, 2016

A Poisson generator will give 1.5 second solves 1% of the time for a 150 second solver. This is what throws off my low N methods (1, 2, or 3 instead of 17) that work on testnet. Is there some way to prove a lower limit on the solve time in order to make a Poisson generator for this testing more realistic?

zkbot pushed a commit that referenced this issue Sep 8, 2016

zkbot
Auto merge of #1338 - str4d:147-tweak-difficulty-adjustment-algorithm…
…, r=ebfull

Tweaks to difficulty adjustment algorithm

This PR changes the difficulty algorithm to adjust from the average difficulty over the
block window instead of from the last difficulty. It also removes the special rules for the
testnet, which are incompatible with difficulty averaging.

Closes #147 again.

@zkbot zkbot closed this in #1338 Sep 8, 2016

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Sep 8, 2016

The only good solution I've found is to not let the difficulty decrease. That seems to solve difficulty / attack problems. If miners find it unprofitable to find blocks because of a decrease in coin value, then mining and coin issuance should stop to balance supply and demand, and switch to fees until demand returns by its increased use in the marketplace, or until computing power increases.[edit: It needs a real timestamp]

zawy12 commented Sep 8, 2016

The only good solution I've found is to not let the difficulty decrease. That seems to solve difficulty / attack problems. If miners find it unprofitable to find blocks because of a decrease in coin value, then mining and coin issuance should stop to balance supply and demand, and switch to fees until demand returns by its increased use in the marketplace, or until computing power increases.[edit: It needs a real timestamp]

@daira

This comment has been minimized.

Show comment
Hide comment
@daira

daira Aug 2, 2017

Contributor

Please note that the difficulty algorithm we actually arrived at is described in the protocol spec, section 6.4. It does not use PID control or any square-root operations.

Contributor

daira commented Aug 2, 2017

Please note that the difficulty algorithm we actually arrived at is described in the protocol spec, section 6.4. It does not use PID control or any square-root operations.

@zawy12

This comment has been minimized.

Show comment
Hide comment
@zawy12

zawy12 Jan 14, 2018

I did a PID controller difficulty algorithm based on the EMA algorithm and compared it to the best algorithms available. It wins based on my metrics (lowest Std Dev for a given speed and fewest delays) and should be able to be adjusted to win based on any other metrics chosen...if simplicity is not one of the metrics.

zawy12/difficulty-algorithms#20

zawy12 commented Jan 14, 2018

I did a PID controller difficulty algorithm based on the EMA algorithm and compared it to the best algorithms available. It wins based on my metrics (lowest Std Dev for a given speed and fewest delays) and should be able to be adjusted to win based on any other metrics chosen...if simplicity is not one of the metrics.

zawy12/difficulty-algorithms#20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment