[Consensus] Problems with difficulty drop consensus rule

[sanket1729]

The difficulty drop consensus rule creates an interesting scenario as follows:

Consider the following simple scenario:

1. Miners wait for difficulty to down significantly low (they can mine BTC meanwhile)
2. They mine 2016 blocks very fast.
3. Back to step 1

With a stable hash rate (stable referring to amount of hash power such that 2016 blocks are mined in 2 weeks), miners can wait for some periods of 12 hours and then mine. Assuming stable hash rate, some basic calculations I did showed that it is possible mine 2016 blocks in 6.3 days while maintaining the same difficulty across periods. I think the problem is even worse because the potential hash power(BTC + BCH) which can mine bitcoin cash is atleast 5 times higher than stable hash power.

I think such a faster rate of coin generation is problematic as it would lead to faster halving(less than 2 years), which leads to faster rise of transaction fees. This leads to the very problem for which bitcoin cash was created to solve. Furthermore, faster generation of coins will also lead faster into unexplored lands of comparable transaction fees overcome block reward incentive.

This is a big problem because if 95% hash power can meetup at a place, it is quite easy to collude for this profit as it fits perfectly in the consensus rules for Bitcoin Cash. Miners are also not necessarily bound to Bitcoin Cash, since they have main bitcoin to fall back on.

A simple solution would be remove the rule by adding a Soft Fork.

I am sure that that the scenario has been considered before designing REQ-7 of UAHF spec, but I feel these consequences do not justify the rationale.

I intended this for a mailing list discussion, but I can't seem to references to it in bitcoinabc.org or bitcoincash.org

[ThomasdenH]

Since we can expect difficulty jumps, at least in the near future, I think it is best to move to a algorithm like the one on Ethereum. That will allow for hashrate drops but also pointless to manipulate.

[sanket1729]

To add to this, Miners can easily manipulate timestamps in history to produce fake difficulty drops while continously mining blocks.

As an example,

1. They create blocks 2016 blocks with timestamps at t+1 sec,... t +2016 secs.
2. Difficulty bumps up 4 times. (difficulty can bump up maximum 4 times)
3. Miners mine blocks at t + 2016 sec + 2 *k hours(k=1,2,3,4,5,6 ...) till the difficulty goes significantly low again. They have gained this time in "real world" by manipulating timestamps in "blockchain time".

Fast Mining is also highlighted by the last period of 2016 blocks which should have lasted 2 weeks but was very short(3 days?)

[1BitcoinOrBust]

Removing EDA entirely and reducing the retarget interval to something like 1, 6 or 12 blocks (down from 2016) should mitigate the stability issue.

[zanza321]

Seems like this could be addressed by included an EDA adjustment in an upward direction also. So sudden influx of hashrate will trigger increase in EDA also. This way the target 2 weeks = 2016 blocks will hold steady in the long term. With the current scenario, the EDA is activating outside of an "emergency," so it seems to not be the original intention.

[ThomasdenH]

The best solution is to have one calculation that just works instead of patching current limitations. I think Bitcoin Cash is able to HF to make these changes, so it's best to do it right. Making difficulty adjustments gradual instead of being dependent on a 2016 block period seems like the simplest and most elegant solution. By adding more EDA's you risk creating an even more severe oscillation or other unpredictable behaviour.

[ser81]

Miners follow the profit and the profitability depends on the price and difficulty. The price can change very quickly (global crypto-market attracts big money as we know :) ). In order to balance the profitability quickly the difficulty has to ajust quickly as well. The solution has to include the reduction of a target interwal to a small amount of blocks (down from 2016). Small block target number means stable system.
I am new to GitHub and have only registered to participate in your discussion.
How will the decision be made whether to make the appropriate adjustment or not?

[DesWurstes]

How about a better difficulty adjustment system? (eg. DarkGravityWave
or digishield)

[mpatc]

I think the adjustment system should reflect the old as much as possible, so it retains as many of the same properties as (pre-fullblock era) bitcoin as possible. But it should be updated.

The 2016-block average has seemed to work well as a basis to set target, and minimized opportunities to game the system, however it is a little too coarse. A better way to do it, while keeping the same properties, would be retain the 2016 block average, but adjust the target on a more regular basis, perhaps every 144 blocks.

This would make the network a little more responsive to changes in hashpower, while retaining as many of the good properties of the original difficulty adjustment as possible.

(This would be post-HF to remove EDA, once the price/hashpower disequilibrium quiets down.)

[TierNolan]

The purpose of the EDA is to prevent the chain going dead when miners switch away. Even with a 144 block update, the system would react very slowly to a sudden loss of hashing power.

The EDA itself takes half a day.

The EDA messes up the actually block rate calculation too.

The calculation assumes constant difficulty since the last update.

bitcoin-abc/src/pow.cpp

Line 13 in a46e63c

uint32_t GetNextWorkRequired(const CBlockIndex *pindexPrev,

In summary:

diff = old_diff * (2 weeks) / (interval)

but old diff is the difficulty after EDA.

A better scheme would be to calculate the total POW over the adjustment period and the total time over the adjustment period.

time_interval = previous.time - previous_2016.time
work_done = previous.chain_work - previous_2016.chain_work
Target = ((time_interval) * (2^256 / (work done))) / (2 weeks in seconds)

(You would also need to limit the changes to X4 and /4)

I think EDA should be kept for emergency changes.

If at least one chain has step changes and miners hyper transfer, then it doesn't seem like there is much of a reasonable way to keep things stable.

[mpatc]

I think if the EDA was kept, should be added to stop the flood blocks that occur whenever it kicks in and miners switch their hash.

A way to do this would be something like

const CBlockIndex *pindex24 = pindexPrev->GetAncestor(nHeight - 25);
    assert(pindex24);
int64_t mtp24blocks =
        pindexPrev->GetMedianTimePast() - pindex24->GetMedianTimePast();
if (mtp24blocks > 3600) {
        return tooQuickMakeItHarder;
    }

to check from whenever the EDA is triggered, until the next regular difficulty adjustment.

This would check if the last 24 blocks happened within an hour, and then do something (increase difficulty by an amount, perhaps a 'reset' by recalculating the last 2016 blocks and using that.

Edit: I'd probably make the check smaller, something like 12 or 18 blocks/hr. But you get the idea.

[sanket1729]

I did some simulations. It showed that you could mine 2016 blocks in about 2 days maintaining difficulty across periods under the assumption that old_diff is difficulty before main adjustment period.

I think the change you recommend should be added, but that won't solve the issue

[mpatc]

A more 'draconian' implementation might solve it? For example, whenever EDA is triggered, adjust difficulty if last block was sooner than 1 hr from 6 block, until next regular adjustment period. It would make it harder to game. This would make the kind of yo-yo effect we're seeing less likely.

[ThomasdenH]

How about adjusting the difficulty every block based on the previous 2016 block average? That way a sudden change in hash power should produce a gradual change in difficulty. It also means every 2016 blocks the difficulty is the same as it would be in the old system.

[mpatc]

@ThomasdenH that approach seems riskier, since it means miners would constantly be mining at a different difficulty level, so this could:

• create more orphan blocks, since it'll be more likely different nodes mine at different diffs.
• change the behavior of on/off decisions of miners based on profitability. (Could be good or bad)
• incentivize block timestamp manipulation that will help profitability.
• open up an DDOS attack surface to get miners mining at different diffs.

I could be wrong about any or all of those, however a more conservative approach just seems less likely to screw with market forces I don't think anyone fully understands. A big chance of unintended consequences, in other words.

[TierNolan]

create more orphan blocks, since it'll be more likely different nodes mine at different diffs.

The difficulty of a block still depends on the parent (and the rest of the chain).

There could be problems with forks that are more than 1 block long, but those are rare anyway.

change the behavior of on/off decisions of miners based on profitability. (Could be good or bad)

I think good, since they would switch to and from the chain more gradually.

incentivize block timestamp manipulation that will help profitability

I think this balances out.

If it is a problem MTP could be used instead.

open up an DDOS attack surface to get miners mining at different diffs.

The total chain work is what matters anyway. Two miners mining on the same parent will have the same difficulty anyway.

Again using MTP would help here.

[ThomasdenH]

Well, every block has a unique and well-defined history. So there will always be consensus about whether a particular block has the correct difficulty. There is no extra chance of orphaning since the validity of a block is independent of external factors.

The profitability could very well change, but it would only become more gradual. For example, currently an oscillating effect could occur:

• Period 1: High difficulty, low profitability.
• After 2016 blocks a downward adjustment occurs.
• Period 2: Low difficulty, higher profitability, more miners.
• After 2016 blocks an upward adjustment occurs.
• Repeat.

Calculating every block would mitigate this and make profitability more predictable.

incentivize block timestamp manipulation that will help profitability.

Possibly.

There isn't any difference in the DDOS attack surface since every miner with the same history agrees about the difficulty of the next block. If they aren't mining on top of the same block they will produce blocks incompatible with the other chain anyway.

[mpatc]

Every block has a well defined history, however each node makes it's own calculation each time the target is changed. Assuming the chance of error is > 0, having each node calculate difficulty each block increaces the chance of error. Each block is a new opportunity for a new error, instead of every 2 weeks.

[ThomasdenH]

The chance of error is exactly 0 because the calculation is only based on information in previous blocks.

[stacksmith]

The timestamp manipulation could also lead to an attack against the emergency mode kicking in, as witnessed in the blockchain (lost the link...) I believe someone pushed the timestamp to an earlier position, so that the difference between 6 relevant blocks was 11 hours and 59 minutes.

[deadalnix]

This problem is not really specific to this adjustment algorithm. It happens for all coins that share the same mining infrastructure and is very well known for GPU mined coins. It did not happen on Bitcoin before because Bitcoin has ASIC and no other SHA256 coins come even close to it.

This is what has changed. BCC is big enough to create that mess. The wild swing will stop either when both chains HF to some smoother difficulty adjustment, or one of the chain dies.

[ghost]

I agree with Deadalnix. The EDA is not really broken, but it's causing this issue because of the situation and relationship BTC and BCH share. Because at one point the profitability for BCH is higher than that of BTC's it makes perfect sense for miners to start mining BCH and liquidate for BTC. However, this drives the price down obviously.

The issue is the wild swing of the network hashrate from one network to the other. Bitcoin Cash needs to be able to scale the difficulty back up when needed for cases like these.

[ghost]

Also, Deadalnix I apologize for the message from Reddit, I should have checked here first. You obviously have a grasp on the situation.

Has anyone generated any ideas toward a solution? Is a hard fork absolutely necessary?

[sanket1729]

@deadalnix ,

This problem is not really specific to this adjustment algorithm.

I disagree. With this specific algorithm, faster inflation rate would lead to faster halvings which would lead to higher fees.

The wild swing will stop either when both chains HF to some smoother difficulty adjustment, or one of the chain dies.

As mentioned above, this is way bigger issue for BCH than for BTC. If both chains are to survive, BCH would be highly inflationary and this should be fixed. As mentioned in the issue OP, a soft fork to remove EDA can be deployed.
This should atleast fix the faster coin generation issue, but could the chances of BCH chain dying.

[brian2001]

@deadalnix

Thanks a lot for your fantastic work on Bitcoin ABC / Cash.
In my view this was needed to save the real Bitcoin.
As well EDA was necessary to make sure BCH survive.

EDA indeed seems to lead to a little bit of inflation and eventually inflation manipulation.

What do you think would be the best way to mitigate this ? more frequent difficulty adjustments (every 128 blocks rather than every 2016 blocks ?) or/and maybe symetric EDA ?

@sanket1729

1. EDA is very important and clever because it ensures that the chain is saved even when the hashrate drops, you probably need to acknowledge that.

2. I disagree with what you say

You say that :
1> "faster inflation lead to faster halving " yes as always, for Bitcoin and other cryptos emission curves. you probably mean that the EDA is asymmetric and faster inflation periods are not symmetrically compensated with lower inflation periods of similar length.

2> "faster halvings which would lead to higher fees" faster halving doesn't lead to higher fees. Fee is voluntary, not miner driven (Higher fee is triggered by full blocks, nothing else, should come when global adoption arrive)

My take
I'm against EDA removal, but I agree we need to work to fix the inflation manipulation byproduct.

[ghost]

Remove EDA, it is not necessary anymore. It was necessary in the early days of BCH to ensure the network could survive the UAHF, and it has. Unless there is another soft fork option available, I think EDA is unnecessary now.

[mpatc]

There have been 25 blocks in the last hour, there really should be some kind of brake to prevent this kind of thing and readjust both ways. We're seeing the fact that EDA is one way being gamed by miners as we speak.

[jzcjca00]

A huge amount of time has been invested on developing difficulty adjustment algorithms for the altcoins that respond quickly to changes in hashpower without over-correcting and inducing oscillations.

Rather than try to reinvent the wheel, I suggest that we replace both the regular and emergency difficulty adjustment algorithms in Bitcoin Cash with one of these existing algorithms:

• Dark Gravity Well (DGW) -- Used in DASH.
• Digishield V3 -- Used in Zcash.
• Multi-Interval Difficulty Adjustment System (MIDAS).

[zanza321]

Agreed, lets convert some time tested and well researched adjustments from other alts. Trying to keep 10 minute block time average. There are lots of alt-coins out there, lets take advantage of their research and benefit from them.

I would like at least to keep in the EDA function such that, in case of WW3, major real world even that the hash rate crashed 99%+, the chain would still be able to recover without needing to make any changes. Honey Badger!

[jzcjca00]

The three algorithms I listed already have the equivalent of our EDA as an integral part of them, as opposed to being a separate piece of code. They render our EDA code unnecessary, and leaving it in would just open up opportunities for abuse.

[brian2001]

jzcjca00 all this sounds good, is there a BCH foundation or coding team that can officially decide of this ? and push this ?
Should we setup a governance model ?

[h0jeZvgoxFepBQ2C]

Why don't we just restore the normal core difficulty adjustment?

[zanza321]

@lichtamberg It will require a HF anyway to make a change. Original BTC difficulty algo is kind of "dumb," meaning it was made a long time ago. I think it can be improved on quite a bit. Have a true "emergency" difficulty adjustment is a good feature to have. Think if certain governments decided to ban Bitcoin mining, or WW3, etc..., this makes it resilient. The problem we see now with BCC is its being "gamed" and in a long term time period, there is much more than 2016 blocks per 2 week period (1 block per 10 minutes) because of this.

[h0jeZvgoxFepBQ2C]

Hm but are these long difficulty adjustment times not by purpose? See this: https://petertodd.org/assets/commitments/52ccc4802bd563076cbd25ec4c1ba88152098cb6aa356ba644c9e79a24182da5.txt

[is55555]

I really don't see how this is controversial at all. The original PoW adjustment is just not very good anyway and it's there simply because a HF is required to change it. If you're hard-forking, then definitely adopt a better targeting system.

On top of that - but this is more debatable - a PoW change would be a good idea, so BCH doesn't have to compete with the strongest PoW chain in the world.

[DesWurstes]

If we change the POW, BitMain will never be on BCC side.

[is55555]

And does that make any difference? They are not committing as things stand. I think that ship has sailed, but as I said this is more debatable. What's sure is that a targeting change is quite urgent. Might also fix other issues if you're hard-forking anyway and a PoW change is something to consider.

[DesWurstes]

Why not just replace the difficulty system with a more modern one? They're simpler (than changing the POW), tested, and don't need to be debated.

[is55555]

Yes, definitely this has to be done. But since a HF is going to happen, it would be a waste not to consider other improvements.

[zanza321]

The only person who spoke of POW change, is LukeJR who is hostile to miners. BCC is friendly to miners, POW change is not urgent issue to consider at this point in time, maybe in the future.

Next HF, hopefully it will be soon ? Will include Difficutly adjustment optimization, and I hope Malleability Fix, just so BCC can be on the same track for LN as BTC (which leaves less reason for people and miners to stick to BTC).

[DesWurstes]

I believe we should choose a difficulty adjustment system first. The most stable one looks like DigiShield? (Stable blue line = stable algorithm) (Implementations: zcash/zcash#147 (comment))

[zanza321]

Digishield looks great. Can it handle sudden 95% or even 99% hash rate crash?

[DesWurstes]

We can improve it >v<, we have good devs.

UPDATE: DigiShield Difficulty system is 50% in https://github.com/digibyte/digibyte/blob/master/src/pow.cpp, 50% in https://github.com/digibyte/digibyte/blob/master/src/pow.cpp

[ghost]

Digishield should be able to handle sudden hashpower drops as it adjusts the mining difficulty between blocks. So in the case of BCC, every 10 minutes. It's definitely the most viable solution for our issue I believe. Dark Gravity Wave is speculated to have some issues with massive drops in hashpower

[slava-vishnyakov]

What is DigiShield & How it Works to Retarget Difficulty
https://www.reddit.com/r/Digibyte/comments/213t7b/what_is_digishield_how_it_works_to_retarget/

We created DigiShield after seeing the threat that multi-pools pose to a crypto currency when they start mining a coin at a very low difficulty in relation to their net pool hash. This allows many coins to be quickly and easily mined before the difficulty increases.

Sounds exactly like what we have after each EDA.

The secret to DigiShield is an asymmetrical approach to difficulty re-targeting. With DigiShield, the difficulty is allowed to decrease in larger movements than it is allowed to increase from block to block.

This keeps a blockchain from getting "stuck" i.e., not finding the next block for several hours following a major drop in the net hash of coin.

The implementation of DigiShield v3/v4 can be seen here
https://github.com/digibyte/digibyte/blob/3593b858681e61270a4a8a2539627926bca0ef3b/src/pow.cpp#L178-L338

Here is also an useful description of DigiShield v3:
zcash/zcash#147 (comment)

DigiShield v3

The algorithm looks at a fixed number of previous blocks, and calculates the actual time they took (using medians to prevent timewarp attacks):

nActualTimespan = pindexLast->GetMedianTimePast() - pindexFirst->GetMedianTimePast();

An asymmetric floor and ceiling is applied:

nAveragingTargetTimespan = nAveragingInterval * multiAlgoTargetSpacing;

nMaxAdjustDownV3 = 16; // 16% adjustment down
nMaxAdjustUpV3 = 8; // 8% adjustment up
nMinActualTimespanV3 = nAveragingTargetTimespan * (100 - nMaxAdjustUpV3) / 100;
nMaxActualTimespanV3 = nAveragingTargetTimespan * (100 + nMaxAdjustDownV3) / 100;

if (nActualTimespan < nMinActualTimespanV3)
    nActualTimespan = nMinActualTimespanV3;
if (nActualTimespan > nMaxActualTimespanV3)
    nActualTimespan = nMaxActualTimespanV3;

The new difficulty is set to difficulty of the previous block (that had a matching mining algorithm) scaled by nActualTimespan/nAveragingTargetTimespan.

(also there is a note on difference between v3/v4)

The only difference between v3 and v4 (that I could see) was that they halved the averaging target timespan and increased the effect of the error.
Thus AFAICT the only difference between them is in tuning, which we need to adjust anyway because we only have one PoW. If you can see any other differences though, I'd like to know.
zcash/zcash#147 (comment)

[ghost]

Digishield would be a good replacement for the current EDA and would give us a more stable difficulty adjustment. We could also probably do merged mining like @maaku suggested. Two viable options, I'm in favor of both.

[josephNLD]

@lichtamberg The attack vector envisioned by Mr. Todd is easily detectable, and further it is an attack based on a false premise endemic to Core's philosophical roadmap, and on a fundamental misunderstanding of Bitcoin.

It is false that your personal wallet-with-chain, (full node), is the best protection against attack.
This notion is the worst kind of centralization, individualism. While the notion of individualism appeals to many of us, it hazards an unnecessary and undesirable single point of failure, oneself.

The best protection against such fraud attacks is a broad consensus among your trading partners, with whom you hope to spend these coins! If it matters, check block explorers at a minimum. You aren't so likely to spend them to yourself, and if you do, it doesn't matter until you attempt to spend them with your trading partners.

Trace Mayer recently quipped how your full-node is your sword and shield. If so, broad consensus among your peers is a machine-gun.

EDA is a helpful improvement on Bitcoin. It provides for a swifter recovery of Bitcoin in case of global emergencies. Consider how fortunate Bitcoin Cash is, that this improvement was left undone by the former gatekeepers of Bitcoin's reference implementation.

I'd question the premise of this thread. What proof has been offered that the EDA will ever have to change? If it is only because of the situation of the moment, please notice that with each oscillation, Bitcoin Cash comes closer to stability. It seems to be doing what it is designed to do, which is to protect Bitcoin from failure. Core ought to adopt it, but they won't out of pride, which goes before a fall.

[ghost]

@josephNLD EDA is helpful, but there were smarter options available. EDA is good for the network in the case of something like a WW III, but not good in the current situation. Perhaps with every oscillation the difficulty adjustments become ever closer to stability, but how long will that take? It's been less than a week since the issue has been noted and the BCC chain is already 1,595 blocks ahead of Bitcoin's chain, which should not have happened. This issue needs to be fixed otherwise the halving will be reached much sooner and the inflation will keep lowering BCC's worth. Bitcoin Cash will die because of EDA if the current situation is not fixed. The EDA could stabilize, but whose to say how long that will take. If it takes another 5 days then another 20,000 BCC will be mined in that time. BTC Core is not dumb, (although we'd like to think so) but maybe they didn't adopt for a reason.

[checksum0]

I'd rather keep EDA until it stabilizes than switching to untest difficulty retarget algorithm. We already played way more than enough with the protocol as it stands. BTC retarget has a 8 years impeccable track record.

@jiosephNLD no it does not stabilize at all, or nowhere close to fast enough. We are 1600 blocks ahead, excluding whatever BTC was already ahead at the time of the fork. That's 20k coins that reach the market more than whatever core trolls dumps suppressing the price.

We got through last difficulty cycle as quick as the other one, it's ridiculous.

I'm all for ditching EDA ASAP (November/December) without changing regular diff. retarget. Please let's not fall into the "HF are hard..." core narrative here.

Also I'd like to consider BCC a serious project, meaning not replacing a huge important part of it with an extensive testing phase by untested or barely tested algorithm (DGW which is flawed anyway/Digishield).

The idea is to stay as close to bitcoin as possible, not getting creative and introducing all kind of shit...

[josephNLD]

There's the thing, how fast does it need to stabilize? What are the metrics?
Bitcoin Core is unusable all the time because of fees.
Bitcoin Cash is unusable for one day out of four for now because of block time, later one of five, then six... Changing it in December if it isn't stable enough makes sense, I'll be surprised to have been wrong about this, and agree. Doubly surprised if Core has resolved the fee issues by then.

Cash has a viability advantage with EDA that Core lacks, and may come to regret not having implemented when they could have done it without their pride getting in the way.

The accelerated emission provides a slight economic advantage later with a sooner halving. The 20k additional coin in the near term (maybe 50-100k+ before stabilization) are a nice early adopter bonus. The economic effect is an increased scarcity to the later adopters and more coin at a lower price to those who are paying attention.

In the end it comes down to survival fitness. The economics in the current environment favor EDA as is, especially as another chain fork is looming for the Core chain.

[checksum0]

I have no clue why I shouldn't just dump all my BCC at one on the market and just by a Chinese mine since I'll just have more BCC after that.

That's how fucking ridiculous the situation is to anyone that is invested in that project the slightest.

[ghost]

BCH just had the EDA kick in and the new difficulty is set at 119369386419 up from about 89000000000. With 14TH/s S9 @ $610 this puts the profit around $6.5k annually, which is quite close to BTC's $6k. It would seem the EDA oscillations are in fact becoming stable.

[TierNolan]

Has there been any coins which vary minting lockouts to improve stability?

The rule could be that you can't spend a coinbase output unless the MTP of the spending block is more than 28 days after the MTP of the coinbase's block and at least 4032 blocks have passed.

Realtime removes the incentive to abuse block rates. You always have to wait 4032 blocks, so that creates an incentive not to cause the chain to stall out.

[ghost]

I like the concept of DigiShield. The core concept, which is having a larger magnitude of downward difficulty adjustment and a smaller magnitude of upward adjustment is interesting. This could be helpful in dealing with a sudden hashrate drop-off, without relying on the unstable EDA. The issue I see with EDA is that you need to accurately time it. Too few EDAs, you don't reduce the difficulty enough; too many, you welcome a massive influx of hashrate. Since BTP.top operator over-slept recently and missed the mark, I say we plan a sunset of EDA.

But I also agree that we should wait until difficulty and hashrate stablizes first. And we have to go through extensive testing too.

[jihanwu]

Merged mining almost as broken as EDA

[schancel]

Closing this as the EDA has since been removed.