Geth 1.10 txpool ordering #269
Comments
|
I hope so, so many bots using same gas price to stuck the blockchain in IDO or pancake listing |
|
I think this relates to below issue. Agree that bsc specific PoA nature may bring some interest conflicts due to this enhancement. |
This has nothing to do with tx ordering by received time.
I don't see how the incentives for nodes to connect to light nodes change based on how validators order their tx. As for your solutions:
If there is randomization involved then people will spam tx to get the lowest random number. It takes us back to where we started.
It is already getting increasingly difficult to run BSC nodes - we shouldn't be trying to make the problem worse. |
|
With deterministic tx ordering, validators and their friends can ensure 100% send the transactions earlier to the deterministic-ordered node. Internet signal latency is limited by speed of light. If my node is located in Africa, it MUST be slower than the node located next to validators. This change has rewarded validators with higher transaction positioning, while penalizing non-validators' nodes. This problem exists in ethereum mainnet, but bsc deterministic validator order magnified the problem. Adding a noise to receive time will solve the problem if we set sigma (standard deviation of the noise) small enough. Send and populate every transaction needs time. Running a bsc node nowadays is difficult. But it does not imply we do not need to enhance the tools. We shall make the playground more fair to all people, not just focus on the benefits of validators. Validators are well paid for the gas fee, and preferential tx ordering is a critical change that shall have at least consulted community. |
|
I think here we are addressing the issue on fairness. Given validators and their associates have the absolute advantage in transactions with the same gas price as they must be the closest with the shortest request time. I believe Binance shall address this issue, to ensure we are participating in a fair game. |
|
Kind of new to this tx ordering thing. Are you saying that as long as the validator node or the nodes in its vicinity are paying the same gas as mine, I would almost certainly be placed after the validator's transaction due to latency issue? |
|
If a validator wanted to insert their own transactions, and reorder transactions, they could regardless of how the txpool is implemented in the standard bsc node. (see eg MEV, flashbots, etc...) Ordering transactions with the same gas price by time of arrival in the standard bsc implementation, instead of randomly, disincentivizes spam, which is good for the long term health of the bsc network. It just sounds like you used to spam and are now frustrated that it no longer works. IMO if binance validators started doing MEV that would look very bad, as they would be directly taking money from their users and reordering their transactions. |
|
Everyone can modify geth for their own interest. But this does not mean we do not need to modify the default geth version. The problem here is the tx pool logic change is favorable to validators and their friends. This is unfair especially due to bsc deterministic validator ordering. I pointed out this undesirable change is caused by preventing spam at the beginning. I am wondering if any method can help fix the issue while promoting fairness. I would not accuse anyone against this is having benefit due to this change. Let's focus on the matter. |
|
@RayCCY Under normal market condition, most transactions are with 5 gwei gas price. Transactions submitted close to validators node will be front, while other nodes and public node (most of us just use public node in metamask) orders are at the back. |
|
@patllc if this is the case, validators can profit a lot from this change. For example, they would definitely be the first user to participate in initial offerings, front running all the other users. Validators can also make use of their advantage to do arbitrage trading on BSC, eliminating all the other regular arbitrageurs. To conclude, this change is so-called a way to prevent spamming, but in fact, the ultimate goal is to benefit the validators with a HUGE amount of extra profit. Therefore we can expect a lot of people with vested interests to oppose your suggestion to revert this change. |
Yes it is related to tx ordering by received time. Consider an initial offering that starts at block X, right now the validators can insert their tx as the first tx in block X, thus front running all the other participants. @Dondudu please correct me if you think otherwise. |
As mentioned by @patllc, any validator can always edit their own geth to have whatever behaviour they like. They can put their own tx ahead of yours even if their gas price is 0. Actually a validator will always use a "plausible" gas price to hide their behaviour since the gas gets paid to themselves anyway. This is called MEV extraction if you want to read more about it. |
|
Yes @Dondudu Most of the transaction concentrated in 5 gwei ( around 70% transactions in below block using 5gwei https://bscscan.com/txs?block=7960114&p=2), |
|
I'm seeing this problem in some frontrunning actions. Some users are just invincible, they always put their transaction just after the needed one and the rest of users can just only look. They don't ever need to spam the 5 gas transaction, they are always ahead. |
|
Orginally bsc chain is random validator order with random transaction position. This means that even submitting order through public node can be ahead of validators' friends. (Not to mention MEV extraction here as seems cannot avoid) It is more fair than current deterministic order and deterministic transaction position. Can we think of some new method that can both avoid spam attack and more fair? I have proposed
Is there any other method? For the benefits of all bsc users, I do think that we can think of some new ways to improve the chain. |
|
Is randomizing the mining order of validators a possible solution to this? I don't see there will be spamming this way while ensuring no one can take advantage of the deterministic mining order. |
|
This has to be reverted to 1.0.6/1.0.7 behaviour... I do not think it was intentionally, but 1.1.0 was released under pressure of time. |
|
@guagualvcha |
|
If you look at this, i think this wont change.. too lucrative for binance. They have a bot at https://bscscan.com/address/0xf662b54e8f96caef0336ef44f7bef6f5921d0e51 |
|
Well... Arbitragers are necessary for an efficient exchange. But is binance necessarily extracting all opportunities? I hope it is not intentional. |
|
@spec42 You have pointed out a serious problem. Geth 1.10 (unintentionally) causing binance and its friends to exploit all the arbitrage opportunities. As all records are traceable in blockchain, I don't think binance is intentionally making money here. Another issue to raise is bscscan stop providing full transaction records. I need some time to dig out when the bot was created At the same time, as most transactions in bsc is 5 gwei, the 5-gwei region is the hunting ground of nodes with preferential orders. Hope this can be improved although the profit is really so big... binance of cause can hire computer experts to earn money from bsc chain, but the go ethereum algorithm shall be fair. |
|
It has to be intentionally. And i think it is unfair with the current transaction ordering, because they use their infrastructure. Sure it could be a modified node too, which i would also name as bot. Miners on ethereum can do that too, and there is no problem, but with binance owning all validators this seems strange. There is no competition possible... |
|
Normal arbitrager would be suffering in this situation, but more importantly, even general user would be suffering too. As ALL users' transaction will be front run-ed or back run-ed with 100% successful rate as well. Logic of BSC chain shall be enhanced but not follow strictly to Ethereum. |
|
From @spec42:
From @patllc:
Uh, isn't that behavior expected of literally all Binance user deposit addresses? Users deposit to a Binance-controlled wallet (which seems to be 0x8def here) in order to get their funds into Binance, and Binance sweeps all balances to their hot wallet every now and then. I'm inclined to believe that's the case since 0x8def only ever transfers coins to the Binance hot wallet -- so far, it hasn't made any other transactions or contract calls. |
|
@zhongfu agree that the address may be binance user address. Another look at this bot. |
|
Multiple wallets sending transactions to the same smart contract. The spam issue seems not fixed. Another reason why we need to change the logic. |
|
Thats how those bots work. Multiple sender accounts. I would not call it spam. About the Hot Wallet address - I think @zhongfu is right, sorry. I ignored the extra hop... |
|
I think we can all agree that ordering transactions by time, like ethereum does, is the best move for the long term security of BSC. When transaction spamming was viable, BSC was on the verge of collapse will full blocks and forks. Any solution other than time of arrival based transaction ordering incentivizeS spam. Also, the claim that binance runs or otherwise props up the successful bots is ludicrous. I don’t see any evidence that that is the case, and I don’t see any reason why changing the transaction ordering code in the standard bsc node would affect that if it happened to be the case. If binance validators started doing that, it would be very bad PR for them. (Also they already make $2m+ per day in txn fees, up to $6m on a good day.) |
|
@heliosVII-tech In ethereum it is okay because of POW. No one can guaranteed mining a block. In bsc, deterministic validator order and deterministic tx order is far less fair than previous geth version. That's why I raise this issue to improve. I did not question why we need to avoid spam. I think people opposing any change in this thread did not answer whether this change is fair or not. They just repeat stating the purpose of this change, which I have mentioned it at the beginning. I see that this change causes the number of arbitragers decrease because only the one closest with validators take all. This is the first fairness problem. Even it is okay to sacrifice these minority arbitragers, general public using standard gas is 100% front run or back run by those successful arbitragers, whether the arbitragers are binance itself or not. This is the second fairness issue. I think @RayCCY solutipn to randomize the order of validators somehow is useful. Validators cannot mine two consecutive blocks are their own hardware problem. Such important change is not stated in bsc change log. I am very puzzled why this happens. |
Thank you, I will have a try later. |
|
bsc have 21 nodes, and the location of each node should be different? So many different areas need to be tested？ |
|
@myuniswap2000 i think so. But even you know the location of each node, you cannot get close enough to beat the existing profit makers. If your strategy relies on latency, it needs no evidence to conclude general public cannot beat the insiders. |
thank you,Another question, eth does not have this problem, is it because the eth nodes are relatively scattered?When will bsc improve this problem~ |
|
Eth has flashbots and Mev-geth |
hello, I don’t think anyone can cooperate with 21 nodes, but there are a small number of people who have a high probability of winning. They must have a way to get a position or static node. Is this inference correct? |
|
how about this impact to frontrun-tx? t+2 packed |
|
The change in transaction ordering does not matter at all. Validators can change the default ordering by modifying this repo, place transactions in whatever order they want, and hide that behaviour by giving their transactions a plausible gas price. If they are not doing this already, it is only a question of time before they do it. There are tens of millions of dollars to make by doing this (if not more) and there is no incentive to not do it. I certainly know that I would exploit transaction ordering to the maximum if I was a validator. Expecting that validators will renounce to that profit just to "act fairly" is wishful thinking. Reputation is not a factor because: 1) You can plausibly deny manipulating the ordering by making sure that the transactions have a plausible gas price, and 2) Nobody cares anyways, except a few experts that probably all commented on this thread (regular BSC users do not care at all about transaction ordering and MEV). On Ethereum, most of the searchers profit goes to miners through flashbot. On BSC, it most likely goes to validators too (or it will soon). It is just more opaque. If your business relies on fair transaction ordering, maybe it's time to pivot... |
|
Interesting thread |
|
@m-e-r-k-l-e-root I totally agree with you. My point was that the default ordering that is implemented in this repo does not matter because validators will simply change the default ordering (with a fork of this repo) to suite their MEV extraction needs. Edit: It does not matter so much if preferential ordering is implemented by either (1) sorting by time received and giving static connections to some privileged nodes (what you accurately describe as the current BSC situation), or (2) validators manually reordering the transactions (probably not happening right now). The effect is that only a select few will be able to efficiently extract MEV and those select few will either pay the validators to do so or be the validators themselves. Eventually, most of the MEV profit will go to validators. As others pointed out, it is also a problem on Ethereum (falshbots) and Polygon, although implemented differently. Very interesting thread by the way. I was wondering if others had noticed this and clearly I'm not alone. |
|
just impl a real time scanner, having found that: almost 99.9999% of txs related to swap/add/remove liquidity are front-runned by contracts with close relations to validators ... estimated 5-10M usd daily ... what a number there! |
I am also frustrated with the ones that are better than me at snipping :) but it doesn't mean automatically "close relations to validators" . The edge in this game might not be about "relations to validators" .... but more about so many other stuff like: knowing how P2P works for geth... the locations and configurations for validators/sentry nodes...the setup for Binance JSON-RPC Endpoints ..customizing geth & gobal infrastucture...cloud latencies....Virtual Machine Networking Performance .Even finding validators locations doesn't guarantee success.... because the sentry nodes might be behind firewall also and have their p2p port (e.g. 30311) closed for incoming connections. |
|
@patllc How do you even test if a node is validator/sentry node? You said that even if you know the location of each validator, you can't get close enough to defeat the profit makers, if you can get close to them, isn't that good enough? @gillbates how were you able to deduce if an account/contact have relations to a validator? @misu200 the problem is to minimize the latency to such validator/sentry nodes |
I think you are right, can you explain the conditions you mentioned？ what means the setup for Binance JSON-RPC Endpoints? |
|
I'm wondering why not more people are talking about this. For a few months, t+1 is no longer possible and it's always t+2 or more, no matter the gas price. Did anything change except increasing txs and growing chain size? Are the devs working on a solution to get back to t+1? |
AFAIK it's been pending block + 2 (mined after 2 blocks) for as long as I've used BSC. |
|
Agree with the OP. Not only is this change unfair, it doesn't stop spamming either. Two of the "anointed ones" with unexplained powers to almost always beat everyone else to the punch here: https://bscscan.com/address/0xaf25d6494a341675186f975c4cbc081fdac05dbc and here: https://bscscan.com/address/0xfb921a3925e751103a50c50fa8805798cae65d75 are spamming 30 or more transactions each in a single block when they both spot a big haul. @guagualvcha, @kyrie-yl, @j75689, @yutianwu - please - I and others implore you to roll back the transaction ordering algorithm back to the random order that it was, or at least some other method that is at least fair. Perhaps, as @patllc suggested, sorting the transactions of equal gas price by hash would reduce spam? It would incentivise arbitrageurs to "hold fire" on a single transaction until they have expended what they estimate to be enough hash power, rather than just immediately spamming dozens of transactions as soon as they spot a mark. It would also provide a feedback mechanism: bots monitoring pending transactions would see the current "best" hash and know not to send their current work in progress if its hash is "worse". This would, in theory, drastically reduce bot spam. It would turn the competitive playing field into a traditional PoW hash battle - the burden of which is born by the bot operators instead of the network. Bot operators get a fair slice of the pie proportional to hash power invested, and regular users and node operators will no longer be bogged down with spam. Although it has been correctly pointed out in this thread that validators are free to run whatever code they want, it does look like they just upgrade when a new version is released, with no conclusive evidence of direct MEV exploitation. |
|
Interesting thread. I have developed a transactions scanner that catches all arbitrage attempts and observed that I most frequently could not catch the transaction hash that wins the arbitrage. Is my scanner buggy or are these hashes generated by validators ? |
Closes harmony-one#4113 by sorting transactions by time received instead of using a hashmap based transaction ordering. This sorting is on top of the gas price and nonce sorting already implemented. Effectively, this allows the production of almost a deterministic order of transaction ordering, as opposed to a heap-based hash map ordering which is affected by Golang's internal operations. Consequently, arbitrageurs, who spam the network with a view to exist around arbitrag-able transactions, will have to focus on latency instead of network spamming. See also bnb-chain/bsc#269 and ethereum/go-ethereum#21358 for related issues in other chains.
Closes #4113 by sorting transactions by time received instead of using a hashmap based transaction ordering. This sorting is on top of the gas price and nonce sorting already implemented. Effectively, this allows the production of almost a deterministic order of transaction ordering, as opposed to a heap-based hash map ordering which is affected by Golang's internal operations. Consequently, arbitrageurs, who spam the network with a view to exist around arbitrag-able transactions, will have to focus on latency instead of network spamming. See also bnb-chain/bsc#269 and ethereum/go-ethereum#21358 for related issues in other chains.
|
Hi guys! Is anyone else trying to create an arbitrage earning system at this time? In my opinion, validators have nothing to do with it. I have been creating my script for arbitration for a long time and during this time I have had a lot of successful transactions. From receiving the window signal to committing the transaction, I spend 0.012ms, but even this is not enough for permanent success. I think it's all about strategy. |
@MaksimKel Can you share more information about your "system". Btw < 1ms signal delta was very impressive, congratulation! |
|
@MaksimKel oh is that so? And how did you get past the t+2 problem? |
|
"transactions with the same gas price are ordered by receive time" - what is the issue if you can simply add more gas and be first in the queue, assuming, that validators do not rearrange their own transactions? |
where is your node is hosted? |
and others
Rationale
With the latest update of geth 1.10, transactions with the same gas price are ordered by receive time, rather than by random order.
This receive time ordering may not be suitable to binance smart chain. Since BSC's validation mechanism is 21 validators taking turn deterministically to wrap transactions, deterministic transaction ordering can lead to below problems :
Also, I think this change is vital and wonder is that any reason not stating it in the change list?
Implementation
The origin of this change in ethereum mainnet is to avoid spam back running transactions occupying block spaces. To achieve the same goal in bsc, could we propose the below change instead?
Welcome any thoughts!
The text was updated successfully, but these errors were encountered: