Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Share miscalculation/share scamming #620

Closed
linas opened this issue Dec 29, 2020 · 18 comments
Closed

Share miscalculation/share scamming #620

linas opened this issue Dec 29, 2020 · 18 comments

Comments

@linas
Copy link

linas commented Dec 29, 2020 •
edited
Loading

There appears to be reasonable evidence that someone has figured out how to confuse the calculation of shares for (an older version) of this pool software. It appears to affect at least two eight different pools. It's a doozey. Here we go.

An earlier version of this bug is reported here: tevador/RandomX#200 because I mistakenly assumed this was a RandomX weakness; it almost surely is not, its a pool weakness. To recap:

QRL, the "Quantum Resistant Ledger" uses RandomX. It's a small coin, about 14 MH/sec grand total network hashrate. Pool2mine is a small pool for QRL, about 84KH/sec. So, about 3x or 4x per day, someone joins this pool, mines at a rate of 2-5MH/sec for 2-5 minutes, and then leaves. When they mine, they pick up almost every block, and every block they pick up has a difficulty of 1% to 50%.

At first I thought: "a hah, someone is guessing the easy blocks and sucking them up". Nope. What I now think is happening is that someone is aiming a firehose of shares: about 100x more than what this pool normally sees, and approx 1/3rd of the total network hash-rate, and is tangling up the share calculator to credit them with more than they actually shared. (I mean, if this was a RandomX weakness, they would just solo-mine, instead of attaching to a pool.) This miner is probably a regular Monero miner (Monero has a network hash rate of 1.5GH/sec, so a 3MH/sec miner is not that big. But for QRL, 3MH/sec is huge.)

  • MiningOcean is seeing the same thing - every few hours, someone comes in with 25MH/sec to 30MH/sec -- this is 2x the network rate. It's huge. (You can see this on the dashboard)
  • Miningocean update: The "usual" miningocean rate appears to be about 1MH/sec - the last big peak was about 28 Dec 2020 5PM at 15 MH/sec. Since then there are many many more peaks, but they are smaller: instead of being 3x/day@25MH/sec they are now every 15 minutes, peaking at 7MH/sec and dropping to zero (well, dropping to 1 MH/sec which is the normal baseline for this pool). This behavior started after I reported this bug, which may be a coincidence, or not. It might be an attempt to hide the ludicrously large hash-rate spikes...)

For Pool2mine you can see evidence for this as follows. Here's some recent activity:

Date                 Block     Difficulty of block sequence
24 December  11:30PM 1319505 - 76% 88% 55% 93% 52% 38% 76% 37% 48%
25 December  4:44PM  1320554 - 16%
26 December: 7:51PM  None
27 December  2:03AM  1322615 - 66% 8% 50% 73% 1% 80% 7% 21%
             3:36AM  1322748 - 53% 75%
             9:16AM  1323005 - 20%, 29% 16% 14%
             8:36PM  None
             9:17PM  None
28 December  3:09AM  1324078 - 27% 7% 8%
             5:31AM  1324194 - 94%
             6:55AM  1324248 - 11%, 8% 6% 12% 14%
             4:11PM  1324858 - 62%

So -- 27 December is a good example. This miner shows up at the indicated time, with maybe 5MH/sec(? approx?) hash power (this is maybe(?) 1/3rd(?) of total hash power for this coin), then, starting with block 1322615 they mined almost every block in a row, every 5 to 30 seconds, and then they leave about 7 minutes later. The network rate is supposed to be one block per minute.

You can view the above at the Pool Blocks page. The hash rate spikes are visible at the Dashboard. Note that the spikes there are are the hash-rate averaged over half an hour; the instantaneous rate is much higher. The miner is Q010300...91977ef and is visible in the top ten charts

  • This activity has been going on for several months (I'm hoping the pool admin will add to this report, and provide more reliable dates)
  • The miner is Q010300...91977ef They've submitted a total of 1985466073088 hashes for about 2335 blocks (so, about $2500 USD) - this is about 60% of the grand total hashes processed by this pool.
  • We're guessing it is someone who normally mines monero, where their hash rate is maybe 0.2% of the total monero network rate. But 2x to 4x a day, they show up at QRL to make a quick buck.

So, how much is being lost? Here's my experiment:

  • For the last ~30 hours, I've aimed a 1.8KH/sec miner at pool2mine.com and mined 218MH The block difficulty is about 725MH to 800MH in the last 24 hours. Reward is about 5.34 QRL/block Thus I expect to get (5.34 QRL / 750MH) * 214MH = 1.55 QRL. Pool is reporting actual earning of 0.987 QRL so the pool has under-estimated my shares by about 1/3rd. -- I'm getting about 2/3rds of what I should have gotten.
  • I'm assuming that the periodic super-miner has tricked the pool shares calculator into over-counting their shares by about 50% (i.e. about 3/2's so that 2/3 * 3/2 = 1)

If this is an old bug or a known bug, let me know. pool2mine seems to be running a Feb 11 2020 version of this s/w. No clue what version miningocean is using.
@linas linas mentioned this issue Dec 29, 2020
Closed
@linas
Copy link
Author

linas commented Jan 1, 2021 •
edited
Loading

OK, so I measured share submissions and payouts to three pools; one pool which blocked the scammer just a few days ago, and two which have not (yet). I submitted shares for 2-3 days, from small hash-power machines (so that it would average out). Here are the results.

First, the baseline: QRL block difficulties are variable; from 29 December 2020 to 1 Jan 2021, the difficulties and payouts low, high were:

 difficulty   reward  QRL/hash      date
  737988332   5.3342  7.228e-9   11AM 1 Jan
  938979454   5.3361  5.683e-9   10PM 30 Dec

Thus, a miner operating over these days might expect rewards in this range. Here are the actual results:

  start date    end date   submitted   balance QRL/Hash  Pool
30 Dec 17:00   1 Jan 17:00  120512453   0.5265  4.369e-9  miningocean.org
30 Dec 04:00  31 Dec 19:00  193467739   1.0224  5.285e-9  herominers.com
28 Dec 04:00  30 Dec 16:00  413303777   1.6050  3.883e-9  pool2mine pre-ban
30 Dec 22:00   1 Jan 16:00  132890290   0.8399  6.320e-9  pool2mine post-ban

This shows the timer interval over which I mined, the hashes submitted, the rewards collected, and the average reward per hash. Times are in GMT, rounded to the hour. All three pools are seeing spiking behavior (actually, eight pools are, more about that below). I alerted the pool2mine operator, who blocked the spiker. Thus, for pool2mine, I can show pre-ban and post-ban rewards. Post-ban, the rewards are within the expected range. More-or-less on-the-button. Pre-ban rewards were a disaster. The other two pools are paying out at well below the expected value.

Again, with the spiking:

  • pool2mine is the smallest pool, with a base rate of about 80 KH/sec. They were seeing spikes around 2-4 MH/sec every 3-6 hours.
  • miningocean.org has a base rate of 700 to 800 KH/sec; they see spikes of 20MH/sec to 50 MH/sec. Note that the base hash rate for the entire QRL network is 14MH/sec, so these spikes are 2x to 4x grand total network rate.
  • herominers.com has a base rate of 1.7MH/sec and sees spikes up to 25MH/sec

Of the three, herminers is the biggest pool, and is thus the hardest to spike, and thus pays out the best rewards (although significantly less than expected). miningocean is getting hit worse. pool2mine was getting crushed, before the ban.

Here is what "spiking" looks like:
herominers-hashrate-2

Note the orange graph upper left. The biggest visible spike has a label: 1/1/2021 5:50:49 PM Pool hashrate: 26.36 MH/sec

@linas
Copy link
Author

linas commented Jan 1, 2021 •
edited
Loading

Recap: if someone can mine at 2x or 3x the network hash rate, then why don't they just mine solo? Why do they hit up 8+ different pools for a few minutes at a time? Answer: they get more credits by hitting up small pools, and getting rewards out of proportion to actual hashes computed.

Who is the guilty party? Who are the affected pools? There are at least two. One is:

 Q010300059eb9badfdd06657299051697472919faf89baa864351e83cd5aec68c1f9957391977ef

He's hit (at least) eight pools. Here are screenshots. Look for Q010300...91977ef in each screenshot, notice how many hashes he's submitted (grand total) - he's always the biggest miner in the pool. Notice the hash rate, (usually 0.0 H/sec) and the "Last Share submitted" date (usually hours or days ago)

scam-aussie-pools com
scam-cp3o com
scam-miningocean org
scam-my2coins com
scam-pool2mine com
scam-qrl newpool cool
scam-qrl pool-pay com
scam-qrlmining com

Bonus: here are all the QRL pools. Note that most of them are seeing spikes.
qrl-spikes

@linas
Copy link
Author

linas commented Jan 1, 2021 •
edited
Loading

What's the detailed mechanism for pulling this off? I dunno. I have noticed only one tiny hint of a problem. You can do this experiment too.

  1. Stop your miner.
  2. Go to pool, record how many hashes you've submitted so far. Call this N0
  3. Start your miner with fixed-size difficulty. Let difficulty=SZ (for most people, this should be between 25K and 100K)
  4. Run it for a few hours/days
  5. Stop your miner. Record number of accepted shares. Call this NB
  6. Go to pool, record how many hashes you've submitted so far. Call this N1
  7. Subtract N1 - N0 this is how many hashes the pool gave you credit for.
  8. Compare to SZ * NR. This is how many hashes you submitted.
  9. They should be exactly equal. Exactly.

When I do this, the number of hashes is sometimes different, by maybe 3 or 18 or something. Out of a hundred-million hashes, whats 3 or 18? Who cares? Well, but this shows the pool is recording a number that is different from the actual number of submitted hashes. Which is wrong. And its wrong not by some even number, but by some weird fraction that came out of thin air.

Now, I can't test to see what happens if I submitted a few billion hashes, but I'm thinking that the math error is going to be ... significant ... significant enough to be worth exploiting. Just guessing here. Circumstantial evidence. But, you too can run these experiments. You too can reproduce these results.

@linas
Copy link
Author

linas commented Jan 1, 2021

p.s. I have not seen spikes on monero pools. However, this does not mean that monero pools are not vulnerable to to something similar, maybe on a smaller scale, or something.

@linas
Copy link
Author

linas commented Jan 2, 2021

FWIW, contacted all the pools that I could. Admins of both miningocean and herominers are in full denial; they're saying "that is just how it works".

@fm407
Copy link

fm407 commented Jan 2, 2021

i feel like this will stay a monologue my friend, this code is unsupported or supported by people that ask you for $ even to answer a simple question. Sad individuals

@muscleman
Copy link
Contributor

muscleman commented Jan 2, 2021 •
edited
Loading

@linas so where is the issue, redis storing the shares, redis accumulating the shares, truncation, miner calculating shares differently to pool.?
3-18 difference is negligible

@muscleman
Copy link
Contributor

systems very easy.

  1. set diff for miner job. https://github.com/dvandal/cryptonote-nodejs-pool/blob/master/lib/pool.js#L508
  2. miner submits solutions, pool records diff for miner in current round. https://github.com/dvandal/cryptonote-nodejs-pool/blob/master/lib/pool.js#L1064
  3. calculate worker shares for each unlocked block. https://github.com/dvandal/cryptonote-nodejs-pool/blob/master/lib/blockUnlocker.js#L126

@muscleman
Copy link
Contributor

muscleman commented Jan 3, 2021 •
edited
Loading

so i've given this some thought. maybe you could run a pool and insert some code into the pool.js module that records all shares in a file.

somewhere around here. https://github.com/dvandal/cryptonote-nodejs-pool/blob/master/lib/pool.js#L1076

add something like this

log('warn', logSystem, 'miners hashes %s', [job.difficulty])

then run you test and accumulate shares from file

would be nice if miner could have share logging turned on, then we could see where the issue occurs

think this would highlight several things

  1. problem is miner side or pool side
  2. problem could be redis accumulation

@linas
Copy link
Author

linas commented Jan 3, 2021

It would improve the "believability" of this issue, if someone could reproduce this, especially, my last comment #620 (comment) - I've spoken to one person who is in complete denial, but is unwilling to perform basic tests (citing "proprietary modifications" to this code base).

As to where the problem might be: I've instrumented the miner (xmrig) to see how it works; I don't think its the miner; when it prints accepted (1305/0) diff 500054 that really is correct; especially when working with fixed difficulty. So in this case, the pool should record exactly 1305*500054 hashes (I haven't been checking this one... I guess I should.)

Being off by 3 or 18 out of a hundred-million hashes smells more like a floating-point rounding error, rather than (say) a thread race condition. Thread-race conditions (e.g. some read-modify-write was not locked/atomic) would wipe out an entire share, not a fraction of a share.

@linas
Copy link
Author

linas commented Jan 3, 2021 •
edited
Loading

Fuuu... in a different pool, I asked it to send me fixed-sized difficulty. I picked exactly 500000. It sends me 500054 ... why? I dunno, I assumed some kind of prime-number nonce magic or something. When the miner finds something, it reports accepted (nnn/0) diff 50054. If I look at the pool, it increments by 500000 and not 500054 .. what happened to those extra 54 hashes? (That works out to 0.0108% which is way too small to be a "pool fee" or "donations to core team") ... is my miner crazy, or is the pool crazy? This is ... irritating... the amounts are small, but why is there any difference at all?

@muscleman
Copy link
Contributor

muscleman commented Jan 3, 2021 via email

@linas
Copy link
Author

linas commented Jan 3, 2021 •
edited
Loading

Here's another screen grab showing very regular, but absolutely immense oscillations of the network hash power:
qrl-hash-power

Found that at https://explorer.theqrl.org/ -- assuming 60 seconds/block the period is about 95 minutes ... why would there be a factor of 10x or 20x oscillation of network hash power, that is extremely regular? What would cause/motivate miners to "pile on" like that? Why would this be a beneficial strategy?

Answer
Q: When is the best time to mine? A: When the block difficulty is low. Q: And when is that? A: Well, you look at the difficulty. If its low, hit the network with 2x to 4x the network hash rate, mine a block every 5-20 seconds. The QRL network notices this, and gradually auto-adjusts the difficulty to make it harder. This takes about 30-60 minutes to happen. Once the difficulty gets hard, you quit, and go back to mining monero. About 30 minutes later, the QRL network difficulty drops, making it easy to mine, again. Rinse and repeat.

Why does this work? Well, if you constantly mined QRL with 2x to 4x the current network hash rate, then the block difficulty would slowly adjust to the new network hash rate -- the difficulty would be 2x to 4x harder. With current QRL network settings, it would take about a day(?) to level out at the new difficulty. But if you can aim a firehose at it, and pulse it, like this, the difficulty stays low. Mining stays easy.

To put it differently: if you control 2x to 4x the network hash rate, and you pulse it, then mining is 2x or 4x easier, than if you mined at a constant rate. For the same amount of hash power, you get 2x to 4x more coin! What a deal!

Somewhere, the QRL specs say that "enough blocks are left for 200 years" but that assumes that the mining rate is approx constant from day to day. i.e. that the network difficulty auto-adjustment is able to respond to mining rate changes. Clearly, it can't: its responding much too slowly. Someone will mine out that 200-year supply in just a decade or so, cause they can mine at 5-20 seconds/block.

Anyway, I think that is the answer for the wild hash-rate swings. Someone is pumping the network, mining while its easy, then backing off soon as the difficulty starts increasing in response. Here's my own, independent graph of time-to-block-discovery, showing the 90-minute cycles.
p2m

@muscleman
Copy link
Contributor

muscleman commented Jan 3, 2021 via email

@muscleman
Copy link
Contributor

muscleman commented Jan 3, 2021 via email

@muscleman
Copy link
Contributor

muscleman commented Jan 3, 2021 via email

@xmrminers
Copy link

noticed the same shenanigans on my pools:

i have set fixed diff in the miner at 200000: "user":"cm.....Z642RC1x.200000"

Pool side: (Thread 2) Accepted trusted share at difficulty 200000/206880

but on the miner side i get: new job from pool.xmrminers.club:4444 diff 200007 algo cn-pico height 630147

So there's def something odd going on

@muscleman
Copy link
Contributor

muscleman commented Jan 4, 2021 •
edited
Loading

well, just checked it out. if you using my pool software. thats version < 2.0.0 there ain't an issue.

So if your looking to mine on one of these pools.

https://fastpool.xyz

https://newpool.pw

https://www.walemo.com/

Screenshot from 2021-01-03 23-16-20

@dvandal dvandal closed this as completed Feb 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@linas @muscleman @dvandal @fm407 @xmrminers