Error Codes

Peter Bushnell edited this page Sep 26, 2017 · 27 revisions

Overview

Here is an overview of the most common error codes and possible fixes.

E0001 Out of Order Share detection

Message

The block with height <i> found share <x> which is < than <y> of block <j> | Sometimes stratum inserts

####Example

2013-10-24 9:03:43 - FATAL --> E0001: The block with height 107643 found share 18 which is < than 19 of block 107642

####Explanation

The findblocks cron has found a block which has a share ID lower than a previous block found. This can happen with fast block finding coins or if the shares tables ID column had a roll over (starting at ID of 1 again).

####Fix

This is not a fatal error yet unless the shares ID column rolled over. findblocks will try to autofix the issue by changing the shares. It will re-order the shares so a following run should not fail.

If a rollover is found, the fix is to run an update on the blocks table to mark all blocks with valid share_id fields to a share_id of 1. That way the cron will skip the E0001 detection and continue to run with the rolled over share_ids. The round boundaries might be wrong, so a possible solution would be to assign a share manually with the highest non-rolled over share_id. That way the round should be calculated properly.

There is no way for MPOS to auto-fix rolled over shares table since it would remove the ability for this and other error detection systems.

E0002 Upstream shares not found

Message

E0002: Failed to fetch both offending shares: <x> and <y>. . Block height: <n>

Explanation

findblocks was not able to fetch both shares that are causing the Out of Order problem. Sometimes, if blocks have been accounted for already, shares can't be fetched because they were deleted.

Fix

Since a block has already been accounted for, there will need to be manually assigned a share to the found block <n>. Some things to keep in mind:

  • Do not use a valid upstream_result share
  • Do not include a valid upstream_result share in that round
  • Use more than just a few shares if possible to make it a round
  • Add valid block finder ID for block <n>
  • Add valid total shares for block <n>

E0003 Failed share update

Message

E0003: Failed to change shares order!

Explanation

findblocks tried to fix the shares order by updating both shares. This failed for some reason.

Fix

Nothing automated can be done here. Check for those blocks in question which share_ids did not get updated. Once found the solution field between those two shares can be manually switched. That will bring the blocks back in order with the shares.

E0004 Failed to reset block

Meessage

Failed to reset block in database: <n>

Explanation

findblocks successfully changed the out of order shares but failed to update the shares, share_id and account_id information in the block <n>.

Fix

Manually reset that block and try again.

E0005 Unable to fetch blocks upstream share

Message

E0005: Unable to fetch blocks upstream share, aborted: <error>

Explanation

None of the available shares were matched for this block. This should never happen due to the fallbacks implemented to match a block against a share. If such an issue arrises, try assigning shares manually.

Fix

See E0002

E0015 Potential Double Payout detected

Message

E0015: Potential Double Payout detected

Explanation

For each block that is going to be paid out, a new value is stored in the settings table: last_accounted_block_id. Note that the system stores the ID of the previously paid out block, not the height. Usually this procedure is pretty solid, but occasionally users run into this error. Most of the time, it was caused by an orphaned block not marked as orphaned at the time the payouts started to run. This is not something MPOS can catch, blockupdate is run prior to doing a payout, but with some coins blocks are not marked as orphaned in the upstream RPC so MPOS wouldn't know about it. Rare occasions showed that crons did not run through, then manual intervention is required.

Fix

If a payout failed in mid-run, first check for previous errors. If the block has not been confirmed yet, a potentially fix would be to delete all transactions related to this block and try to re-run it. If the block did confirm and has already paid out to users, all that can be done is to mark it as accounted. The last block being paid out can be found in the settings table under last_accounted_block_id. That block is probably not marked as accounted. To rectify this change the accounted column flag in the blocks table for the block identified by the number from last_accounted_block_id.

For example:

SELECT * FROM settings WHERE name='last_accounted_block_id';
+-------------------------+--------+
| name                    | value  |
+-------------------------+--------+
| last_accounted_block_id | 318204 |
+-------------------------+--------+
UPDATE BLOCKS SET accounted=1 WHERE id=318204;

If users did not get paid out, the cron will not be able to recover if shares have been deleted already.

In case of an orphan block not marked properly, just re-run the cron with the force option.

E0062 Block has no share_id, not running payouts

Message

E0062: Block has no share_id, not running payouts

Explanation

This is usually a follow up error caused by findblocks not running properly. Check the findblocks logfile and fix those errors, then payouts should continue once you force the run. If not, the same error message will appear.

Fix

See above errors for a fix in findblocks cron.

E0063 Upstream share already assigned to previous block

Message

E0063: Upstream share already assigned to previous block

Explanation

A share was assigned to two blocks simultaneously due to a fast find rate for the pool.

Fix

Two ways, the easy and the hard way:

EASY

  • Remove the block in question and force runs
  • If payouts were done already, those are probably wrong
  • If no payouts were done, this block is lost for payouts

HARD

  • Assign the correct share to the new block
  • Delete all transactions (if any) from that block
  • This clears out any wrong transactions for this block
  • Mark the block as unaccounted, if not already
  • Allow another payout run to pick it up as unapid
  • Change the last_accounted_block_id in the settings table to the block ID before
  • We want this block to payout now, so we have to go back one block
  • Rerun the run-payouts.sh cron

E0078 RPC method did not return 200 OK

Message

E0078: RPC method did not return 200 OK

Explanation

This is caused by the RPC service not accepting the sendtoaddress call for a payout with a proper http 200 OK code. Some coins are causing issues by accepting the sendtoaddress call internally but sending a http 500 Internal server error back to MPOS during processing. For MPOS it looked like the transaction failed but it was indeed accepted by the RPC. To be sure, we are now failing the cronjob and give Pool Ops a chance to check.

Fix

Workaround: Please check your coind transactions for the payout that you can find in the payouts logfile. You can use the supplied scripts/validate_payouts.php script to find any missing records in your coind. Please double check those, the script is not 100% perfect and may sometimes not pick up a transaction when it should. If you are missing that users payout that caused the error, manually trigger a sendtoaddress call from the wallet to pay them out, then force a cron run to restart the payout process.

Keep in mind: This is a workaround for an issue upstream. It does the trick and protects pool operators from double payouts.

E0079 Wallet does not cover payout

Message

E0079:  You have exceeded the pools configured COINS warning threshold. Please initiate a transfer! 

Explanation

Wallet does not have required funds to meet payout demands when cronjob is run.

Fix

*Insure the wallet "" is your main wallet receiving the funds from stratum-mining.

*Check Wallet Information in Admin panel to insure that the wallet funds are there and its reading from daemon.

*Confirm that there isnt something strange going on with payment settings

*Check that the users credits are not abnormal or extremely large.

E0081 Failed to insert new block into database

Message

E0081: Failed to insert new block into database

Explanation

During high load on a database findblocks could run into timeouts when waiting for a transaction lock. You will find this error in your logfile, finding new blocks is aborted until this block has been ported into the database.

Fix

You could try running the findblock cron again. Since the block was not added to the database, another run should try to pick it up again and add it. If this does not succeed find your bottleneck. It could be related to the archive cleanup not working properly and your archive growing way too large. This can be an issue when running PPLNS based payouts.

If that fails and forcing the cron did not fix it, you can insert it manually into the database. The values required are visible in the logfile. The blocks information is printed before the error message in a table format. Add those into your blocks table and try running your cron again.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.