Problem Submitting block 500 Internal Server Error

[nicoschtein]

Hi there, i have been using the PoS Branch without issues, but i have recently found that i was actually missing some block submits.
I have this error multiple times thorough the logs, i cannot find out why.
I can find blocks and submit fine, but sometimes this happen.

Here is the log:

2013-07-28 05:24:14,572 INFO template_registry # Yay, share with diff above 100000
2013-07-28 05:24:14,572 INFO template_registry # We found a block candidate! 00000000a6c596086b13cc16d5fa9845d81e8915d4925228476b2353bd2bd3b4
2013-07-28 05:24:14,580 ERROR template_registry # Final job validation failed!
2013-07-28 05:24:14,623 ERROR bitcoin_rpc # Problem Submitting block 500 Internal Server Error
Traceback (most recent call last):
  File "stratum-mining/lib/bitcoin_rpc.py", line 49, in submitblock
    resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
Error: 500 Internal Server Error

I have looked the code File "stratum-mining/lib/bitcoin_rpc.py", line 49, in submitblock but that was a dead end, just printing the exception that was 500 Internal Server Error.

Also https://github.com/moopless/stratum-mining-litecoin/blob/PoS/lib/template_registry.py#L256 is where the template_registry give the error output, and there is a comment there saying that should never happen.

Any clues?

[moopless]

This one is interesting. As you have already found, the error is being reported in lib/bitcoin_rpc.py in the following function.

def submitblock(self, block_hex, block_hash_hex):
    # Try submitblock if that fails, go to getblocktemplate
    try:
        resp = (yield self._call('submitblock', [block_hex,]))
    except Exception:
        try:
            resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
        except Exception as e:
            log.exception("Problem Submitting block %s" % str(e))
            raise

We try to submit the block by the submitblock method. If that fails with an exception, we try the getblocktemplate method (GBT). We do it this way so that we try to get submit the block as many ways as possible. If the GBT method fails we raise the exception and show it in the logs as we see here. That 500 Internal Error message is was was sent back from the coin daemon on us submitting using the GBT, which is not uncommon. It likely does not have GBT submit in the code any more so it throws that error (I am pretty sure all NovaCoin forks use submitblock). Unfortunately this masks the issue with the submitblock as we cannot see the exception that was raised from that call. You should add a debugging line to the code to see what the submitblcok exception is (why the coin daemon is rejecting the block). Maybe something like the following on https://github.com/moopless/stratum-mining-litecoin/blob/PoS/lib/bitcoin_rpc.py#L47

def submitblock(self, block_hex, block_hash_hex):
    # Try submitblock if that fails, go to getblocktemplate
    try:
        resp = (yield self._call('submitblock', [block_hex,]))
    except Exception as e:
        log.debug("Problem Submitting block with submitblock %s" % str(e))
        try: 
            resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
        except Exception as e:

The other interesting thing is the high difficulty on the share. Typically we will not see this line. https://github.com/moopless/stratum-mining-litecoin/blob/PoS/lib/template_registry.py#L239. Please add some debugging to the code above and let me know what the exception is.

[nicoschtein]

Hi thanks for your help!
Im getting bitcoin_rpc # Problem Submitting block with submitblock User timeout caused connection failure.
I think this related to a few spikes of 100% CPU usage.
Based on timestamps and CPU graphs, they do have almost same timestamps.
Have you ever gotten this error before?

[moopless]

@nicoschtein Good find. It very well could be resource contention. Can you increase the POOL_TARGET to decrease the load on the server and see if the issue persists?

I haven't seen that issue before. It seems that the there is a timeout connecting to the RPC server. Is is it local? Does your server have enough resources?

[salfter]

I've run into the same error with trying to get this working with BitGem. I started with the master branch, with which I lost four blocks before stopping it. I switched to the PoS branch, and lost a block after it had only been running a few minutes (fortunately). This is what's in the stratum server's log:

2013-08-04 15:22:09,277 INFO interfaces # 0000008b7319f110f596685d2541bd42542fb0b45ab4be26686cc6bf61c16ce0 (469) valid files
2013-08-04 15:22:09,612 INFO template_registry # Yay, share with diff above 100000
2013-08-04 15:22:09,613 INFO template_registry # We found a block candidate! 000000001f3b376c3d3ddc1bddde647b2644da72b1ed4e3081f504d038d5feb2
2013-08-04 15:22:09,621 INFO interfaces # 000000001f3b376c3d3ddc1bddde647b2644da72b1ed4e3081f504d038d5feb2 (537183) valid files
2013-08-04 15:22:09,636 ERROR bitcoin_rpc # Problem Submitting block 500 Internal Server Error
Traceback (most recent call last):
  File "/home/salfter/stratum-mining-litecoin/lib/bitcoin_rpc.py", line 49, in submitblock
    resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
Error: 500 Internal Server Error
2013-08-04 15:22:11,177 INFO interfaces # 000005b3c888beae234722f5dc46371b736f131703d50bfad43dde7d47205df4 (44) valid files
2013-08-04 15:22:12,294 INFO interfaces # 000000271846e8a5606c28f25f391531b5f4608c383943d0b307e50c451e2df0 (1676) valid files
2013-08-04 15:22:13,688 INFO DBInterface # Inserting 6 Share Records
2013-08-04 15:22:13,762 INFO block_updater # Checking for new block.
2013-08-04 15:22:18,762 INFO block_updater # Checking for new block.

The bitgemd log doesn't indicate anything's amiss. Is there something I can enable to get more debugging output from either the stratum server or bitgemd? The stratum server had DEBUG set to false and LOGLEVEL set to INFO for the log given above; I'll try again with true and DEBUG, respectively and see if anything useful comes up.

[moopless]

@salfter BitGem seems to be a PoS coin, so please use that branch, but I have not tested it at all. It should work since it is a fork of novacoin, but may have changed something with the 3 types of gem payout. Please see #48 (comment) on how to put in the debugging logging. I'll push it in the next couple of days.

@nicoschtein Any update on this issue? Are you still seeing the issue?

This in general is an external issue from stratum-mining-litecoin. Basically when we try to use the rpc standard to submit a coin we are getting an exception. To accommodate multiple coins we try submitblock first and if that fails we try a getblocktemplate submit. This unfortunately masks the logging for the submitblock, which is what you will be adding. It should help to identify issues.

[salfter]

BitGem seems to be a PoS coin, so please use that branch, but I have not tested it at all. It should work since it is a fork of novacoin, but may have changed something with the 3 types of gem payout. Please see #48 on how to put in the debugging logging. I'll push it in the next couple of days.

AFAIK the gem payout thing is just some sort of optional feature of the WIMP clients, which I've not used as my mining rigs are headless. Underneath that, there's still just one Bitgem unit. bitgemd has a getgems API call, but I've not tried doing much with it. I think it can be safely ignored.

I've edited lib/bitcoin_rpc.py as mentioned above and am waiting for another block to come in (it's on the slower of my two GPUs, so it'll probably be a few hours).

[salfter]

Looks like neither submitblock nor getblocktemplate are working:

2013-08-05 09:16:40,115 INFO template_registry template_registry.submit_share # Yay, share with diff above 100000
2013-08-05 09:16:40,116 INFO template_registry template_registry.submit_share # We found a block candidate! 000000003221ffc92833248afb0eb0e9a8a503473daf8ebb31ec3485d827c295
2013-08-05 09:16:40,124 INFO interfaces interfaces.on_submit_share # 000000003221ffc92833248afb0eb0e9a8a503473daf8ebb31ec3485d827c295 (334650) valid lanbox
2013-08-05 09:16:40,125 DEBUG protocol protocol.writeJsonResponse # < {"error": null, "id": 318, "result": true}
2013-08-05 09:16:40,133 DEBUG bitcoin_rpc bitcoin_rpc.submitblock # Problem Submitting block with submitblock: 500 Internal Server Error
2013-08-05 09:16:40,140 ERROR bitcoin_rpc bitcoin_rpc.submitblock # Problem Submitting block with getblocktemplate: 500 Internal Server Error
Traceback (most recent call last):
  File "/home/salfter/stratum-mining-litecoin/lib/bitcoin_rpc.py", line 50, in submitblock
    resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
Error: 500 Internal Server Error

[moopless]

We need to see what the requirements for submitting a block are for this coin daemon. The 500 Internal Server Error in my experience means that the coin daemon do not support the method. We have tried both submitblock or getblocktemplate and got the same issues. Can you check with the dev or in the source on what the requirements are for submitting a block?

[salfter]

bitgemd has a few of these in its log:

ThreadRPCServer method=submitblock
Sign failed

Early on, I had to make a change in script_to_pubkey() in lib/util.py. It assumed that coin addresses are 33 characters. I tweaked it to allow for the 34-character address I had been using for mining, but I'm guessing there's an incomplete address copy somewhere.

This code doesn't appear in the master branch, so in my copy of the PoS branch, I've replaced script_to_pubkey() in lib/util.py with script_to_address() from lib/util.py in master and replaced lib/coinbaser.py with the version in master. I'm now letting it run again, on the theory that the coinbaser in the PoS branch might've been out of date.

...time passes...

Still not working...got the same error in the bitgemd log as above. Creating an issue...done:

zeubea/bitgem#3

[moopless]

The code in the PoS branch was changed to work with PoS coins due to the signing. You mentioned that you made a change to the code to allow 34-character addresses. What did you change to make? Do you have the public key or the address set as CENTRAL_WALLET? It should be set to the pubkey for PoS coins.

[salfter]

The first change I made was to accept an address of either 33 or 34 characters. I was using an address, not a pubkey...if a pubkey is what I should've been using, I'll back out my changes and try again. I just got the bitaddress.org code working with Bitgem today, so I can get the pubkey for the address I've been using.

[moopless]

The signing would be wrong because it is missing the pubkey and instead is using the address. I would say back out all of the changes and set CENTRAL_WALLET = pubkey on the PoS branch. You can get the pubkey with coind validateaddress <address> and look for the pubkey.

$ diamondd listreceivedbyaddress 0 true
[
    {
        "address" : "dV1y8i31RKKy4WjxQLs5fyNJ4Qgc8YQMCC",
        "account" : "",
        "amount" : 0.00000000,
        "confirmations" : 0
    }
]
$ diamondd validateaddress dV1y8i31RKKy4WjxQLs5fyNJ4Qgc8YQMCC
{
    "isvalid" : true,
    "address" : "dV1y8i31RKKy4WjxQLs5fyNJ4Qgc8YQMCC",
    "ismine" : true,
    "isscript" : false,
    "pubkey" : "03a7ec501a5e1901307fc13e3ffd6bf2d4fc5e1c94db4ee3b67c79310292c2d3a7",
    "iscompressed" : true,
    "account" : ""
}

So I would set CENTRAL_WALLET = 03a7ec501a5e1901307fc13e3ffd6bf2d4fc5e1c94db4ee3b67c79310292c2d3a7 for this implementation.

[salfter]

I went ahead and did that...Bitgem txid a73e938756f2d41badc244276c8e0aac58389928661cc71404cbab77afe70080 was successfully mined through the stratum server. Next step is to try getting mmcfe-ng (or something similar) running.

[moopless]

Thank you for the update. Closing as it is working.

[Lollaskates]

I personally still get 500 server errors on block submits (specifically with Starcoin at the moment). It uses submitblock, and will occasionally error out with a 500 response from the coin daemon. It could very well be a bug in the coindaemons that we're all experiencing. I will put a few retry attempts in the code first and see how it fares.

I was able to submit the block after manually (that originally failed with a 500) successfully, so it is certainly sporadic.

EDIT: I managed to circumvent the issue somewhat with a minimum of 3 attempts of submission before it actually gives up and accepts the 500 error. In my test, it error'd once, and then was immediately able to submit it successfully right after. So that worked out very well. For the time being, put a simple 3-5 iteration or so loop around the submission code and add a break after the submission line that is relevant to you (submitblock or blocktemplate).

[nicoschtein]

I have this problem with cosmoscoin and bountycoin that will Always fail. 

But with starcoin, I'm having it only when the CPU usage is too high. 

I also have a weird amount of load from starcoind compared to other daemons. 

N I C O

On Fri, Aug 16, 2013 at 5:16 PM, Geoffrey Tucker notifications@github.com
wrote:

I personally still get 500 server errors on block submits (specifically with Starcoin at the moment). It uses submitblock, and will occasionally error out with a 500 response from the coin daemon. It could very well be a bug in the coindaemons that we're all experiencing. I will put a few retry attempts in the code first and see how it fares.

Reply to this email directly or view it on GitHub:
#48 (comment)

[torzsy]

Every PoS coin have this issue I tried: ORB / ADT / COL / MAP.

[kounav]

I m trying to connect it with PoS coin. This is the error:

ERROR bitcoin_rpc # Problem Submitting block 500 Internal Server Error
Traceback (most recent call last):
File "/dmdpool/PoS/lib/bitcoin_rpc.py", line 51, in submitblock
resp = (yield self._call('getblocktemplate', [{'mode': 'submit', 'data': block_hex}]))
Error: 500 Internal Server Error
2013-11-16 10:59:43-0500 [HTTPPageGetter,client] Unhandled error in Deferred:
2013-11-16 10:59:43-0500 [HTTPPageGetter,client] Unhandled Error
Traceback (most recent call last):
Failure: twisted.web.error.Error: 500 Internal Server Error

I use PoS branch. I tried to use pubkey but it says that it s invalid...
Please help...

[psandrene]

Hello,

I'm having the same "Sign error" problem that people were having with BitGem. I'm mining TAGCoin which has similar properties.

I can't put the pubkey in CENTRAL_WALLET because the code checks for a valid address at startup. Even if I comment out that particular check, the miner will not work properly without a valid address in CENTRAL_WALLET.

My question is: How can I get past the "Sign Error" issue if I can't use the 'pubkey' workaround that was suggested?

Thanks,
Pete

[fitsbach]

I'm getting the same problem with GrainCoin,

Can't use the pubkey as it will say it's invalid for CENTRAL_WALLET.

Anyone has an idea on how to workaround this?

Thanks,
Jimmy

[thatfrugalguy]

I am trying to set up a pool with GrandCoin and I also get the 500 Internal Server Error. When I post the data myself using curl, it returns: {"result":null,"error":{"code":-32601,"message":"Method not found"},"id":"1"}

Curl command:
curl -X POST -H "Content-Type: application/json" -H "Authorization: Basic " -d '{"params": [""], "jsonrpc": "2.0", "method": "submitblock", "id": "1"}' http://localhost:12378

Any thoughts?

[ethought]

I am still seeing this error on few POS coins. No solution yet?

Strange that the problem seems intermittent. Seems as if it is more of a POS coin problem than a stratum problem.

[uwemi]

Hello there i started mining before 12 hours with 1000khash/sec but the GDC profit was 220 ?

In coinswarz it says that with 1000khash/sec i should get 4.000 GDC for 24 hours ?