got no confirmation from server! busy? #99

Closed
mpenagar opened this Issue Mar 14, 2017 · 34 comments

Comments

Projects
None yet
6 participants

Hi!

I am mining with pool.burstmining.club and geting many "no confirmation" error messages (25% of submitted nonces):

09:57:44: XXXX: got no confirmation from server! busy? (4d 16:13:28)

I noticed this when I updated from version 1.4.3 (not shure) to 1.5.0 & 1.5.1. Any idea? Should I try just changing the pool or does any other get the same error on diferent pools?

Thanks!

exciler commented Mar 14, 2017

Hi!

I also noticed this problem but was not sure if it is a problem with my setup. I tried different pools and the problem stays the same.
Also I noticed that the "got no confirmation.." message appears immediately after the nonce is submitted. In mining.conf I have set timeout to 30 and also tried 45... this does not affect the situation at all.
Also I noticed some "Segmentation faults" since version 1.5.0... Just to mention that. As I´m pretty busy right now, I can not dig into it until next week or so...

This is what i see on the log file (see Connection reset by peer):

14.03.2017 10:16:53 (5, src/Miner.cpp, 327, Information): Mikelon: nonce found (1d 22:35:28)
14.03.2017 10:16:53 (5, src/NonceSubmitter.cpp, 56, Information): Mikelon: nonce on the way (1d 22:35:28)
14.03.2017 10:16:53 (5, src/NonceSubmitter.cpp, 67, Debug): Submit-loop 1 (1d 22:35:28)
14.03.2017 10:16:54 (5, src/NonceSubmitter.cpp, 81, Information): Mikelon: nonce submitted (1d 22:35:28)
14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/NonceSubmitter.cpp, 67, Debug): Submit-loop 2 (1d 22:35:28)
14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/NonceSubmitter.cpp, 67, Debug): Submit-loop 3 (1d 22:35:28)
14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/Response.cpp, 51, Error): Error on receiving response!
Connection reset by peer
14.03.2017 10:16:54 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

14.03.2017 10:16:54 (5, src/NonceSubmitter.cpp, 96, Debug): JSON confirmation (1d 22:35:28)

14.03.2017 10:16:54 (5, src/NonceSubmitter.cpp, 167, Warning): Mikelon: got no confirmation from server! busy? (1d 22:35:28)

Pascal66 commented Mar 14, 2017 edited

mpenagar commented Mar 14, 2017 edited

Yes, I am using https://wallet3.burstnation.com :

"wallet" : "https://wallet3.burstnation.com:8125"

Pascal66 commented Mar 14, 2017 edited

Try to launch your own server wallet, and then use it as wallet when it's synchronized
Using an external wallet cause lag for best deadline.. Imagine 10.000 users use the same wallet all around the world in the same time...
Your own wallet use peers to quickly know what's the best DL around. Then avoid your 25% of reject...

exciler commented Mar 14, 2017

I don´t think it is the wallet... the error occurs on the pool connection.
My first impression: timeout is not applied correctly, if I set a timeout of 30s it seems wrong that the above error occurs immediately for 3 Submit-Loops within the same second. Look at the timestamps...

wallet syncing... I will tell you if there is any improvement with the local wallet.

Owner

Creepsky commented Mar 14, 2017

I dont' think this is the wallet. I won't stop you to use a local wallet, because it is always a good idea to use one (safer, faster), but there is nothing wrong in using an online wallet.
If it worked in previous versions, it should work in the current one. So the error came with one of the recent additions.

I'm using the same online wallet (https://wallet3.burstnation.com:8125), everything fine on Windows. However, same system, on Linux it don't work. This indicates a difference in the implementation of a system resource (most likely Mutex or Socket, for example the timeout like exciler said).

Also, the error occurs on communication with the pool, not the wallet. Only if you use the wallet as the pool (solo mining), this would be an issue.

I'm debugging it right now, so if you give me some time, I'll eliminate the error.

Axadiw commented Mar 15, 2017

Is it possible to bypass this bug before fix will be released? I've tried to increase timeout, but unfornatelly it didn't helped

@Creepsky Creepsky added a commit that referenced this issue Mar 15, 2017

@Creepsky Creepsky Version 1.5, hot fix 2, #99 6a8c6b5

@Creepsky Creepsky added a commit that referenced this issue Mar 15, 2017

@Creepsky Creepsky Changed X_Plotfile in the submission-request to plain text #99
Before this fix it was a base64 encoded string, that in some cases had an CR LF in it.
d378be9
Owner

Creepsky commented Mar 15, 2017 edited

The timeout was a false alarm.
It was a really well hidden bug in the POST body of the nonce submission to the pool.
The body looks like this:

var1=val1
var2=val2
...

A (CR, only Windows) LF indicates the next variable. With the forwarding system I build in a new POST variable "X_Plotfile", so that the forwarding miner can know in what plotfile the nonce was found.
It was encoded in base64. Under some circumstances the result base64-string had a LF in it, so that the body looked like this

var1=val1
X_Plotfile=<HERE COMES A LONG BASE64 STRING><LF = LINEBREAK>
<HERE IS THE REST OF THE BASE64 STRING>
var2=val2

This is a malformed body and gets rejected by the server.
Just the technical background :)

I pushed a new hotfix: https://github.com/Creepsky/creepMiner/releases/tag/2.5.2
Sorry that it took so long.

Axadiw commented Mar 15, 2017

I've installed https://github.com/Creepsky/creepMiner/releases/tag/2.5.2 and this problem still persists.

Do I have to change anything in the config file?

Owner

Creepsky commented Mar 15, 2017

Hm strange 😕 On what pool do you mine?
Normally you don't have to change anything.

It could be that your pool server has deactivated chunked transfer encoding.
Can you please test it with the branch https://github.com/Creepsky/creepMiner/tree/fixed-content-length?
There I changed it to fixed content length.

Axadiw commented Mar 15, 2017

The same problem persists on `fixed-content-length' branch.

I'm mining on pool http://pool.burstmining.club:8124 with this configuration:

 "urls" : {
            "miningInfo" : "http://pool.burstmining.club:8124",
            "submission" : "http://pool.burstmining.club:8124",
            "wallet" : "https://wallet4.burstnation.com:8125"
        }

Logs:

18:22:26: XXXX-XXXX-XXXX-XXXX: nonce found (3d 10:06:44)
18:22:26:       nonce: 42985948
18:22:26:       in: /mnt/burstcoin/optimized_plots/XXXXXX_40000000_4000000_4000000
18:22:26: XXXX-XXXX-XXXX-XXXX: nonce on the way (3d 10:06:44)
18:22:26: XXXX-XXXX-XXXX-XXXX: nonce submitted (3d 10:06:44)
18:22:26:       nonce: 42985948
18:22:26:       in /mnt/burstcoin/optimized_plots/XXXXXX_40000000_4000000_4000000
18:22:29: XXXX-XXXX-XXXX-XXXX: got no confirmation from server! busy? (3d 10:06:44)

@Creepsky Creepsky added a commit that referenced this issue Mar 15, 2017

@Creepsky Creepsky Test for #99 e49ab96
Owner

Creepsky commented Mar 15, 2017

Then I have some more questions:

  1. What is your submissionMaxRetry?
  2. What is your timeout?
  3. Can you please copy paste the whole log from 18:22:26: XXXX-XXXX-XXXX-XXXX: nonce found (3d 10:06:44) to 18:22:29: XXXX-XXXX-XXXX-XXXX: got no confirmation from server! busy? (3d 10:06:44) from your logfile? For example into gist.
  4. Do you get this error on every found nonce? Or just on some?

The only workaround that I can think of is to raise your submissionMaxRetry and timeout or you could use an older version (1.4.6).

Axadiw commented Mar 15, 2017

submissionMaxRetry = 3
timeout = 600

I have changed config file to log all:
log: https://gist.github.com/anonymous/cd2b8168be86a990a6a285e4a832eef2

I guess thta here's an error:

15.03.2017 19:04:46 (5, src/Response.cpp, 51, Error): Error on receiving response!
No message received
15.03.2017 19:04:46 (5, src/Response.cpp, 53, Error): Stackframe
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)

It's happening for every nonce.

I've tried with raising timeout and submissionMaxRetry, but it haven't changed anything

Owner

Creepsky commented Mar 15, 2017

Thanks a lot!
Yes the error you are showing is really strange. No message received means the server sent us a message, but it is empty.
So he accepted your request, but somehow he decides that a response is not needed. Or he can't answer because it is malformed again.

Sorry for bothering you, but can you please test the updated fixed-content-length branch? I kicked out every custom POST parameter (this will break the forwarding feature).

Axadiw commented Mar 15, 2017

Hey,

this fixed-content-length branch work's great - nonces are now confirmed by the server

Good job!

Owner

Creepsky commented Mar 16, 2017

Nice.
If some of you guys have time, could you please check the devlopment branch or the newest pre-release 1.5.2?
If it is ok I will merge it into master.

Ok,

I will update my nodes tonight and tell you.

Thanks!

Axadiw commented Mar 16, 2017

Hmmmm, looks like my optimism was premature - I've tested miner on my other machine (that is basically forwarding nonces sent by other miners), and I'm still recieving

got no confirmation from server! busy? (03:57:15)

error there.

Detailed log:
https://gist.github.com/97ffd5c540e281ec8698a2dd778c127f

It looks like the problem is here:

16.03.2017 14:51:37 (5, src/Response.cpp, 51, Error): Error on receiving response!
Invalid socket
16.03.2017 14:51:37 (5, src/Response.cpp, 53, Error): Stackframe
Miner::submitNonceImpl (in "src/Miner.cpp", line 432)
NonceResponse::getConfirmation (in "src/Response.cpp", line 87)
Response::receive (in "src/Response.cpp", line 29)
Owner

Creepsky commented Mar 16, 2017

If you are using fixed-content-length - the forwarding feature is broken there.
Please try the devlopment branch or the newest pre-release 1.5.2.

Axadiw commented Mar 16, 2017

I'm using 1.5.2 version (it's even included in log file)

Owner

Creepsky commented Mar 16, 2017

Hm okay.
I pushed a new commit today into the development branch and re-tagged 2.5.2 to this commit.
I mean this here: https://github.com/Creepsky/creepMiner/commit/466b5b920e0ba21b7089aac4999aea874e1635d9.
Should have said this, sorry.

Axadiw commented Mar 16, 2017

But I have been using newest commit (466b5b9 to be exact).

In order to confirm it, I have cloned the repo again, and checked out 466b5b9 commit, run mining again. Unfortunatelly got no confirmation from server are still happening for every nonce.

Here are full logs:
https://gist.github.com/a470690da55b5d4bca084b28dbdea2b8

PS.
I'm not 100% sure, but I think that yesterday I was running 2.5.1 completely fine, and those errors have came up out of nowhere (I haven't changed anything in my configuration, and haven't been restarting the miner).
So maybe it's connected somehow with the pool I'm currently using? (http://pool.burstmining.club)

Owner

Creepsky commented Mar 16, 2017

Ah ok. Thought you were on the old 1.5.2.
Something I noticed in your logs are the following lines:

16.03.2017 19:22:21 (16, src/MinerServer.cpp, 192, Debug): Request: /burst?requestType=submitNonce&nonce=2813750561031&accountId=8754600706522934201
16.03.2017 19:22:21 (16, src/RequestHandler.cpp, 294, Information): Got nonce forward request (17:51:33)
	nonce: 2813750561031
	account: XXXX-XXXX-XXXX-XXXXX
	in: L21udC9hY2Qvandhc2Zhd2ZAZ21haWwuY29tL3Bsb3RzLzg3NTQ2MDA3MDY1MjI5MzQyMDFf

To be exact:

in: L21udC9hY2Qvandhc2Zhd2ZAZ21haWwuY29tL3Bsb3RzLzg3NTQ2MDA3MDY1MjI5MzQyMDFf

This strange hash (actually it should be the name of your plotfile 😄) means your hub is 1.5.2, but your worker < 1.5.2. But in the end it should not change anything.
Just out of curiosity I tested to mix different versions (hub = 1.5.2, miners = 1.5.1 and 1.5.0), with the result, that on my system it was all confirmed.

Maybe you are right and this is pool related. I should hop over to your pool and test it there.

Also you could try to copy this request.cpp into your own local request.cpp and then rebuild the miner with make (should be enough if you do it only for the hub).

Axadiw commented Mar 16, 2017

Ok, to sum up my config:

Let's say that I'm using 2 configurations:

  1. 1 single miner with couple of TB's attached directly. On this miner I'm using 1.5.2 CreepMiner that's working correctly (1.5.1 was triggering got no confirmation from server errors).

  2. Multiple small miners (that are connected to one of the cloud drives) that are forwarding all of their nonces to hub, that is actually connected to target pool. My philosophy here was to "set and forget" all those small miners, and point them to "HUB" that I will be able to upgrade / change to other pool without bothering small miners.

You're correct about version of creepMiner, I've installed 1.5.1 successfully on small miners, and now I was just working on Hub's installation of creepminer.

And here's interesting part:

I have tested version 1.5.1 of the miner on the HUB, and it looks like it's working correctly!
Just when I was going to write on github about this discovery, @Creepsky have posted new version of request.cpp file. As you have requested, I have installed 1.5.2 version of the CreepMiner, than I have updated request.cpp with the new version, and compiled everyting.
In the end it looks like this combination is working as well!

So to sum up, it looks like:

  • Configuration 1) from above example (single miner) is working correctly only with 1.5.2
  • Configuration 1) from above example (couple of small miners + hub) is working correctly only with stock 1.5.1 or 1.5.2 with request.cpp fix from last comment installed on the hub.

Same for me....

  • 1.5.1 working

  • master(1.5.2) 100% unconfirmed

  • development (1.5.2) 100% unconfirmed

  • development (1.5.2) + request.cpp working

Owner

Creepsky commented Mar 16, 2017

I hopped over to your pool and got the same error.
Then I tried different things and it seems, the server don't like this line here.

I will change the code and re-tag 1.5.2 again. Maybe you can give it another shot.
Thank you all for testing and helping (and for your patience)!

1.5.2 release
development branch

development branch: All nonces confirmed, well done!!!

Axadiw commented Mar 17, 2017

I also confirm that current 1.5.2 release is working fine on the hub

Owner

Creepsky commented Mar 17, 2017

finally we got it! :)
I will merge this into the master branch and then close this issue.

Creepsky referenced this issue Mar 17, 2017

Merged

hotfix #99 #100

Creepsky closed this in #100 Mar 17, 2017

bryhardt commented Jun 5, 2017

I am experiencing this issue regularly on the .club pool with 1.5.2 and 1.6.0. I believe this occurs when the pool is too busy to handle the request. As pointed out earlier, the submit-loop occurs instantly and burns through the max retry count immediately since the server can't handle any of the requests. I put a sleep, 5 seconds, in the submit try loop after the first attempt and i get much better results. I don't know CPP very well, so I am sure something better could be implemented to delay sending the next request if the connection was no good.

Owner

Creepsky commented Jun 5, 2017

@bryhardt Very nice idea! Can you please open a new issue for this + if you have git, can you make a pull request on development branch?
std::this_thread::sleep_for(std::chrono::seconds(5)); should do the trick.

bryhardt commented Jun 5, 2017

i will do a pull... that is exactly what i used...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment