Syncing gets stuck around block >~2450000 #8194
Comments
What exactly happens then? Does it stay stuck on the same block and stop syncing? Does it start to ban other peers? |
The errors and warnings stated above happen constantly (see gist for details) until I give the hangup signal around Indeed, peers are banned. |
|
One thing I didn't mention, if I continually restart the daemon, it seems to sync a few chunks of 20 blocks upon start, but it only works for a minute or two until getting stuck like in the log. It's obvious to me that there are some malicious nodes out there just handing me junk blocks. But could it really be an eclipse attack? It seems to me like adding more peers would reduce the chance quite a bit, but I'm not sure the problem here is just that. It seems like maybe a combination of a bug and malicious peers, but that's just speculation. |
|
Which log level is the log you posted?
You appear to have this issue for a month and no one else reported something similar so I don't think so, otherwise there would be more reports about similar issues. It's probably either an issue with your setup or a bug that gets triggered in your setup, but looking at the logs I don't see anything obvious yet. |
Log level 1. Let me know if you want a different log level. |
|
Can you post log level 2? |
|
Thanks for your quick feedback. I use the same VM image for several coin daemons, and only Monero has syncing problems, but it's the only cryptonote coin I run. I can't really imagine a more generic setup, but after searching around, you're definitely right that no one seems to have this problem. I'll make a clean VM and see how that does. |
|
I did a quick dist-upgrade to Debian 11 without rebuilding Monero and the problem seems to have resolved itself. Really strange. No problems now. Why it happens at 95% synced is beyond me. It doesn't happen at earlier blocks. It seems that running Debian 10 and other non-new distros should be advised against. Debian 10 is "oldstable" which is still supported, so that's mildly concerning that people are probably using it still. I'll post some new logs so they can be compared against the ones that failed. |
|
Log level 2 after the problem was resolved by upgrading to Debian 11: https://gist.githubusercontent.com/tryphe/fac92713a0fab5aa6a75c6ae40905573/raw/c3262979e8e0f299b290bde46ac096c48cd6e144/gistfile1.txt Thanks again for your feedback @selsta! I can't spot anything in the log that's different. The warnings and errors look the same. It must be related to some buggy lower level library running on some Linux distributions. Hopefully this might help someone that has a similar setup but this does not necessary qualify as a Monero issue. Maybe a check for some older library could be enforced, although frankly I have no idea which library it could be 😛 |
|
Does everything still work? It's unlikely that your OS version is related here. |
|
@selsta yes. Ran dist-upgrade, did nothing else, and it's been synced ever since. I upgraded from Buster (Debian 10) which was up to date via apt, so there's a problem there. |
|
It started doing it again today and now the node is behind. Will try and make a new machine. And again, none of my other coin daemons have this problem, so it's a monero problem. But it doesn't really make sense that i was able to sync ~100k blocks fine at a point where it was stuck, and then nothing. |
That's what moneromooo saw from the logs. Now it still doesn't explain why there aren't more reports about this. I guess testing a different machine would be helpful. |
Not sure if this is helpful for your setup but I thought I'd add it too. |
|
Hi @selsta , I had a similar issue when trying to host the blockchain on my NAS. Turned out the problem was the NAS had some hidden service that "reindexed" its files every time ANY change was made on disk. This would lead monerod and this index service to compete for access to the file, which would grind the whole thing to a halt. Once I removed this NAS service, I was able to full sync from 0 in about 1.5 days total, with zero issues. Could you possibly also have a similar service interfering with your blockchain file? |
|
I was able to reproduce the bug with a new KVM instance with Debian 11 amd64 (vm software is up to date, host is up to date). Host is running other coin daemons with a similar setup. I tried to get a setup that is most similar to the one producing the bug. Steps I took:
It got all the way up to and stuck at block ~2508000 for hours, with many peers and lots of network traffic. Restarted the daemon, and it's syncing okay again for a few minutes. Same issue as in OP. We'll see how long until it gets stuck again. I'll try to see if I can reproduce without x running ( |
|
Is the blockchain stored inside the VM or on an external network drive? |
The blockchain is always stored locally and no processes are running that seem like they could interfere. Most processes have a fraction of a second of total cpu time used. |
|
The steps taken above were an effort to try and reproduce the bug on a different VM but on an identical setup. The problem still occurred. I've taken some additional steps to reduce the complexity and was able to reproduce the bug without X or related processes running ie. xfcond and I was still able to reproduce the bug.
Logs during this run: There is a reoccurring problem with |
|
I'll build with |
|
I'll try to sync up on a VPS and see if I can reproduce. Just to confirm, you used the master branch? |
Thanks. Indeed, head is at |
|
After 10 minutes of running I was able to reproduce the bug in debug mode. 7 blocks sync and then it gets stuck, lots of network chatter but no blocks being added. I gave the hangup signal after 10 minutes to keep the log smaller. Log with --log-level 1: https://gist.githubusercontent.com/tryphe/bc38a41e752fd127d3092901e1178051/raw/c17ca0b278450c5d29d89014604eb9fe87861525/gistfile1.txt Please let me know if there is anything else I can provide. |
|
It looks like the errors are possibly unrelated to the syncing problem. I tried a handful of old release versions back as far as 0.17.1.3. I still experience the syncing problem but no runtime errors. |
|
I started the sync process on a fresh VPS now. Will update you if it syncs fine or if I gets stuck too. |
|
8 hours later, so far no issues |
|
I unfortunately couldn't reproduce this on a fresh VPS with v0.17.3.0. Sync was without any issues. |
|
I'm not really sure what the differences are between our machines. While syncing past 95%, it takes me somewhere between 5 to 40 minutes to experience the stuck syncing problem. I'm on a residential connection so I have some packet loss. About 0.05% to 0.1%. Maybe that's the problem? I'll try and induce more packet loss to see if it makes the problem worse. |
|
Check for any options for long term tests. It's possible your CPU starts going wonky once hot. |
Checked everything related to CPU (temp, freq) and it's OK. My blockchain synced to 100% in 3 days having "monerod" service restarted every 30 mins. Looks like a workaround for an issue. |
|
Just a quick update The VM on the AMD machine is synced but still exhibits the strange behavior usually every few hours, but sometimes takes longer, 20-30 hours. Although I didn't pop any blocks to see if it could sync from farther back on the i7 machine. But I'm assuming the cause of the behavior while synced and unsynced are identical because the behavior is the same. I recently ran prime95 (random FFT) for a while on the AMD machine, so it's definitely stable. It never runs hotter than 30C. I ran the RandomX tests and everything does pass fine. It also runs other VMs and various software with no strange behavior. So I'm not really sure what's going on. I think I have an old Athlon II X4 around here somewhere :D It's practically the same CPU on a slightly older socket and different chipset. |
tryphe
changed the title
tryphe
changed the title
|
I updated the OP so it should be a bit more clear what the behavior is now. |
|
Can you run RandomX tests in a loop for 60 hours or so and see if it passes 100% of the time? That would be the most similar test to running monerod for a long time. |
|
I just completed a sync on a laptop made in 2007 with a Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz with 2GB RAM and a 7200 rpm spinny HDD |
|
I ran into this issue too. I spent a few hours trying to debug it. Apparently, it crashes when trying to get the length of some string. I haven't found which one exactly it is, as when I try to debug it, but the strlen cal is way too often called for that to be feasible. Also, I tried to compile from the master branch, but that failed with a bunch of C++ errors in multiple libraries (cross-compiling for windows 64-bit from ubuntu). But back to the topic. The error shows up as an "FILE SYSTEM LIMITATION" in procmon. |
|
@agowa338 when did this issue start showing up? could it be a corrupted database? |
|
I dought that it's a corrupt database. After all it started syncing from noting yesterday. |
|
And it instantly crashes after startup? |
|
Not instantly, but immediately after the "SYNCHRONIZATION started" message. I tried to run it with increased loglevel, but even with 4 I couldn't see anything related to this (or I missed it, it was a lot of log output). |
|
I assume starting with If it doesn't help I would move / delete the blockchain (data.lmdb) and try to sync from scratch again. The blockchain can corrupt when there is a power loss or when the external harddrive gets disconnected during sync. You can also add |
That wasn't the case. The computer was still running. All that happened was while the initial sync it already crashed at that position. And now every time I restart it crashes there again. And the harddisk is an NVME where also the OS is installed on, so it is probably also not that (no disk errors within the eventlog either). I can delete it and restart the sync, but as it was basically the initial sync, I have my doubts that it'll work the 2nd time...
|
|
What you posted shows that there is an error when adding something to the database. Now I'm not really sure what "FILE SYSTEM LIMITATION" means exactly, but if it starts showing up a second time we can be sure that it's not a corrupted database. @hyc any idea about ^ ? |
|
To clarify, with what you described (internal NVMe, no power loss) it's unlikely that your database is corrupted but it's the easiest first step to help us locate the issue. |
|
"FILE SYSTEM LIMITATION" is the result of the API call to SetEndOfFile https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setendoffile which sets the physical size of a file (There is still about 156 GB free on that 1TB drive, so it's not an out of space issue either).
Ok, going to delete the database now. But it'll take about a day to sync. So that's why I didn't necessarily want to do it now while debugging. |
|
And do you know how to get an all threads backtrace on Windows? I'm unfortunately only familiar with gdb / lldb. |
|
Only in visual studio. Haven't done it in the x64dbg or windbg yet. |
|
Here you go, the horizontal lines separate threads: |
|
This isn't readable unfortunately, seems to be either a stripped binary or some Windows specific thing. |
|
Those are the exact positions where the instruction pointer was currently pointing to. Sadly it doesn't show the function name. For that I have to click on each individual one and look a few lines up in the asembly... |
|
Apparently, that was my fault. I didn't download the symbols within x64dbg. This already looks a bit better (but I don't have a PDB file for monerod (it's not within the downloadable zip), so it cannot resolve these) |
No idea how strlen() has anything to do with filesystem limitations. What is the size of the data.mdb file, and what kind of filesystem is in use on that partition? |
|
Filesystem NTFS the strlen() is called once I hit "step over", so it's probably the exception handler within monerod I suppose (hard to tell without the PDB file). |
|
And when looking at the asm code of strlen we can see why it crashes. Somehow it ends up with a null pointer in RAX and tries to read from it. And reading from a null pointer aparently throws an access violation exception... |
|
I deleted the database and let it resync again (this time with |
|
Closing as there were no other reports about this in a while and I suspect the issue was hardware related. |
and others
Describe the problem
Behavior: At some point after starting monerod, ranging from a few minutes to many hours, syncing "gets stuck". Whether synced or unsynced, blocks stop being added to the chain until monerod is restarted. The monerod process continues running and is responsive, however, and exits normally if given HUP etc.
The behavior repeatedly appears for me, in particular, starting around block >~2450000. Not any specific block, just random, but not at any point before ~2450000. If monero is already synced, it takes longer for the behavior to appear, sometimes several hours longer.
There are plenty of connections and new connections being made during the behavior.
Monerod gives me two messages like this every few minutes, gives the correct number of total blocks, but is still stuck at the same block where the behavior appeared:
Workaround: If I create a cron script to restart monerod every ~15 minutes, it eventually syncs.
A few things about my node and what I've tried:
I've been running monero for over 5 years with no syncing issues, until about a month ago.
Minimal setup Debian 10 VM with ntpd and monerod (Update: tried with a new Debian 11 machine and the same behavior occurs).
Syncing works fine until about 95%, or near block 2450000.
Removed the entire .bitmonero directory and synced from scratch several times with the same result.
Removed my config and am running with no arguments to reduce complexity, but get the same result.
Running on clearnet.
Using the blocklist at https://gui.xmr.pm/files/block.txt, but I tried without the blocklist and get the same result.
Using outbound connections only to try and sync currently to avoid possibility of attack.
Tried running with things like
--out-peers 25or--out-peers 50but get the same result.Running the latest master branch 9aab19f but also tried with signed releases:
which experience the same behavior but do NOT show errors in the log.
Update: I originally thought the syncing problem was related to these errors, but this is not the case. Will leave this section here
Some errors/warnings that frequently occur:
There's also a loop that produces hundreds of these at once for the same peer:
Here's a short run:
https://gist.githubusercontent.com/tryphe/0a09526e0c9aa1e6154b4e2b0c6e2e13/raw/a56cca8895370d6e91a27b5f4a4b887d4e064d2d/gistfile1.txt
(Note: I gave the hangup signal near 03:16:52.049 where it shuts down cleanly)
Any feedback is welcome and appreciated. Let me know if you need more information or if I can do anything to make debugging easier. Thank you.
The text was updated successfully, but these errors were encountered: