Snapshot generation takes too long

[PanosChtz]

System information

Geth version: 10.0.4
OS & Version: Linux (Ubuntu 20.04.2 LTS) on Raspberry Pi 8GB

Expected behaviour

Complete pruning in a reasonable time

Actual behaviour

Pruning still runs after one week

Steps to reproduce the behaviour

I was running geth 1.10.2 on my RPi4 with the parameters --txlookuplimit=0 --snapshot=false to minimize cpu load. However since my 2TB SSD is now about 79% full for some reason, I decided to make a snapshot before pruning. So I updated to geth 1.10.4 and took these parameters out, which enabled snapshot generation by default. However the snapshot generation for some reason seems to take for ever, it's been running for over a week now. Is this normal behavior? Anyone experienced this on a RPi4? I'm afraid that if it takes like a month or so, I will be forced to delete all chaindata and resync from scratch since the disk will eventually become full. Geth log below.

Backtrace

Jun 20 22:25:47 validator geth[141868]: INFO [06-20|22:25:47.392]  Aborting state snapshot generation       root=fc3c41..3de29d  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=30h30m53.595s eta=954845h59m47.71s 
Jun 20 22:25:47 validator geth[141868]: INFO [06-20|22:25:47.393]  Resuming state snapshot generation       root=de9fba..9e6921  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=30h30m53.597s eta=954845h59m47.71s 
Jun 20 22:25:47 validator geth[141868]: INFO [06-20|22:25:47.488]  Imported new chain segment               blocks=1          txs=221          mgas=14.985 elapsed=55.112s       mgasps=0.272 number=12,667,033  hash=f400dc..2934de age=1d1h42m   dirty=63.90MiB

after a few days, ETA became a negative number which is very weird. Also noticed the "at=", "accounts=", "slots=" and "storage=" fields remain the same all the time (not sure if this is normal).

Jun 23 19:46:06 validator geth[141868]: INFO [06-23|19:46:06.198]  Aborting state snapshot generation       root=d68b93..5017c3  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=99h51m12.401s eta=-1999            565h54m27.431s 
Jun 23 19:46:06 validator geth[141868]: INFO [06-23|19:46:06.199]  Resuming state snapshot generation       root=b8e029..57229d  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=99h51m12.402s eta=-1999            565h54m27.431s 
Jun 23 19:46:06 validator geth[141868]: INFO [06-23|19:46:06.299]  Imported new chain segment               blocks=1          txs=205          mgas=14.947 elapsed=57.745s       mgasps=0.259 number=12,671,447  hash=f5ebab..            6ea02c age=3d6h23m   dirty=63.90MiB

next day:

Jun 24 14:14:47 validator geth[141868]: INFO [06-24|14:14:47.175]  Aborting state snapshot generation       root=19b56a..a9808e  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=118h19m53.378s eta=-1421367h10m46.674s 
Jun 24 14:14:47 validator geth[141868]: INFO [06-24|14:14:47.175]  Resuming state snapshot generation       root=23e508..51cee6  at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB  elapsed=118h19m53.379s eta=-1421367h10m46.674s 
Jun 24 14:14:47 validator geth[141868]: INFO [06-24|14:14:47.297]  Imported new chain segment               blocks=1          txs=196          mgas=14.961 elapsed=54.891s        mgasps=0.273 number=12,672,616  hash=49a53b..aef2bd age=3d20h27m  dirty=63.90MiB

today ETA is positive again, yet no end in sight:

Jun 26 17:00:05 validator geth[141868]: INFO [06-26|17:00:05.833] Deep froze chain segment                 blocks=2          elapsed=354.340ms      number=12,585,938 hash=263676..32248c
Jun 26 17:00:57 validator geth[141868]: INFO [06-26|17:00:57.165] Aborting state snapshot generation       root=f341d7..05697d at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=169h6m3.369s   eta=167265h29m48.892s
Jun 26 17:00:57 validator geth[141868]: INFO [06-26|17:00:57.166] Resuming state snapshot generation       root=3b05a5..dee933 at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=169h6m3.369s   eta=167265h29m48.892s
Jun 26 17:00:57 validator geth[141868]: INFO [06-26|17:00:57.293] Imported new chain segment               blocks=1          txs=156         mgas=14.931 elapsed=55.737s        mgasps=0.268 number=12,675,939 hash=ac825c..93c444 age=5d10h53m  dirty=63.90MiB

[karalabe]

Something seems off there. Then again, your block imports are excruciatingly slow too. 55s for a single blocks is quite bad. Snapshot generation can only be done while you are not importing blocks, so as long as you're out of sync, it won't be able to progress.

How much RAM do you have, how large cache allowance are you running with? The less RAM you allot to Geth, to more it needs to push stuff to disk, which causes it's database to grow faster.

At this point it might be worthwhile to resync. You can keep the ancients folder within your chaindata so you won't need to download the entire chain all over, and you can just snap sync on top (default in 1.10.4). That requires a download of about 30GB and it should be a lot faster to get the snapshot live too afterwards, since snapsync downloads those (just needs to fix them).

[PanosChtz]

I ran geth with 256 cache, I changed it to 2048 and restarted it. The log now looks like this:

Jun 28 16:50:43 validator geth[248458]: INFO [06-28|16:50:43.947] Aborting state snapshot generation       root=4acad3..2fe4be at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h34m56.347s
Jun 28 16:50:43 validator geth[248458]: INFO [06-28|16:50:43.949] Resuming state snapshot generation       root=b89a68..fb09ad at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h34m56.349s
Jun 28 16:50:43 validator geth[248458]: INFO [06-28|16:50:43.961] Imported new chain segment               blocks=1 txs=232 mgas=14.935 elapsed=54.263s      mgasps=0.275 number=12,678,983 hash=6ce2d3..7bc95d age=6d23h21m dirty=194.65MiB
Jun 28 16:51:19 validator geth[248458]: INFO [06-28|16:51:19.429] Deep froze chain segment                 blocks=2 elapsed=117.585ms    number=12,588,983 hash=03b747..8ef5e5
Jun 28 16:51:37 validator geth[248458]: INFO [06-28|16:51:37.996] Aborting state snapshot generation       root=b89a68..fb09ad at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h35m50.396s
Jun 28 16:51:38 validator geth[248458]: INFO [06-28|16:51:37.997] Resuming state snapshot generation       root=e8ed6b..809158 at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h35m50.396s
Jun 28 16:51:38 validator geth[248458]: INFO [06-28|16:51:38.004] Imported new chain segment               blocks=1 txs=173 mgas=14.943 elapsed=54.043s      mgasps=0.276 number=12,678,984 hash=02b83c..82579b age=6d23h22m dirty=196.76MiB
Jun 28 16:52:31 validator geth[248458]: INFO [06-28|16:52:31.857] Aborting state snapshot generation       root=e8ed6b..809158 at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h36m44.257s
Jun 28 16:52:31 validator geth[248458]: INFO [06-28|16:52:31.858] Resuming state snapshot generation       root=e64254..71ca57 at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=1h36m44.258s
Jun 28 16:52:31 validator geth[248458]: INFO [06-28|16:52:31.871] Imported new chain segment               blocks=1 txs=183 mgas=14.951 elapsed=53.866s      mgasps=0.278 number=12,678,985 hash=f6874e..06263d age=6d23h23m dirty=198.71MiB
Jun 28 16:53:19 validator geth[248458]: INFO [06-28|16:53:19.554] Deep froze chain segment                 blocks=2 elapsed=123.729ms    number=12,588,985 hash=6a68b0..e90b81

If I end up resyncing with keeping ancients folder as you suggested, this is done through geth removedb?

[i-norden]

@PanosChtz if you do geth removedb the first y/n prompt will be for removing the full node state db and the second for removing the ancient db (and third for light node db).

[PanosChtz]

Ok thank you, I will leave it running for a few more days before giving up and will update this thread accordingly.

[PanosChtz]

I also noticed that there is no ETA any more in the log. Again, not sure if this is normal or a bug.

Jun 30 01:57:45 validator geth[248458]: INFO [06-30|01:57:45.651] Aborting state snapshot generation       root=e8a3dc..6729ce at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=34h41m58.051s
Jun 30 01:57:45 validator geth[248458]: INFO [06-30|01:57:45.665] Resuming state snapshot generation       root=27b663..6db5e1 at=000218..fce2a8 accounts=4480 slots=4963 storage=559.52KiB elapsed=34h41m58.065s
Jun 30 01:57:45 validator geth[248458]: INFO [06-30|01:57:45.776] Imported new chain segment               blocks=1    txs=133     mgas=14.994 elapsed=1m4.198s      mgasps=0.234 number=12,681,016 hash=1bc277..a22888 age=1w1d42m  dirty=366.64MiB

[PanosChtz]

So after letting it run for more than 5 days for a second time, I decided to resync (did not delete ancient db).
The resyncing was quite bumpy though, my Raspberry Pi froze 2 times during the resync and had to power cycle it. The log looked something like this:

Jul 05 01:03:34 validator geth[1870]: INFO [07-05|01:03:34.391] State sync in progress                   synced=79.26% state=95.86GiB accounts=119,133,147@25.59GiB slots=341,410,381@68.54GiB codes=338,316@1.72GiB    eta=3h10>

when the synced finally reached 100%, it started a "state heal":

Jul 05 13:32:46 validator geth[1869]: INFO [07-05|13:32:46.444] State sync in progress                   synced=99.97% state=125.72GiB accounts=148,925,190@32.29GiB slots=456,445,106@91.31GiB codes=412,073@2.12GiB eta=-1m2.9>
Jul 05 13:32:55 validator geth[1869]: INFO [07-05|13:32:55.369] State sync in progress                   synced=99.99% state=125.74GiB accounts=148,957,742@32.29GiB slots=456,480,918@91.32GiB codes=412,131@2.12GiB eta=-1m12.>
Jul 05 13:32:58 validator geth[1869]: INFO [07-05|13:32:58.520] State sync in progress                   synced=100.00% state=125.74GiB accounts=148,968,965@32.30GiB slots=456,486,075@91.32GiB codes=412,169@2.12GiB eta=-1m15>
Jul 05 13:33:08 validator geth[1869]: INFO [07-05|13:33:08.083] State heal in progress                   accounts=0@0.00B              slots=0@0.00B              codes=0@0.00B         nodes=303@157.42KiB pending=4847
Jul 05 13:33:09 validator geth[1869]: INFO [07-05|13:33:09.261] Imported new block headers               count=1    elapsed=16.243ms   number=12,767,774 hash=78e75c..e9a555 age=1m10s

which still goes on. Is this normal?

Jul 05 16:47:27 validator geth[1869]: INFO [07-05|16:47:27.247] State heal in progress                   accounts=224,693@10.96MiB     slots=423,973@32.24MiB     codes=243@2.12MiB     nodes=2,058,288@500.27MiB pending=149,222
Jul 05 16:47:32 validator geth[1869]: WARN [07-05|16:47:32.694] Unexpected bytecode heal packet          peer=190f8332 reqid=4,258,055,458,344,850,575
Jul 05 16:47:33 validator geth[1869]: WARN [07-05|16:47:33.064] Served eth_chainId                       conn=127.0.0.1:46282 reqid=1                         t="65.184µs"  err="chain not synced beyond EIP-155 replay-protection fork block"
Jul 05 16:47:34 validator geth[1869]: WARN [07-05|16:47:34.889] Served eth_coinbase                      reqid=3                         t="254.869µs" err="etherbase must be explicitly specified"
Jul 05 16:47:34 validator geth[1869]: INFO [07-05|16:47:34.976] Imported new block headers               count=1    elapsed=16.799ms   number=12,768,633 hash=579574..6667ad
Jul 05 16:47:36 validator geth[1869]: INFO [07-05|16:47:36.199] State heal in progress                   accounts=225,043@10.98MiB     slots=424,187@32.25MiB     codes=244@2.12MiB     nodes=2,060,164@500.74MiB pending=149,506
Jul 05 16:47:45 validator geth[1869]: INFO [07-05|16:47:45.448] State heal in progress                   accounts=225,443@11.00MiB     slots=424,464@32.28MiB     codes=244@2.12MiB     nodes=2,062,275@501.24MiB pending=149,275
Jul 05 16:47:48 validator geth[1869]: WARN [07-05|16:47:48.708] Unexpected bytecode heal packet          peer=48ea4315 reqid=8,714,096,766,713,270,556
Jul 05 16:47:53 validator geth[1869]: INFO [07-05|16:47:53.805] State heal in progress                   accounts=225,747@11.01MiB     slots=424,833@32.30MiB     codes=244@2.12MiB     nodes=2,064,862@501.73MiB pending=150,159
Jul 05 16:48:01 validator geth[1869]: WARN [07-05|16:48:01.790] Unexpected bytecode heal packet          peer=4329141d reqid=3,032,379,909,792,565,840
Jul 05 16:48:02 validator geth[1869]: INFO [07-05|16:48:02.247] State heal in progress                   accounts=226,158@11.03MiB     slots=425,087@32.32MiB     codes=245@2.12MiB     nodes=2,066,728@502.14MiB pending=150,153
Jul 05 16:48:03 validator geth[1869]: WARN [07-05|16:48:03.738] Unexpected bytecode heal packet          peer=61551118 reqid=9,127,912,323,441,804,620
Jul 05 16:48:06 validator geth[1869]: WARN [07-05|16:48:06.561] Unexpected bytecode heal packet          peer=40258617 reqid=3,718,725,767,528,652,524
Jul 05 16:48:08 validator geth[1869]: WARN [07-05|16:48:08.318] Unexpected bytecode heal packet          peer=190f8332 reqid=2,572,868,575,016,331,520
Jul 05 16:48:10 validator geth[1869]: INFO [07-05|16:48:10.326] State heal in progress                   accounts=226,754@11.06MiB     slots=425,917@32.38MiB     codes=247@2.15MiB     nodes=2,069,897@502.64MiB pending=148,904
Jul 05 16:48:18 validator geth[1869]: INFO [07-05|16:48:18.509] State heal in progress                   accounts=227,022@11.07MiB     slots=426,502@32.42MiB     codes=248@2.15MiB     nodes=2,072,248@503.12MiB pending=149,714
Jul 05 16:48:26 validator geth[1869]: INFO [07-05|16:48:26.557] State heal in progress                   accounts=227,473@11.09MiB     slots=427,286@32.48MiB     codes=249@2.16MiB     nodes=2,075,098@503.59MiB pending=149,055

[PanosChtz]

Looks like state heal finished, but the snapshot generation thing started again. Hope this won't take forever this time.

Jul 06 02:41:58 validator geth[1869]: INFO [07-06|02:41:58.197] Resuming state snapshot generation       root=e7b3ac..734f9b at=0a8a7c..f2b2ed accounts=5,781,583            slots=20,966,364           storage=1.86GiB    elapsed=1h9m5.323s  eta=26h48m48.914s
Jul 06 02:41:58 validator geth[1869]: INFO [07-06|02:41:58.233] Imported new chain segment               blocks=1       txs=154        mgas=14.998 elapsed=835.939ms   mgasps=17.941 number=12,771,284 hash=067346..08c92d dirty=385.43MiB
Jul 06 02:41:58 validator geth[1869]: INFO [07-06|02:41:58.271] Unindexed transactions                   blocks=1       txs=160        tail=10,421,285 elapsed=23.605ms
Jul 06 02:42:00 validator geth[1869]: INFO [07-06|02:42:00.090] Aborting state snapshot generation       root=e7b3ac..734f9b in=0a8c60..5b9cdc at=290dec..f3e563 accounts=5,785,629            slots=20,967,299           storage=1.86GiB    elapsed=1h9m7.217s  eta=26h48m22.472s
Jul 06 02:42:00 validator geth[1869]: INFO [07-06|02:42:00.093] Resuming state snapshot generation       root=3fefa8..831579 in=0a8c60..5b9cdc at=290dec..f3e563 accounts=5,785,629            slots=20,967,299           storage=1.86GiB    elapsed=1h9m7.220s  eta=26h48m22.542s
Jul 06 02:42:00 validator geth[1869]: INFO [07-06|02:42:00.163] Imported new chain segment               blocks=1       txs=189        mgas=14.978 elapsed=1.915s      mgasps=7.821  number=12,771,285 hash=47492a..a601d1 dirty=386.03MiB
Jul 06 02:42:00 validator geth[1869]: INFO [07-06|02:42:00.168] Unindexed transactions                   blocks=1       txs=80         tail=10,421,286 elapsed=3.995ms
Jul 06 02:42:01 validator geth[1869]: INFO [07-06|02:42:01.166] Downloader queue stats                   receiptTasks=0   blockTasks=0   itemSize=56.79KiB  throttle=1155
Jul 06 02:42:08 validator geth[1869]: INFO [07-06|02:42:08.096] Generating state snapshot                root=3fefa8..831579 in=0a9270..56b39d at=259e8a..926542 accounts=5,798,695            slots=20,999,760           storage=1.86GiB    elapsed=1h9m15.222s eta=26h47m42.792s
Jul 06 02:42:10 validator geth[1869]: INFO [07-06|02:42:10.383] Aborting state snapshot generation       root=3fefa8..831579 in=0a9270..56b39d at=6bf037..21c485 accounts=5,798,695            slots=21,037,188           storage=1.86GiB    elapsed=1h9m17.510s eta=26h48m35.907s
Jul 06 02:42:10 validator geth[1869]: INFO [07-06|02:42:10.393] Resuming state snapshot generation       root=982edb..0bbe0f in=0a9270..56b39d at=6bf037..21c485 accounts=5,798,695            slots=21,037,188           storage=1.86GiB    elapsed=1h9m17.520s eta=26h48m36.14s
Jul 06 02:42:10 validator geth[1869]: INFO [07-06|02:42:10.435] Imported new chain segment               blocks=1       txs=163        mgas=14.963 elapsed=1.844s      mgasps=8.114  number=12,771,286 hash=3af14f..5ff515 dirty=386.25MiB
Jul 06 02:42:10 validator geth[1869]: INFO [07-06|02:42:10.447] Unindexed transactions                   blocks=1       txs=211        tail=10,421,287 elapsed=10.941ms
Jul 06 02:42:16 validator geth[1869]: INFO [07-06|02:42:16.382] Aborting state snapshot generation       root=982edb..0bbe0f at=0a9596..56c37d accounts=5,805,327            slots=21,119,629           storage=1.87GiB    elapsed=1h9m23.509s eta=26h48m58.011s
Jul 06 02:42:16 validator geth[1869]: INFO [07-06|02:42:16.383] Resuming state snapshot generation       root=9a125c..dab6e7 at=0a9596..56c37d accounts=5,805,327            slots=21,119,629           storage=1.87GiB    elapsed=1h9m23.510s eta=26h48m58.034s
Jul 06 02:42:16 validator geth[1869]: INFO [07-06|02:42:16.428] Imported new chain segment               blocks=1       txs=121        mgas=14.956 elapsed=1.455s      mgasps=10.272 number=12,771,287 hash=4efaaf..6697eb dirty=386.05MiB
Jul 06 02:42:16 validator geth[1869]: INFO [07-06|02:42:16.440] Unindexed transactions                   blocks=1       txs=129        tail=10,421,288 elapsed=9.328ms
Jul 06 02:42:20 validator geth[1869]: INFO [07-06|02:42:20.285] Deep froze chain segment                 blocks=7       elapsed=133.476ms   number=12,681,287 hash=2d7f7c..ff120a
Jul 06 02:42:24 validator geth[1869]: INFO [07-06|02:42:24.453] Generating state snapshot                root=9a125c..dab6e7 at=0a9e27..f5cab0 accounts=5,823,749            slots=21,139,223           storage=1.87GiB    elapsed=1h9m31.580s eta=26h46m47.202s
Jul 06 02:42:32 validator geth[1869]: INFO [07-06|02:42:32.464] Generating state snapshot                root=9a125c..dab6e7 in=0aa365..2be556 at=f65289..0c4669 accounts=5,835,070            slots=21,161,599           storage=1.87GiB    elapsed=1h9m39.591s eta=26h46m38.357s

[PanosChtz]

So eventually today my long journey in the snapshot world ended:

Jul 07 02:03:13 validator geth[1869]: INFO [07-07|02:03:13.540] Unindexed transactions                   blocks=1       txs=268        tail=10,427,535 elapsed=12.016ms
Jul 07 02:03:21 validator geth[1869]: INFO [07-07|02:03:21.542] Generating state snapshot                root=b557a5..461144 in=ffeb92..4e53dc at=56cd68..145605 accounts=140,417,975          slots=465,945,433          storage=39.88GiB   elapsed=24h30m28.669s eta=27.507s
Jul 07 02:03:29 validator geth[1869]: INFO [07-07|02:03:29.582] Generating state snapshot                root=b557a5..461144 in=ffeb92..4e53dc at=74ee37..736077 accounts=140,417,975          slots=465,971,169          storage=39.88GiB   elapsed=24h30m36.709s eta=27.509s
Jul 07 02:03:31 validator geth[1869]: INFO [07-07|02:03:31.981] Deep froze chain segment                 blocks=6       elapsed=139.177ms     number=12,687,534 hash=9639c3..f8d675
Jul 07 02:03:37 validator geth[1869]: INFO [07-07|02:03:37.651] Generating state snapshot                root=b557a5..461144 in=ffeb92..4e53dc at=e5141c..8ef25f accounts=140,417,975          slots=466,065,871          storage=39.89GiB   elapsed=24h30m44.778s eta=27.512s
Jul 07 02:03:45 validator geth[1869]: INFO [07-07|02:03:45.739] Generating state snapshot                root=b557a5..461144 in=ffecdf..d83e4b at=bafd0d..b243d9 accounts=140,420,771          slots=466,170,763          storage=39.89GiB   elapsed=24h30m52.866s eta=25.766s
Jul 07 02:03:52 validator geth[1869]: INFO [07-07|02:03:52.798] Aborting state snapshot generation       root=b557a5..461144 in=ffee25..b72fbb at=881e6d..cfc831 accounts=140,423,515          slots=466,215,087          storage=39.90GiB   elapsed=24h30m59.925s eta=24.05s
Jul 07 02:03:52 validator geth[1869]: INFO [07-07|02:03:52.854] Resuming state snapshot generation       root=9acc8c..21242f in=ffee25..b72fbb at=881e6d..cfc831 accounts=140,423,515          slots=466,215,087          storage=39.90GiB   elapsed=24h30m59.981s eta=24.05s
Jul 07 02:03:52 validator geth[1869]: INFO [07-07|02:03:52.948] Imported new chain segment               blocks=1       txs=253        mgas=14.970 elapsed=2.216s        mgasps=6.753  number=12,777,535 hash=e54270..f9c290 dirty=510.24MiB
Jul 07 02:03:52 validator geth[1869]: INFO [07-07|02:03:52.967] Unindexed transactions                   blocks=1       txs=192        tail=10,427,536 elapsed=10.173ms
Jul 07 02:04:00 validator geth[1869]: INFO [07-07|02:04:00.854] Generating state snapshot                root=9acc8c..21242f in=fff811..2104d7 at=290dec..f3e563 accounts=140,444,633          slots=466,243,946          storage=39.90GiB   elapsed=24h31m7.981s  eta=10.686s
Jul 07 02:04:05 validator geth[1869]: INFO [07-07|02:04:05.738] Generated state snapshot                 accounts=140,461,993          slots=466,257,725          storage=39.90GiB   elapsed=24h31m12.865s
Jul 07 02:04:19 validator geth[1869]: INFO [07-07|02:04:19.335] Downloader queue stats                   receiptTasks=0   blockTasks=0   itemSize=56.84KiB  throttle=1153
Jul 07 02:04:20 validator geth[1869]: INFO [07-07|02:04:20.506] Imported new chain segment               blocks=1       txs=195        mgas=14.997 elapsed=2.372s        mgasps=6.322  number=12,777,536 hash=c16d08..984a20 dirty=510.30MiB
Jul 07 02:04:20 validator geth[1869]: INFO [07-07|02:04:20.540] Unindexed transactions                   blocks=1       txs=99         tail=10,427,537 elapsed=22.995ms

So not sure what was the cause of the initial problem that somehow caused the snapshot generation process to stall. Maybe if I realized that it had stalled sooner I would have just done the resync and save 2 weeks of waiting. I'm closing the issue but I suggest the developers to look into this ETA issue (have seen crazy ETAs in other online sources where people suggested to wait it out), maybe if the log is more clear (like percentage completed in snapshot generation?)

[mohamedmansour]

I am in the same boat as you are, I have been trying for ~1 week now getting mainnet to sync using snap sync under different cloud hardware. That ETA is really troublesome, and I wish it was more strict saying "Hardware not supported, exiting.." so we know early on that this will never complete, due to some heuristics.