Skip to content
This repository has been archived by the owner on Jun 12, 2023. It is now read-only.

Rak - No inbound/outbound/block-height 1, after 12.29.2 update #1325

Open
smeakles79 opened this issue Jan 2, 2022 · 54 comments
Open

Rak - No inbound/outbound/block-height 1, after 12.29.2 update #1325

smeakles79 opened this issue Jan 2, 2022 · 54 comments

Comments

@smeakles79
Copy link

Have reports of several Rak hotspots that, after the 2021.12.29.2, are connecting to wifi/ethernet and obtaining an IP and connecting to the local router, no longer have an inbound/outbound connection. Block height on these Raks are either unknown or at 1.

Some unasserted, but staying synced hotspots are.
Narrow Umber Stork
Exotic Satin Anteater
Alert Lead Tapir
Restless Pewter Anteater
Hidden Brick Wasp

Asserted Units
Muscular Turquoise Donkey

@Ramsin
Copy link

Ramsin commented Jan 2, 2022

Add Jumpy Berry Sheep to the list... Had it fully synced a few days ago, unplugged it... Turned it on a few days later to fully sync it before taking it to the location it was going to be installed at and I get no in/out, 0% synced. Running 2021.12.29.2

(I have a fully working hotspot at the same location so internet is good)

@smeakles79
Copy link
Author

smeakles79 commented Jan 2, 2022

Powerful Crepe Cat is another members hotspot. Same issue. 2021.12.29.2 firmware.

@MrBlue995
Copy link

Gorgeous Banana Shrimp

@Swoesh33
Copy link

Swoesh33 commented Jan 3, 2022

I am the owner of Powerful Crepe Cat,
I received the hotspot on 31dec and set it up (without setting location) and it started syncing without a problem the 1st I unplugged it and drove to my location and put the location in the app. from that point it has not done anything ....
even back at home where it worked before.
now I have no inbound/outbound so it's just doing nothing.
Hope this will get fixed so I can drive again for 3 hours but now it will work.

@hcl-83
Copy link

hcl-83 commented Jan 3, 2022

I have 3rak the same issues:
Quaint Champagne Bee
Plain Sable Cod
Clumsy Rusty Monkey

@Peanudt
Copy link

Peanudt commented Jan 3, 2022

Same issue: Cool Taffy Alpaca

@pajche
Copy link

pajche commented Jan 3, 2022

Tart Tiger Mammoth, worked fine for more then month.
Now since last update no activity.

@m1xmstr
Copy link

m1xmstr commented Jan 3, 2022

Can someone who is having this issue try changing their IP address and see if their hotspot starts working.

If you have a separate router and modem:
First check what your IP is by Googling "What's my IP"

-Power off both router and modem.
-Power on router
-Change mac address (usually under clone mac just change last octet) and save/apply
-Power on Modem
-Verify IP is different ~ Google "What's my IP"
-Restart RAK Hotspot

See if this resolves your issue.

@smeakles79
Copy link
Author

I’ve got a Static IP from my ISP, so I can’t make a change. Powerful Crepe Cat is at a business, so can’t make a change there.

@hcl-83
Copy link

hcl-83 commented Jan 3, 2022

I try on fiber static ip and gsm lte connection no change no connection to p2p network

@stuck2007
Copy link

Hi m1xmstr, I just tried changing the public IP address on my miner but it made no difference. I can provide packet traces from the router if needed...

@Swoesh33
Copy link

Swoesh33 commented Jan 3, 2022

i plugged it in at 2 different places both don't work anymore, where the first one did work

@m1xmstr
Copy link

m1xmstr commented Jan 3, 2022

Hi m1xmstr, I just tried changing the public IP address on my miner but it made no difference. I can provide packet traces from the router if needed...

Okay thank you for the update. Someone mentioned this on a discord channel and thought it was worth a try. I'll keep digging and post if I find anything.

@gdanskgrom
Copy link

gdanskgrom commented Jan 3, 2022

Hi m1xmstr,
I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

@m1xmstr
Copy link

m1xmstr commented Jan 3, 2022

Hi m1xmstr, I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

Okay thank you for the update. It was worth a try.

@kibria88
Copy link

kibria88 commented Jan 3, 2022

Attractive Daisy Shetland
Inbound/Outbound Ok
Block height on rak either 0% or at 1
Firmware Update: 2021.12.29.2

@Swoesh33
Copy link

Swoesh33 commented Jan 3, 2022

Hi m1xmstr, I just tried this also on two of mine, without success. Main Opaque Guppy and Tiny Opaque Mule. Both are Gold Spots.

Blockchain height on both is 1. Completely different networks. SD cards replaced also.

mine is on height 0 and when I "send diagnostic" I see this

Connected to Blockchain:
Dialable:
Height: Unknown
NAT Type:
(most are just empty)

@Ramsin
Copy link

Ramsin commented Jan 3, 2022

It's 100% related to the firmware, luckily I had made backups of my SD Cards before even powering up the hotspots. When I restored my problematic one (Jumpy Berry Sheep), it had In/Outbound, and was at 84.x% synced. It updated the firmware automatically and went back to being problematic. (Original FW that came with mine was 2021.08.something.something)

@m1xmstr
Copy link

m1xmstr commented Jan 3, 2022

This feels a lot like the issue that occurred here back on early 2021. There wasn't a clear fix other than OTA

https://engineering.helium.com/2021/03/17/rak-firmware-issue-2.html

@pajche
Copy link

pajche commented Jan 3, 2022

Something is wrong with newest firmware I think. But my other RAK works fine.
I just came back from location. Diagnostics are okay. Tho before I changed my IP only Outbound was OK, after changing the IP both Inbound/Outbound are OK.
Blockchain is 100%, and with block number it was synced.
Disk is OK
And last challenge was 6 days ago, and that's when it stopped doing any activity.

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

@Ramsin
Copy link

Ramsin commented Jan 4, 2022

But my other RAK works fine.

Are you sure? My Goldspot drastically dropped in earnings once this firmware kicked in, but at least it’s working….my other GS is totally dead in the water…

@beervoley
Copy link

After the firmware upgrade my RAK MINTD stopped working too. Inbound/Outbound are shown as working but it's offline and sync is stuck at 99%~

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time.

I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

@pajche
Copy link

pajche commented Jan 4, 2022

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

I can confirm now, in my case changing IP fixed Inbound Status and its now showing activity, started working again. At least it works for now.

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time.

I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

Also this is a problem as well, I noticed very low activity on some other miners where I have log access and I could see a lot of failed to send witness errors, couldn't dial challenger, after I checked most of my peers were RAK devices and most of them offline, after rebooting hotspot and refreshing peer list, activity was much better and less errors.

@MrBlue995
Copy link

I don't think that changing IP solved the issue, still no activity after few hours, I will wait some longer, hopefully it will do some activity.

I can confirm now, in my case changing IP fixed Inbound Status and its now showing activity, started working again. At least it works for now.

I been trying to debug for about 4 days now, there is another thread about it. Something with the Erlang libp2p that doesnt let peers communicate , log files show a bunch of errors when talking to connected peers. I think the Helium Team is working on it, but its been days. I been searching the explorer to see if its just me. its not. everyone for about 14 days around me went down, there are a select few that I have noticed that are still doing good but the majority has gone down big time.
I have tried all sorts of diffrent firmware options from diffrent manufacturers. so if they say its specific to your manufacture. its specfic to the libp2p on the main helium repo

Also this is a problem as well, I noticed very low activity on some other miners where I have log access and I could see a lot of failed to send witness errors, couldn't dial challenger, after I checked most of my peers were RAK devices and most of them offline, after rebooting hotspot and refreshing peer list, activity was much better and less errors.

how did you exactly changed your ip? did you just connect the miner to another network? could it work?

@pajche
Copy link

pajche commented Jan 4, 2022

how did you exactly changed your ip? did you just connect the miner to another network? could it work?

I rebooted my internet modem, at that location I don't use static IP, so that was a solution. But yes, connecting to another network could also work. Or probably trying to connect it over VPN.

If you can try and check diagnostics and share results with us it would be great to know. I see for some people it doesn't work. But in my case it worked.

@Peanudt
Copy link

Peanudt commented Jan 4, 2022

Initially trying to setup a unit without location asserted. After assertion latest firmware did not work.
Flashed an older firmware version and then the hotspot was operating as normal. All connections green and block height increasing.
During the night, the Hotspot received over the air update to latest firmware and peer2peer connections and block height are grey again.
So for my situation it all points towards an issue with my hotspot + latest fw version:

  • issue not related to location assertion
  • sd card is not corrupted
  • with old fw connections are good (using dmz, because other unit in network is using port forwarding). One static IP from ISP.
  • Similar behavior wired and wifi. Hotspot gets IP and diagnostics accessible via BT.

@jonasjv
Copy link

jonasjv commented Jan 4, 2022

Rapid Vanilla Kookaburra
Same problem here
Inbound Outbound grey - O% blockchain sync

In my case i tried to sync before assert location

@smeakles79 smeakles79 changed the title Rak - No inbound/outbound after 12.29.2 update Rak - No inbound/outbound/block-height 1, after 12.29.2 update Jan 4, 2022
@gdanskgrom
Copy link

Initially trying to setup a unit without location asserted. After assertion latest firmware did not work. Flashed an older firmware version and then the hotspot was operating as normal. All connections green and block height increasing. During the night, the Hotspot received over the air update to latest firmware and peer2peer connections and block height are grey again. So for my situation it all points towards an issue with my hotspot + latest fw version:

  • issue not related to location assertion
  • sd card is not corrupted
  • with old fw connections are good (using dmz, because other unit in network is using port forwarding). One static IP from ISP.
  • Similar behavior wired and wifi. Hotspot gets IP and diagnostics accessible via BT.

Exactly the same situation for 2 of my units as well. Flash back to 12.14.1 and everything is operational until the OTA is consumed.

@jonasjv
Copy link

jonasjv commented Jan 4, 2022

Do you know where i can get fw 12.14.1?

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

well does that work?! rolling back...? Im going to test it now. The issue is we dont have controllo f our hardware. only the manufacturers have control of our hardware. Which isnt de centralized what so ever. You cant just plug in a cable and get things working how you want. Its like Xfinity or any of those types of companies that control your hardware remotely. I had an issue with my hardware that's the only reason I had to get into making my own firmware. But then you lose all support from your manufacturer. see Xfinity ATT all them monopolized companies, you rent there modems etc... but we own this hardware, And you cant even log into it to change anything. I don't mean to sound against it... but the manufacturers that make all this hardware are Chinese ... so ... if you know how China works then you know how this works.. Also 5G dont expect that anytime soon. The FAA alreaady is pushing back ATT from setting up 5G networks in big cities with the Airports pushing back on them here in the US. so dont expect that freedomfi modem anytime soon

@smeakles79
Copy link
Author

Do you know where i can get fw 12.14.1?

https://docs.helium.com/troubleshooting/replace-sd-card/

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

Do you know where i can get fw 12.14.1?

https://docs.helium.com/troubleshooting/replace-sd-card/

I got a 5 TB hard drive connected to mine... so... not an SD card issue

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

Also. The new PoC and the new HIP ... means we dont get rewarded for mining anymore... so we are being forced to continue mining and following the blockchain, for no reason. They should have rolled out gateway-rs at the same time they took away the rewards for mining. The manufacturers are still selling out of 500$ machines that are hoard bought by people who sell them for 1000$ + when you dont need that level of hardware. It was a bad momemnt for them to roll out that HIP, without the supporting software.

@smeakles79
Copy link
Author

What are you going off on? You are 100% incorrect that “we don‘ t get rewarded for mining anymore” please stop the FUD. Unless you have something to help, stop the FUD and let the Devs pick up the issue vs filling this specific item with garbage.

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

we dont only validators do. what do we get? .003 every 6 hours for crerating challenges? im sorry if I sound like im going off, im just frustrated at this.

@smeakles79
Copy link
Author

Validators are rewarded for being in Consensus group. That’s been off the miners since July. Validators do not do proof of coverage, hotspots do. Leave this post to the specific Rak issue. If you want to argue on the above, go to discord.

@Ramsin
Copy link

Ramsin commented Jan 4, 2022

We get that, but how does it contribute to this specific issue? You're just flooding this with nonsense at this point, and this issue will be ignored...as it apparently already is.

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

Your absolutely right. let me not mess your attempt at resolving this, I will not speak up to try and motivate any sort of call to action, to fix the current bugs and issues. IF anyone finds a fix, please do @ me to it.

@smeakles79
Copy link
Author

From Raks Discord Update on firmware 2021.12.29.2 issues and SD image availability

While firmware 2021.12.29.2 brought huge stability improvements, bringing more than 8000 hotspots back to life, we've been monitoring a small cluster of hotspots which stop showing inbound and outbound connections when updating to this release (but appeared fine in the 2021.12.24). Right now we have seen about 50 reports of these and it seems to affect particularly freshly flashed or brand new hotspots, especially those without location assert.

These findings have been communicated both by the community and ourselves to Helium but we're still waiting on a solution from them. We believe this should be coming soon via another update.

Until the core issue is identified and confirmed by Helium we are also not offering our SD card image to reflash miners as we're not sure what the problem is yet . We strongly recommend not re-flashing hotspot microSD cards right now until there is an all clear.

@Tylerlhess
Copy link

I have this issuse on my Bobcat 300. It synced at my home (without an asserted location) fine but when I moved it to its location it stopped working. I thought it might have been the business internet and moved it to another location. No dice.

Cheery Blush Zebra
no inbound outbound connections
Stopped working when I asserted it while on the 2021.12.29.2 firmware.

@serbyxp
Copy link

serbyxp commented Jan 4, 2022

From Raks Discord Update on firmware 2021.12.29.2 issues and SD image availability

While firmware 2021.12.29.2 brought huge stability improvements, bringing more than 8000 hotspots back to life, we've been monitoring a small cluster of hotspots which stop showing inbound and outbound connections when updating to this release (but appeared fine in the 2021.12.24). Right now we have seen about 50 reports of these and it seems to affect particularly freshly flashed or brand new hotspots, especially those without location assert.

These findings have been communicated both by the community and ourselves to Helium but we're still waiting on a solution from them. We believe this should be coming soon via another update.

Until the core issue is identified and confirmed by Helium we are also not offering our SD card image to reflash miners as we're not sure what the problem is yet . We strongly recommend not re-flashing hotspot microSD cards right now until there is an all clear.

You need to find someone that is fluent in Erlang code. The libp2p is throwing errors that have been posted in another thread I will post them here maybe it will shine some light on the issue.
(helium/erlang-libp2p#417 (comment))

@kovicak
Copy link

kovicak commented Jan 4, 2022

my MNTD has had 0 earnings since Dec 17. I've reset, many times and says 'syncing'. It's been syncing for the past 6 days. Please add:

Sticky Cerulean Nightingale

@herculanocosta
Copy link

Firmware 2021.12.29.2 update
The Helium core team has identified the issue with firmware 2021.12.29.2 where the miners may show issues connecting to the blockchain and showing no inbound or outbound.

This issue only affects some miners that have had a location asserted after the 2021.12.29.0 update, Hotspots asserted before the 2021.12.29.0 will not experience this issue (but may show similar symptoms due to other problems like network connection)

Helium is fast tracking version 2022.01.04.0 to deal with this issue.

Follow Rakstars discord for more updates. Thanks for your patience.

@Tylerlhess
Copy link

the new firmware image fixed my hotspot yesterday

@herculanocosta
Copy link

This issue has been cleared. This thread may be closed

@kovicak
Copy link

kovicak commented Jan 6, 2022 via email

@smeakles79
Copy link
Author

smeakles79 commented Jan 6, 2022

Could you point me to the instructions for updating the firmware?

-- Thanks, Tony Kovicak

If it’s a Rak, use this https://support.getmntd.com/hc/en-us/articles/4418137820951-I-need-to-replace-the-microSD-card-on-my-hotspot

@velasquez87
Copy link

I have same issue with my bobcat300

Please add
Basic onyx snake

@redleg105
Copy link

redleg105 commented Jan 29, 2022

Having the same problem.
Special Strawberry Platypus

@serbyxp
Copy link

serbyxp commented Jan 29, 2022

The issue persists . Even in the latest and in the alpha and betas. I have tried them all. The issue has been documented in a sense in the documentation about relayed challengers and hotspots. I do agree a roll back might work. But with the last beta that turned into the latest release I had started beaconing a lot more , and my witness list has been expanding ( I would have more witnesses if I put my hotspot back outside where it belongs ) but I am still trouble shooting this errors. And it’s easier to have the thing on my desk. The line errors and bad arg errors have disappeared since the 1/12 for me Atleast, which was the conflict I thought originally on arm vs amd images . But I get alot of not found errors , which are typically challengers outside of my country. Which is not always the case , some are in my country but if you look at those hotspots they get 0 witness from challanging hotspots near me. So I thought that is due to them being relayed. That’s not the case , they must have inbound firewalls blocking our outbound attempts. Which leads me to believe that their hotspots including hotspots and validators outside of my country or even in my country , ( I say outside my country because that’s where 90% of the failed to connect or not found errors occur including an error that is a “connection refused”) that means to me that either personal firewalls, ISP firewalls, and government firewalls, are at play here for this issue. I hope gRPC fixes that. But it may very well not. Since it’s an inbound issue to the challanger. And the validators are having this issue with me also, the gRPC is ment to send all witness receipts request to validators only , but they are having that issue also. The newer images , maybe it’s just luck or a coincidence are working a little bit better, but I think it has to do more with who your seed nodes or peer nodes are, like if I manually connect to nodes near me I get better results then just getting the random connections … I don’t know if that’s true or just a coincidence. But the issue definitely persists. I mentioned or suggested a pool or mechanism for creating challanges . That excludes or filters based off asserted location. Like US915 challanges US915 EU->EU CN->CN ..etc.. , as well as not allowing challanges to be created by relayed hotspots. But I think that’s the main issue now that line arguments have been eliminated out of the equation. I mean I had to manually look up all the peers that failed to connect and that’s what I have narrowed it down to… firewalls etc..

another thing I noticed even when my witness is successful it’s not successful on the first attempt, it’s only successful on the second attempt. Or the retry.

now if your miner or your hotspots manufacturer has a VPN or ToR network enabled by default, that may be the reason some hotspots don’t have the issue, because the vpn or ToR network can make the hop through those isp / government firewalls.

@serbyxp
Copy link

serbyxp commented Jan 30, 2022

I notice it took almost 3 hours today for upgrade to latest firmware just leave it on for a looooooong time it has to do a gateway_v2 ledger upgrade and if you keep turning it on and off it will never finish.

it would have been nice if someone mentioned this , I found out the hard way but I share with you so leave it on Atleast 3 hours and check back to it. When it finished it was fully synced…

@brutusar
Copy link

brutusar commented Feb 1, 2022

Can confirm the above.

After reflashing the new SD card with Firmware: 2022.01.04.0 it took about 5 hours to get to Firmware: 2022.01.24.0.

You will have no inbound, no outbound, 0 synch with blockchan, no block hight ect, As it would be dead.

You also wont be able to check the open port through telnet. The only positive news i got while waiting was the change of firmware version around the 5hour mark.

After approximatelly 10 additional hours it was online and 100%synced to the Blockchain.

I needed to restart my router to get the port 44158 going again.

@serbyxp
Copy link

serbyxp commented Feb 2, 2022

Heh yeah I can imagine on an SD card it would take a lot longer, I use a ssd so that makes sense. Atleast they put out the _GA release so most hotspots should update automatically now.

I was watching what it was doing , and it has to populate the entire ledger.db folder with all new files that pertain to gateway_v2 upgrade. Maybe if your lucky your manufacture might push you all the files and folder similar to the snapshot for blocks / snaps if that’s even possible would be a nice feature

Atleast once you have it upgraded to the gateway_v2 it didn’t have to do it again, when the _GA release came out , so it should just be a one time thing then you should be good moving forward.

as far as it fixing any failed to dial issues, it most def did not fix that but.. I have started beaconing Atleast 2-3x a day so that’s a positive.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests