Significant reward drop. Affecting mostly DIY miners. Seems peerbook related

[apppie123]

Since the Dec 29th update a growing group of users is reporting a significant reward drop of >80% from one day to the other.
Hotspots that normally witness beacons several times per hour. Now only get rewarded several times per day.

What these users have in common:

1. Mostly DIY packet forwarder users with HIP19 approved ECC key and OG helium miners with alpha key.
2. Problem is tied to the P2P address or complete wallet containing multiple hotspots.
3. Are relatively high HNT earners.
4. All the hotspots have stopped beaconing for days(mine stopped beaconing for 10 days and counting).

1 ->
This could be a logical consequence since DIY users can see the miner logs and learn what is happening.

2 ->
The issue is not tied to one brand. Even when reverting to original manufacturer image does not solve the issue. The local network is OK as tested with helium status and other networking tools. The issue is both with relayed and non-relayed hotspots. Mostly DIY setups but there are reports of original manufacturer hotspots as well.

The affected hotspots do see a high amount of witnesses throughout the day. This is confirmed using the miner logs.
It is not a classical case of p2p issue for the hotspot not being able to dial the challenger. The logs clearly show a successful submission of PoC witness to the challenger.

There is currently a lively discussion ongoing on #diy-packet-forwarder and #poc-discussion on the Helium Discord.
Unfortunately no one from the Helium team has chimed in.

Affected hotspots still do send out challenges. However the result is usually a low number(0-6 witnesses) indicating that challengees cannot reach the challenger.

"Add ability to control peer gossip for debugging purposes"
This new piece of code could be related.
When the gossip is disabled, rewards will slowly die down as the miner cannot be found in the network.
Are hotspots selectively grey listed? Like a HIP40 deny list?

helium/erlang-libp2p@853cada

This is how the reward drop looks like:

Miner log, most of these PoC submissions were not rewarded:

[ricopt5]

This is not new and there's already a github issue for this, but it has been closed: #1238

[apppie123]

This is not new and there's already a github issue for this, but it has been closed: #1238

This is not just a small fraction. This a complete drop in rewards as if these miners were shut off the network for a seemingly specific group users. When this was the same issue, you would expect that the drop in rewards wouldn't last for over 10 days now with consistent 0 rewards.

It is not the case of "failed to send witness, max retry".
These hotspots send PoC witness receipts with "Successfully sent witness to challenger", so they can actually dial the challenger but from there the PoC witnesses disappear. See the picture in OP of one of the logs.

[ricopt5]

This is not new and there's already a github issue for this, but it has been closed: #1238

This is not just a small fraction. This a complete drop in rewards as if these miners were shut off the network for a seemingly specific group users. When this was the same issue, you would expect that the drop in rewards wouldn't last for over 10 days now with consistent 0 rewards.

It is not the case of "failed to send witness, max retry". These hotspots send PoC witness receipts with "Successfully sent witness to challenger", so they can actually dial the challenger but from there the PoC witnesses disappear. See the picture in OP of one of the logs.

Don't give too much attention to the title, check the content. It's the same issue where witness receipts are delivered but there's no poc receipt in the blockchain.
There are people who confuse stuff and started posting the "max retry" logs, but that's not what the issue is about.

50% of my miner delivered witnesses (around 46%-47% to be exact) end up with no poc receipt in the blockchain. And this has been happening since the start, even when my sensecap was stock.
It can happen for others for that percentage to vary, which is your case in which maybe 90% of witnesses end up with no poc receipt.

[FezzFest]

I can confirm @ledmaniak's findings. Multiple hotspots whose earnings went down to ~10% of what they were before January 1st.

None of the hotspots above have sent a single beacon since January 1st, while they sent multiple before.
Configuration of these hotspots remained unchanged during this time. Every single attempt to make them work again so far has been unsuccesful.

[apppie123]

Don't give too much attention to the title, check the content. It's the same issue where witness receipts are delivered but there's no poc receipt in the blockchain. There are people who confuse stuff and started posting the "max retry" logs, but that's not what the issue is about.
OK, I see you're right. It is good that it is brought under attention again as it might have become worse?

50% of my miner delivered witnesses (around 46%-47% to be exact) end up with no poc receipt in the blockchain. And this has been happening since the start, even when my sensecap was stock. It can happen for others for that percentage to vary, which is your case in which maybe 90% of witnesses end up with no poc receipt.
I can understand that these things happen. But this 80-90% drop(well, it's almost at 100% now) happened from one day to the other for 15 of my units and it's happening since Dec 29th and not just me. No 50% drop, but close to 100% for a lot of days.

I have onboarded 4 new units several days ago. They worked for two days and are now completely flatlined as well.

So far the only response I have seen is one of the team members saying that they don't see this in their data and the network is running(and thus the issue doesn't exist?). It's disturbing.

[ricopt5]

So far the only response I have seen is one of the team members saying that they don't see this in their data and the network is running(and thus the issue doesn't exist?). It's disturbing.

Yeah, no...

This data shows that roughly 50% of challenges, end up with no poc receipt.
Of course, this can be caused by p2p issues where the challenger isn't able to dial the challengee, but as we can see in our miner logs, that's not the only issue.
Unfortunately, there's no way to get statistics of the "delivered witness but no poc receipt" issue.

[PaulVMo]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

[ash-vd]

I have the same issue, with a complete wallet. These miners are professionally installed in radio towers, and were all earning 1HNT+ per 24 hours. From the 1st of January this has been decreasing, with 0,25HNT on 1-1 and now 0.08HNT per miner.

This is a screenshot of the complete wallet, with 67 miners in it:

30-12: 47HNT
31-12: 23HNT
1-1: 12HNT

The drop on the 26th and 27th is expected as there were issues with the internet connection. I've checked, and the connection is stable now.

There is 1 miner in this wallet that has "custom" firmware: the packet forwarder and miner run as a docker container on a linux arm64 OS. These containers are the open-sourced containers from Helium without any changes. When I check the logs for this miner I see many messages that the witness has successfully been sent to the challenger, but almost none of these gets rewarded. Please see the following screenshots. In the miner logs there are clearly multiple witnesses sent in just an hour. However, I only got rewarded for 1(!) witness for the whole day.

Here are some screenshots of other hotspots in that same wallet:

[davidwang1030]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

I have the same problems，can we talk about more details，can you give me ur emails？

[skylign]

#1339 (comment)
agree with you
I think your answer is other question's solution , my problem is more similar with your answer . I don't receive challenge for more 4 days , not confirmed in blockchain , but don't receive , I doubt my address don't be recorded by other HS in their peerbook and what can I do ?

[zxgangandy]

Regarding potential peerbook issues, the fact that witnesses are delivered to the challenger ("successfully sent witness" in the log) seems to indicate that the DIY miners peerbook it Okay. That log message is written when a connection is established with the challenger but technically before the witness data is actually sent on the connection. Nonetheless, if there was a peerbook issue getting the challengers p2p listen address, this message would not be seen as a connection to the challenger would not have been established. This may indicate that the witness is being dropped somewhere else along the way - either in the transmission over the connection or during validation by the challenging hotspot.

On the beaconing drop off, the lack of receiving challenges could still indicate a peer book issue. The miners address may not be appearing in challenger miners' peerbooks or unable to be resolved via ARP. This is evident by the lack of log messages showing that p2p messages are being received and decrypted by the onion server for beaconing.

Same like this issue, please help to anlyze.

[JJsilvera1]

I can confirm @ledmaniak's findings. Multiple hotspots whose earnings went down to ~10% of what they were before January 1st.

None of the hotspots above have sent a single beacon since January 1st, while they sent multiple before. Configuration of these hotspots remained unchanged during this time. Every single attempt to make them work again so far has been unsuccesful.

Same issue with a few OG helium miners that I have and friends have. I am unsure how to debug them or to dive down deeper to see whats wrong. But I assume this is the issue. After the newest update... just flatlines and no activity.

[timetraveler90]

@JJsilvera1 this is exactly the same issue i am experiencing right now with my sensecap. From nearly 0.5hnt a day dropped to 0.02

[SrxDev]

Same issue here. No beacons in 10 days and lost ~80% of witnesses. None of the other miners in the nearby hex have been impacted.

Miner is only issuing challenges.

[apppie123]

It was noticed by several people that the few witnesses that are registered on the blockchain come from challengers from makers FreedomFi, Syncrobit and PantherX.

Are they late with updating the latest image? What mechanism has changed that other brands like sensecap and bobcat never seem to be able to complete a PoC submission from the witness?

[tomasz-grabowski-netbulls]

I'm experiencing the same issue on my diy miner.

[Woutch]

Same issues: Since 5 day's only challenges. Networking wise all tested and ok. And since the latest network var update last day I also see data packets again. Went from 65 witnesses to 0...

Mine is a SenseCap M1: Eager Watermelon Wombat.
Strange thing is also the low beaconer witnesses, if I check most hotspots, they are in populated hexes and should see many more than 0 or 2....

[FezzFest]

It was noticed by several people that the few witnesses that are registered on the blockchain come from challengers from makers FreedomFi, Syncrobit and PantherX.

I can confirm this as well. Looking at one of the affected hotspots, I see:

Looking at an unaffected hotspot closeby, I see:

[semihsmg]

All of my hotspots also flatlined, 6 of them at the same time. Decrease in rewards started in January 3rd but flatlined in 5th. All hotspots lost %90 earnings. All was above average.

[semihsmg]

Same issue here. No beacons in 10 days and lost ~80% of witnesses. None of the other miners in the nearby hex have been impacted.

Miner is only issuing challenges.

my hotspots also only get reward from challenges

[zxgangandy]

The truth is that the helium technical team only five people, they have no processing ability!

If this is true. Why not the helium technical team transfers the project to another team who can hold all the situations. Instead of always putting us to wait and wait.

[velasquez87]

I have the same issue on my bobcat since 29 Dec. Around me everybody is earning 0.5hnt a day while mine is at 0.05...almost no witness and beacon 0 0

B

[serbyxp]

Ditto. I notice the logs are sending the receipts, sometimes it needs to try 2-4 times . Is this an issue on the Erlang and MAC address? Is MAC or HARDWARE address some how used in the the Erlang to obtain a connection / accept receipts with peers? Could p2p peers with a ports with 2154 or any other random port not opened by the container be an issue? Nebra miners have a upnp container, Should one be implemented by all hotspots, or any further ports be open either outbound inbound on the container, or on the network environment?

The receipt gets accepted, but I have not seen all the witnessing rewards on receipts. I understand not all witnesses get put into the PoC rewards if its over X amount of witnesses I think 10 now .. and that is " randomly selected" are the hotspots that dont "win" the PoC reward if total witnesses are over X amount not get their " receipts reported"? and/or are " invalid witness sent and then discarded by the end peer due to a " flagged" invalid status of some sort?

[JJsilvera1]

One of my OG miners is only able to see beacons from other OG miners. It no longer is able to get rewards if it sees bobcat or rak or any other kind of manufacturer.

[serbyxp]

One of my OG miners is only able to see beacons from other OG miners. It no longer is able to get rewards if it sees bobcat or rak or any other kind of manufacturer.

Could this be a lora_hal issue? incompatible? Is the sx130x_hal Packet Forwarder still the defacto? or has it changed to a newer packet_forwarding service that isnt the Semtech flavor or something different? The packet_forwarder from lora-net sx1302_hal is what im running which is the Semtech UDP packet forwarder. Is it possible that something in the msg header that is conflicting with the recipient of our PoC receipt, like if the msg string in the MAC or euid could be formatted differently causing this?

** also I notice in the logs whenever a client connects to the miner to send a message, it isnt using port 1680, its using the second port number in the sys.config ..

      {radio_device, { {0,0,0,0}, 1680,
        {0,0,0,0}, 31341} }
    ]}
    

I noticed that the 31341 port is the one being used. Should our container reflect this or our networks?
if our packet forwarder is in another container or running in shell would that cause a beacon not to be sent? I have seen no error, on this. I have just noticed it. The radio indicator light runs on that it sent a packet. Im not familure with what happens after a packet is sent over the radio?? like their a an ACK reply or packet containing metrics of the concentrator in a msg or string format that could be conflicting with the port, or the p2p network?

[FuraoEvil]

This looks quite similar to my wallet
But no DIY miners here, 7 Bobcats and 2 Heltecs
I sold one Bobcat but it doesn't look better in the new owners wallet. Almost as if they were blacklisted.

[zxgangandy]

I have the same issue:

[biohazd]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

[Woutch]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

Did you do something? I already power cycled, reset my blocks and start from new snapshot + turbosync (sensecap). But still no beacons for 7 day's now.

I'm not relayed, and sending challenges, but these never go over 2 responses. mostly 0.

Thanks!

[biohazd]

I have the same issue. What’s weird is since 4 hours ago it has picked back up and I’m seeing more activity.

Did you do something? I already power cycled, reset my blocks and start from new snapshot + turbosync (sensecap). But still no beacons for 7 day's now.

I'm not relayed, and sending challenges, but these never go over 2 responses. mostly 0.

Thanks!

I did a reboot about 4 hours ago.

I did also reboot yesterday and the day before and those did nothing. My miner is always fully synched and has good internet connectivity.

[Woutch]

@zxgangandy @Woutch Please read @BenoitDuffez's comment carefully. Kerlink did not change anything. Helium pushed an update to the seed nodes after which the issue was resolved. I'm wondering if a helium dev could enlighten us to what was changed on the seed nodes?

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

This is what we see as well. I checked a random Kerlink hotspot and notice that during the period between January 1st and January 5th, most challengers were Synchrobit, FreedomFi and PantherX. This also seems symptomatic for the issue we currently still experience.

Thanks to point this out. Going to try the next workaround, put miner in relay status by closing port. Seems to be counter productive. But anything is better than only creating challenges... (and maybe wait on an official statement from the Helium team).

[apppie123]

@zxgangandy @Woutch Please read @BenoitDuffez's comment carefully. Kerlink did not change anything. Helium pushed an update to the seed nodes after which the issue was resolved. I'm wondering if a helium dev could enlighten us to what was changed on the seed nodes?

What I saw was a lot of witnesses sent successfully but they never reached the Blockchain. It was fixed by itself.

This is what we see as well. I checked a random Kerlink hotspot and notice that during the period between January 1st and January 5th, most challengers were Synchrobit, FreedomFi and PantherX. This also seems symptomatic for the issue we currently still experience.

Thanks to point this out. Going to try the next workaround, put miner in relay status by closing port. Seems to be counter productive. But anything is better than only creating challenges... (and maybe wait on an official statement from the Helium team).

Already tried that. It doesn't help either.

[dogweather]

Does anyone know where these seed node changes would show up? Is there a relevant repo? Or blockchain transaction type?

We should be able to find what was changed, and who was responsible.

[dogweather]

This is a graph posted to the Kerlink Discord. It supports the theory that Kerlink was disproportionately affected.

So it'd be very interesting to know what the Helium devs changed that resolved this.

[FezzFest]

@dogweather As far as I know, the code running on the seed nodes is built from this repository. See tag 'seed0.0.45'.

[FezzFest]

After extensive debugging and reverse engineering, the issue was found.
The Helium developers included an unofficial modification in the docker container pushed on quay.io.
This unofficial modification blocks people from participating in Proof-of-Coverage, thus reducing their hotspots earnings to zero.

If you want to know whether you are affected, I created a github repository with a tool that returns true (if you're banned) or false (if you're fine): https://github.com/FezzFest/check-helium-bannedlist

[SrxDev]

This might help solve the issue. https://github.com/FezzFest/check-helium-bannedlist .created byFezzFest

[Woutch]

YES! Found my hotspot in the list... (eager-watermelon-wombat)

I do not know why, I have a regular Sensecap M1 with an 5.8dbi omni on my roof. Normally earning only a small amount, 0.1 to 0.2 HNT per day.
No modification whatsoever.

At least I don't need to keep debugging all things I can think off, just waiting on the new 2022.01.11 firwmare.

I hope if the list comes back, I'm not on it anymore...

[FuraoEvil]

Ok so let me get this straight:
"The peoples network" banned thousand of miners in an act without vote or mandate by "The People" following non existent rules.
And instead of apologies they are now pulling a quick poll to have it validated afterwards by the community?

So let me line things up:
Consider a wallet with 10 miners in it, all legit and above average got blacklisted entirely since 3 weeks
This counts for lets say 2 HNT per day/40 HNT over the ban period for that wallet alone.
These 10 Hotspots didn't send any Beacons which results in a loss of 200 beacons sent.
Which means potentially 3600 witnessed beacons gone missing for the surrounding Hotspots
Oh and there are the challenges that never reached their destinations and therefor couldn't be beaconed and witnesses
That's another 1800 PoC events not taken into account by completely innocent Hotspots around the world.
Do I get this right? And the best you can do is to say you acted in the best interest of "the peoples network"?

Heads up to the hard work of the community members who did the work over several days to find and prove this as Helium obviously decided to not tell anyone till they were forced to break their silence today.

For me that's a very weak stance but I let you all judge for yourself...

Check below if yours on the list or not

The discussion on Reddit: https://www.reddit.com/r/HeliumNetwork/comments/rywhtm/3000_hotspots_flatlined_since_dec_27th/
The inoffical list: https://gist.githubusercontent.com/ricopt5/d7a8fd85de6dc18cbb72ab74b92d0223/raw/640f84a05c6bdbef2133e4638682646006175478/helium_blacklist.csv
The declaration of Helium in their Discord > Announcements

[serbyxp]

Ughh this seems all sorts of messed up. Cus if they are black listing people based on “statistics” if I’m 1 wallet with 1 hotspot on it. And I get black listed, and the person with 20 hotspots on 1 account doesn’t … “statistically” speaking the person with 20 hot spots on 1 account is more then likely the person “gaming” if they are . So the statistics of the 20 hotspots vs my 1 makes me the “gamer” ?? If all the RSSI data and all the SNR data is being quantified from “false data” then the person with accurate data is looked as the “gamer” ..?

I'm not on that list... So That doesn't explain very much for me. Unless the people around me are on that list... or the peers / challangers are on that list, that could indirectly affect hotspots not on that list.

[dogweather]

@FezzFest Can you show us the source of your blocklist data?

I looked through your commits and source code but couldn't find info about how you gathered the data.

Your work looks excellent, but I'd like to verify it.

[dogweather]

@dogweather As far as I know, the code running on the seed nodes is built from this repository. See tag 'seed0.0.45'.

Wow, those are some unsavory coding practices: all those dependencies are pinned to particular commits. This makes the changes completely opaque and difficult to follow.

[FezzFest]

@dogweather The code was extracted from the miner_poc_handler.beam file in the docker image hotspot vendors pull from https://quay.io/repository/team-helium/miner. You can disassemble the file with the following erlang oneliner: file:write_file("/tmp/miner_poc_handler.disasm", io_lib:fwrite("~p.\n", [beam_disasm:file(miner_poc_handler)])).

The blacklist was only in the compiled images, it was never published in any of the github repositories.

[FuraoEvil]

Any news about how to appeal? My whole fleet is blacklisted and my hosts and I are getting nervous about the way Helium (doesn't) handles this ....

[apppie123]

Any news about how to appeal? My whole fleet is blacklisted and my hosts and I are getting nervous about the way Helium (doesn't) handles this ....

They are going to remove the list in the coming release. When voting is done they are restarting this list again. It is currently unknown who will stay on the list and who won't. There is no way to appeal. Helium thinks they know 100% for sure who to ban and who not.

[serbyxp]

How and where is the voting take place? Or is that only for validators?

[FuraoEvil]

How and where is the voting take place? Or is that only for validators?

heliumvote.com

[serbyxp]

hmm I started going through manually * if someone has a script for this would be great* inputting all the /p2p/ addresses that libp2p_framed_stream dial failed . Reason not found. Well they are " validators" for the most part, but validators with more then 1 validator. Not sure if this means anything but, most of the validators on the list show like 0 earnings, and/or huge spurts or spikes of earnings. Can this be explained?
for example.
https://explorer.helium.com/validators/11jshWUZ1c917ozDs3N3bwjasf9sVct7MASR69ih4x4LboAkrCq x4
https://explorer.helium.com/accounts/13zCZhFapZHTgsDDba76Ts9yATNAhULH2Dm1df6AxbrMuZqySP5/validators
https://explorer.helium.com/accounts/13zCZhFapZHTgsDDba76Ts9yATNAhULH2Dm1df6AxbrMuZqySP5

https://explorer.helium.com/validators/14jtNUDMrNaMbxXwPRqF3wor3zXUvvN7Xf2xYc3hsHtxm7UeYmx x3

• https://explorer.helium.com/accounts/14QLjXRXhsN4FcLCiuuxJDjawf2uasWN65jgEr3m1h3iWcQPsWR/validators

https://explorer.helium.com/validators/13497zkEKsGzJLxT1oQGTEtz995SKiS7USDyzZNQGMq1LaZC8u8
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/13L6PhNrLWnfksmDADNgExNtF3mPYma5XZDwWnYErxpyyhA11Yr
https://explorer.helium.com/accounts/14WKefruV8g5DuDTzeH4vUYc1QAvfzcEuT5vKsvBeQnySwCoBmr/validators

https://explorer.helium.com/validators/11RvgBqESA44SAh7WbkFRAgocGL36E8zDgGKf34znJp1T1xc35X
https://explorer.helium.com/accounts/147g1izCWbUdi3VQEh75QP5P5PBmQbsYyj26KD5XmqVyFGgWQHq/validators

https://explorer.helium.com/validators/13SaX9vaYFyPC2btEYjwuWEnaZTtJVTYNnzu5HUrJuRKaXDwaGX
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/112EqfDG3go7XYmzLdbai7Fonenc4kzEa6Qo5JwpzA8bWmBauyCn

https://explorer.helium.com/validators/1127bXb2j64qzuVp573PjUqpmtseyMTp5cpuc24qxB1FDsh8KCCn

• https://explorer.helium.com/accounts/14a9yBWqiKjSt8JtbaejVGgNwhYh3vMeoN7QjT6W7MynbdVvZCm/validators

https://explorer.helium.com/validators/112f8TusdvqgBtVL8wx3yvKH6wDvyqPTUU7xRSY1z2VGQY2u7eb4

• https://explorer.helium.com/accounts/14a9yBWqiKjSt8JtbaejVGgNwhYh3vMeoN7QjT6W7MynbdVvZCm/validators

https://explorer.helium.com/validators/11vEzB8zvKxfs7BgRRDpcB7mXqDRi9xwTeGCM65xHZyR67ALVcR
https://explorer.helium.com/accounts/14bPkx9h3JkkfZWXXVCsjdPbKKdXxS5LrA4PF9YjpLFvxM5qHzw/validators

https://explorer.helium.com/validators/11Yaouq8YGSvQybkn7QWuGvKz59EAz343yYygcrMm9xAUfV54kr
https://explorer.helium.com/accounts/14ra2V8TYTvvvZazqjh4i41ixfswa6czJBHDc4eZuJz13gmqYT2/validators

https://explorer.helium.com/validators/11jshWUZ1c917ozDs3N3bwjasf9sVct7MASR69ih4x4LboAkrCq x4

https://explorer.helium.com/validators/11jshWUZ1c917ozDs3N3bwjasf9sVct7MASR69ih4x4LboAkrCq x4

• https://explorer.helium.com/accounts/14RddnkW1UGNwNipHWw3LxJHTAxdMHRRw55Hx8YnufLfDJQwvgn/validators

https://explorer.helium.com/validators/14NZXZ86Gs1Wwn5nhUUkYzrrjFeirpsMbMWH4jAVU3i8QtTpAMP
https://explorer.helium.com/accounts/14mCCTAW5QtmdoJ5TTsnTDEJjjvUAwDeYwhW7p1So7wiyWiMS4p/validators

https://explorer.helium.com/validators/11oD2dJfNEj9znU9Q544hDr4ZuTKo1HrKp59KEHxUKMBPwXfgF7

• https://explorer.helium.com/accounts/13rboQo6T3SWZn2FYCArLfvgM8PfRZrobUw1QNjn1eUoDryYhhV/validators

https://explorer.helium.com/validators/11p3DuvqZEDaT84bC98j3PtYFUfeVS9aZj1QvSkaruLjpgaNiTp

https://explorer.helium.com/validators/113QZg4FhdJ6PDUDYHcNKUsdZ5v4XtmAettFq6jBi7eSrNQSPkQ

https://explorer.helium.com/validators/141rLnqkuP6seaHhVofo4opvGWAPGZr4nL2fAfwXMma68kMG8FZ
https://explorer.helium.com/accounts/14q2ieVdr7UDhuqftgPF6GgASLFgZcqqZc2GELek1eARATLNX8R/validators

https://explorer.helium.com/validators/13RowdgDYCQnhJCHWNGZmzvgo52Z5TVYehn7b8mqD3oCdhMwHFb
https://explorer.helium.com/accounts/142fmJcnY5rUoH6A4Wb8EMHnpjxmZATCiQiaT1mZ4LYffoM8uwm/validators

https://explorer.helium.com/validators/112nd6CE2vfYvbupXhFvH4XHBDtZ8GuHmtPwdSkbYoYn5EBKZLR7
https://explorer.helium.com/accounts/14nYSAgAP4z6QqdnDgcyAPS9Hg6fCQXkDmyFH8nqMFX29UYhXCJ/validators

https://explorer.helium.com/validators/11JJcn8qiHBGJBUgRVG34P23B7gNS2S1XeTh7FsCdhKaqWUAiwo
**_https://explorer.helium.com/accounts/13zCZhFapZHTgsDDba76Ts9yATNAhULH2Dm1df6AxbrMuZqySP5_**/validators

https://explorer.helium.com/validators/13ZD67gCjaGweEmGbCgzAr3BxdEk6X51jxmU8X4S2ZMLGrXGdsr

• https://explorer.helium.com/accounts/14QLjXRXhsN4FcLCiuuxJDjawf2uasWN65jgEr3m1h3iWcQPsWR/validators

https://explorer.helium.com/validators/112F36EoDVbZ8VH74rzLoTuDszvgrqBNwf8qaoCURdKjB8bLkBUL
https://explorer.helium.com/accounts/13xamyeKEuEasPVZVzxCB7W1dFWh26pbiQmpRgFMNF2ULSZqngJ/hotspots

https://explorer.helium.com/validators/14kpF5MwjyADkDeSzRSJBD7zqGEyzFbRFZCNz4WPfnuKGgVMZ6t
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/14rd6M5wQ94jMELqADpE1UW1LgLSYkhDqyXQemDEJzdvezjwSHW
https://explorer.helium.com/accounts/13yXU6s9tCw5M96ZoYj5rR2DBPU3XGCNAtr8gg3yWyHUNBZej3A

https://explorer.helium.com/validators/11Yowc1QunPPgEdD7ZzQVnp3ok29QU2gw6Q6a1uogfEF9Mb626U
https://explorer.helium.com/accounts/14bPkx9h3JkkfZWXXVCsjdPbKKdXxS5LrA4PF9YjpLFvxM5qHzw/validators

https://explorer.helium.com/validators/14oikmuDmi1rW3bJ8X4pFnjRWsYDrxSN4kZ72iJuABXw2c5rC4x
https://explorer.helium.com/accounts/13nfQH4DTFTJLfmtZqER2xwfzcEYJ4xdGb5fAaymsJG1xnVtD7D/validators

https://explorer.helium.com/validators/11acHehPDWwcLVE2EQ8ymYMiqSoHucXgjXsvWwARmWxApMDp3FH
https://explorer.helium.com/accounts/14bPkx9h3JkkfZWXXVCsjdPbKKdXxS5LrA4PF9YjpLFvxM5qHzw/validators

https://explorer.helium.com/validators/11acHehPDWwcLVE2EQ8ymYMiqSoHucXgjXsvWwARmWxApMDp3FH
https://explorer.helium.com/accounts/14bPkx9h3JkkfZWXXVCsjdPbKKdXxS5LrA4PF9YjpLFvxM5qHzw/validators

https://explorer.helium.com/validators/112UTYuc7w1p8Au98Jk579ZkMLUCRn6SFeg9QrzGBaHGTfK726an
https://explorer.helium.com/accounts/14bPkx9h3JkkfZWXXVCsjdPbKKdXxS5LrA4PF9YjpLFvxM5qHzw/validators

https://explorer.helium.com/validators/14pyC3eQz7su4n73b53JEp4BXeDw84JfcSSUrS3D2KFV4AQqxzn
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/116R9q9YynKvRV3qxEHkoyLFdfdSRwhSRd9gr2BTwtu38aYcVNK
https://explorer.helium.com/accounts/13r2n7nPRDdrH6oh7CKQaRFPe3dikTrsBBUei6Dn8u8BaSNw5eS/validators

https://explorer.helium.com/validators/13gg3u2tzp11EXwUAEVWzgBRFeRPtzo5NCDnwKEVQnwg5CFuGnk
https://explorer.helium.com/accounts/14FBiE13Xm5PtvkDJp26K99XEz1tT2B3qYVeJ1zAGWwTBdfiZwp/validators

https://explorer.helium.com/validators/112hBQEtFkG5F7oZo7Z1T56j37Jf4YMiyn2k48VkNRknCzYgbQFi
https://explorer.helium.com/accounts/141YKgZaTHMYrLctvPN8bUpwMPomMcrXzcRS3aaDWWwRYWkyHxk/validators

https://explorer.helium.com/validators/11qLhHFqHLAfEYjdkgJppVxaRh8Neqbmn6cZmSH2Eewff1DWQd6

• https://explorer.helium.com/accounts/13rboQo6T3SWZn2FYCArLfvgM8PfRZrobUw1QNjn1eUoDryYhhV/validators

https://explorer.helium.com/validators/14jtNUDMrNaMbxXwPRqF3wor3zXUvvN7Xf2xYc3hsHtxm7UeYmx x3

• https://explorer.helium.com/accounts/14QLjXRXhsN4FcLCiuuxJDjawf2uasWN65jgEr3m1h3iWcQPsWR/validators

https://explorer.helium.com/validators/13wx1FoauHcrQGscYkkurjkzBPY8WBxDDo85VV1Pp1FzAPTnSYM
https://explorer.helium.com/accounts/139kfjEtGbYpwwLpdX1bFhLFLE8ugoY4R9R5EtgjcG81SnrjBSy/validators

https://explorer.helium.com/validators/116dEcasQ4LSoG3oWnzpXfAz5ENkaaAFE8deUdQwRbo9pQtQTab
https://explorer.helium.com/accounts/13xUgU5QC1v6SLv3DeTibiV5WwA9af4yTvHRqpmvZ42BPAVXQmi/validators

https://explorer.helium.com/validators/112huhFo1NA9GfUdV9yvj9D1aNtYpNikKqqYkSejp9VrSvFbeidX
https://explorer.helium.com/accounts/14sMaPVnd8bYmcWgX5bcWqeMDRKK9rnKMziZFeyHxRoDp2kBeC7/validators

https://explorer.helium.com/validators/11jshWUZ1c917ozDs3N3bwjasf9sVct7MASR69ih4x4LboAkrCq x4

• https://explorer.helium.com/accounts/14RddnkW1UGNwNipHWw3LxJHTAxdMHRRw55Hx8YnufLfDJQwvgn/validators

https://explorer.helium.com/validators/14jtNUDMrNaMbxXwPRqF3wor3zXUvvN7Xf2xYc3hsHtxm7UeYmx x3

• https://explorer.helium.com/accounts/14QLjXRXhsN4FcLCiuuxJDjawf2uasWN65jgEr3m1h3iWcQPsWR/validators

https://explorer.helium.com/validators/14ZyCzZ9BvCSCka8CF7XRW4VxKS5JY1zznMPx4FjCwQyexCNqDf

https://explorer.helium.com/validators/11DpH7MXtk5zMfyYvaieGRW7kcL69bu6Mjcw8fPdXX4axpLG2Pb
https://explorer.helium.com/accounts/13TUSEFc3YYPnTcSD9XTvrLovsvMhJjiByWQhGPFdjF9RdNfFFV/validators

https://explorer.helium.com/validators/11MjSbF5q4fjsLk9BmRBVTMJkp83AAiRMhV1iWurupijLE4CELf
https://explorer.helium.com/accounts/13jmQY9KrTNyf9VTLzejMTT7LjLMZ918eS4wcoZ5Ekz1LKK5ns6/validators

https://explorer.helium.com/validators/14PjfFv8WXLUuwD9yyw6AisWeZfjbcqiUhXrzH1Lv1KTCPrJMh7
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/11AyUWjwYcg2J9vmyxtmaBgjULMyRWM9fSrqzKwYdFiX98wteEL

• https://explorer.helium.com/accounts/14RddnkW1UGNwNipHWw3LxJHTAxdMHRRw55Hx8YnufLfDJQwvgn/validators

https://explorer.helium.com/validators/112VmAZZ9crhyvJYwr73jULqjDxQFPhxwHKBqxrsnAxP1xwxiJGU
https://explorer.helium.com/accounts/14qbGa9gJrpvJ4RybVxK9jjzxEMUfzJhB58AN9PiMhRTMniZy2z/validators

https://explorer.helium.com/validators/14Fs7aVBXfDvsng5PfzyfmRWsnt2cC2RkqU61gzfZWVn6gnrzJY
https://explorer.helium.com/accounts/146jMfmxD5vUYSUHsePJX3UJEVpNgNTBKdHWkYgfLTbHHwGRFFN/validators

https://explorer.helium.com/validators/12yPSSxQ3Vzkpv8i9tZEY7jMpEJZydKZu15NWM4BZaW1KCj4EXR
https://explorer.helium.com/accounts/13hKQD1iGeD4UNRDDBq1nXM5im9YWq8sewum76KMgj7xVsNRhPV/validators

https://explorer.helium.com/validators/112KNsqZFoyJNbQDtQfFzGnsAtBD9vZNS8QLiBM5M7rwyEoZechP
https://explorer.helium.com/accounts/14R7CozB2UfPQ6W14QZJYpdrJRqewJKYHhvGVt3AV2wkJ1cLSD9/validators
https://explorer.helium.com/validators/112Hdez8TmtiDAx2NV1vvrGaWwnpmZE6qvxv1pMpmLsHaHe7yBjM
https://explorer.helium.com/accounts/14R7CozB2UfPQ6W14QZJYpdrJRqewJKYHhvGVt3AV2wkJ1cLSD9/validators

https://explorer.helium.com/validators/112rVDSqwLZqmW9iQcGMKBmPKuFjY7iYvLbmKJXCx6aKuyYwGYuK
https://explorer.helium.com/accounts/136nSZ4niPtQNLqgVKi2uj5KVBw363vnjTciBYkv1LcVvda7fgP/validators
https://explorer.helium.com/validators/112indCEozYgWnjK9hPqyzjFWPiXVsDkFXX8eXBLadUnctc4dAkR
https://explorer.helium.com/accounts/136nSZ4niPtQNLqgVKi2uj5KVBw363vnjTciBYkv1LcVvda7fgP/validators

Hotspots within my country US915 failed to dial

https://explorer.helium.com/hotspots/112LXs2UQmgcQdiWzEQaJU32kbQ1FMgQ5cVv7ipJNftpBFPuFgy x2
https://explorer.helium.com/hotspots/112LXs2UQmgcQdiWzEQaJU32kbQ1FMgQ5cVv7ipJNftpBFPuFgy x2
https://explorer.helium.com/accounts/13hJwbP63XbfNLPsdcLAbxVRsGmfuFeqYKcrYtPiubmDSW66LYd
https://explorer.helium.com/validators/11RvgBqESA44SAh7WbkFRAgocGL36E8zDgGKf34znJp1T1xc35X

Out of my country US915 Hotspots that have failed to dial

https://explorer.helium.com/accounts/149x7MNhGz7YAz88YohsrGr7f5iqJAdWQqJ9GAtjsdjJntN4M9j

https://explorer.helium.com/hotspots/11nAKmfcUf7qKEJ9cHzs7C2uz4QKUHzDiwnkHd7MDmvMbMGhHzbx2
https://explorer.helium.com/accounts/13KvVWpSyykM74KGsBUa24XGoLi6sngqJ1WzhdQNGzk1yTQRLgJ/hotspots
https://explorer.helium.com/hotspots/11nAKmfcUf7qKEJ9cHzs7C2uz4QKUHzDiwnkHd7MDmvMbMGhHzbx2
https://explorer.helium.com/accounts/13KvVWpSyykM74KGsBUa24XGoLi6sngqJ1WzhdQNGzk1yTQRLgJ/hotspots

https://explorer.helium.com/hotspots/11e6ZMF3oWnoqKbhfQpn6cn7Mw2v2H8vDnmtEci87D5yP7oBvCm x2
https://explorer.helium.com/hotspots/11e6ZMF3oWnoqKbhfQpn6cn7Mw2v2H8vDnmtEci87D5yP7oBvCm x2

https://explorer.helium.com/hotspots/112mhubbZzaumbsR7oPCo9uG75ciGgZUWwxVEQGJ55aQ31mRA1U1
https://explorer.helium.com/accounts/14tA57nCLwzgaygSQz7U91FUiPQEDUiBKPU4U22a87TLLu6h8Jy/hotspots

https://explorer.helium.com/hotspots/1122YgvJhFuiZh8uxUFdTTs4pHFRE6E2ufBwTiENfJ5Y4osSk6UJ
https://explorer.helium.com/accounts/13TpC7QqjoEd1ChvjUDRzBRLjtDhjGgbbET9CxyFQH8fgaX6qzk

I didnt show up on that black list. But this was just today I got tired of manually inputting this. But Im not sure whats the deal. Just saw the update going to see if that fixes it.. Ill come re format this later the bold is one set of validator accounts the italic another bold italic another - bold - Ill figure it out but definatly something wrong with the fact I cant connect to whole groups of validators under the same account

[mehmet0150]

I have a problem that I haven't been able to solve for about a month. I'm using the Sensecap M1 and suddenly I've had a drop in the number of witnesses. Everything is normal, the device is not transferred, the synchronization is ok. Also, my device is not blacklisted. I'm tired of looking for a solution.

[serbyxp]

I know this isnt available through the normal or what is "standard" setup using the ARM64 image, I had syncing issues that the API didn't update my syncing status for 2 weeks. I ran the miner on AMD64 image while still running the ARM64 miner like normal, and it updated the syncing within an hour of doing that. Ive written several posts and have documented everything somewhere on this Issues board, I don't work in IT nor am I associated with anyone or anything,

My non " expert " hypothesis is either the manufactures are using their own compiled software which conflicts with other platforms.

But this is not manufacturer specific as all my tests have been using Helium Official REPO and Helium Official Documentation.

How I got the 2 different versions to work at the same time is " Not official* I ran that as a test and that's what I concluded. I dont think anyone else on this planet has done this, Or maybe they have but Im the only one that seems to be documenting it so that the Dev Teams or someone that knows how to code, can use it to help better understand whatever issue they are having between dependencies. CLEARLY the network is compromised.

I suggest and propose, a STANDARDIZED dependency Library, The main repo on Helium miner is, STANDARD. but the dependencies are not. They are compiled from diffrent dependencies and/or versions of Erlang or LibP2p whatever it is, Stems from that. They*(ARM64 vs AMD64)
The compile instructions on the main repo if you scroll down a bit, Lists those differences *in dependencies and repos.

If you want to get into conspiracies do that on your own time , you got the right to. But Some may believe it to be " Gaming" between manufacturers and or " users " I don't believe this but You never know in crypto " game " that could be considered a way of " Gaming the network"
Clearly the network is compromised. What is the fix? I don't know,

But note, Validators need to run in or on an AWS server, Which uses AMD64 firmware image. AMD64 firmware image requires a AVX capable core, A AVX Iimage * can not run on some ARM64 processors. BUT the ARM64 firmware image can run on any AWS system. I Even think its more economical to run ARM64 images on AWS paid servers vs the AMD64 version, It could be vice versa but I know that to be true.

Basiclly the way to fix it from a non " expert " point of view. Is plain and simple, not complicated.

Use 1 image that works across all platforms, That image exists. The ARM64 version. If they just stick to 1 official image, Then that would solve this issue. Does anyone listen to me, My wife doesn't, So Im sure Helium wont either.

[chmichael]

Nebras have the same issue!

[jwmakesgames]

Im not the original mind behind the helium miner master folder but i might be able to aid a bit The miner master folder is a compilation of settings for the most common devices breaking down each the code attached can be both helpful and confusing. I would use it as a template to find out a little more behin the scenes and what the standard is for flow for a machine which is desired My helium florida archive is just this it contains miner/master but its contents have been used with balena os and helium console. The platforms tested on both raspberry pi 4 aarch and pi 4 arm64 which are both available in the m1 build by seeed First off it might not be u. It may infact be a new obstcle or structure in line of site. To test the theory you could use helium rf to find out what your line of sight os and if there is any natural or structural blockages. Secondly. I would consider running th m1 on a second antenna linked by t connection. M1 antenna up wards and a lead out to antenna two. Go for 8-10 dbi Also the m1 lorawan gateway chip support 12.8 dbi without antenna. You could boot up then plug up your antenna after boot. Then get a slight boost off if you dont have a option for a second antenna. Reflections space is a factor. Im sure you know all this. Outdoor mounting is best. Omni directiinal always on antenna 1 and directional if you know your target direction also another soft remedy is rewards scale and virtual placement with hotspotty try for seven hundred or better connection score in your strategy. Weed out any connected network below .60 Hipe this helps a little

…

Sent from my iPhone On Jan 19, 2022, at 4:43 PM, serbyxp ***@***.***> wrote:  I know this isnt available through the normal or what is "standard" setup using the ARM64 image, I had syncing issues that the API didn't update my syncing status for 2 weeks. I ran the miner on AMD64 image while still running the ARM64 miner like normal, and it updated the syncing within an hour of doing that. Ive written several posts and have documented everything somewhere on this Issues board, I don't work in IT nor am I associated with anyone or anything, My non " expert " hypothesis is either the manufactures are using their own compiled software which conflicts with other platforms. But this is not manufacturer specific as all my tests have been using Helium Official REPO and Helium Official Documentation. How I got the 2 different versions to work at the same time is " Not official* I ran that as a test and that's what I concluded. I dont think anyone else on this planet has done this, Or maybe they have but Im the only one that seems to be documenting it so that the Dev Teams or someone that knows how to code, can use it to help better understand whatever issue they are having between dependencies. I suggest and propose, a STANDARDIZED dependency Library, The main repo on Helium miner is, STANDARD. but the dependencies are not. They are compiled from diffrent dependencies and/or versions of Erlang or LibP2p whatever it is, Stems from that. They*(ARM64 vs AMD64) — Reply to this email directly, view it on GitHub<#1339 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AICUODUZCKKZIP3WYS2P34TUW4WBRANCNFSM5LMNFKMQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[malder2508]

I've been looking into why my hotspot (good setup, beacons never get more than 10 witnesses) and my neighbouring hex hotspot (good setup, never more than 10 witnesses) could fail to witness each other so often.

In looking through the last 24 hours of my HS logs, I found my HS had witnessed 24 times

Out of htose 24:

• 3 were good and resulted in rewards
• 11 Failed to connect to the Challenger (timed-out)
• 5 were sent successfully to the Challenger but the entire PoC is missing from explorer.
• 2 were sent to the Challenger but there were more than 14 witnesses (assume I lost the lottery)
• 1 was sent to the Challenger and was apparently ignored/lost
  (I get rewarded only for a small fraction of my actual witnessing #1238 (comment))

I turned on a debug level of logging to investigate the failures to connect to the Challenger and found that my hotpsot would always get failed to dial challenger "/p2p/...": not_found

Until this happens:
@libp2p_multistream_client:negotiate_handler:{21,13} Negotiating handler for "miner_poc/1.0.0" using ["miner_poc/1.0.0"]

And then the next connection works.

When my HS times-out from too many retries, it never has that log from libp2p_multistream_client
But this is only seen in the debug log.

Here is a typical witness:

2022-02-09 08:41:47.744 [info] <0.1953.0>@miner_onion_server:try_decrypt:{466,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM found poc. attempting to decrypt
2022-02-09 08:41:47.824 [info] <0.1953.0>@miner_onion_server:decrypt:{362,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:41:47.824 [info] <0.1953.0>@miner_onion_server:decrypt:{372,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM could not decrypt packet received via radio: treating as a witness
2022-02-09 08:41:47.824 [info] <0.12714.0>@miner_onion_server:send_witness:{188,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:41:47.955 [warning] <0.12714.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11ZwbKf2HKEqhEq21kcpkvekQAJvGns8VBXKFFs9nkEygYyWC8z": not_found
2022-02-09 08:41:58.219 [debug] <0.1958.0>@miner_poc_statem:requesting:{205,29} request not allowed @ <<207,100,43,251,171,18,1,160,172,242,57,239,90,57,130,56,227,196,141,170,133,97,53,40,38,142,252,216,228,207,175,17>>
2022-02-09 08:42:11.896 [info] <0.12542.0>@miner_onion_server:send_witness:{246,37} poc_id=XWaLY-z4a7bfrW64AVwNJ1H1hJSD4EKz_6MLP2mzupg re-sending witness at RSSI: -131, Frequency: 917.6, SNR: -11.0
2022-02-09 08:42:12.010 [warning] <0.12542.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/112KpGZKM1cS7eZwxaLfUWMnkXZtkFufVF1tDefL44yPHz5QDvsL": not_found
2022-02-09 08:42:17.956 [info] <0.12714.0>@miner_onion_server:send_witness:{246,37} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM re-sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:42:19.372 [debug] <0.12714.0>@libp2p_multistream_client:negotiate_handler:{21,13} Negotiating handler for "miner_poc/1.0.0" using ["miner_poc/1.0.0"]
2022-02-09 08:42:19.861 [info] <0.12714.0>@miner_onion_server:send_witness:{251,37} successfully sent witness to challenger "/p2p/11ZwbKf2HKEqhEq21kcpkvekQAJvGns8VBXKFFs9nkEygYyWC8z" with RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:42:19.862 [info] <0.12742.0>@miner_poc_handler:handle_info:{74,5} client sending data: <<18,177,1,10,33,0,194,62,139,100,39,226,150,133,50,158,92,38,180,33,113,58,209,253,239,48,38,137,54,17,183,211,163,69,17,235,99,111,16,132,164,156,167,195,177,132,233,22,24,137,2,34,32,247,224,19,145,112,150,139,139,142,127,165,111,184,129,21,84,36,182,129,145,34,76,9,179,91,208,200,40,118,211,114,204,42,71,48,69,2,33,0,218,164,161,48,20,65,144,223,79,218,210,38,147,106,241,173,124,29,52,156,73,249,205,180,170,119,236,247,77,96,117,150,2,32,65,117,215,47,104,228,26,54,139,94,249,23,172,208,86,126,84,19,126,254,26,101,135,146,241,154,67,125,255,114,248,120,53,0,0,104,193,61,154,89,101,68,64,3,74,8,83,70,57,66,87,49,50,53>>

[serbyxp]

I’m in Florida and I’m a dense area , but I have been beaconing etc, but most come back with the full 14, some come back with less then 10 and I’ve had 0 as well , but I noticed there is a new error that I’m getting some yamux dial error I haven’t dug into it more but, I also have a lot of errors for multi stream dial, which from my light digging seems to be when I challange or when I beacon, after all the witness receipts come in I get them I receive them but then when I go to send them one of those errors occurs . Not sure what or why. But they fix one thing and a new thing pops up band aides for everything on the network .. I read somewhere that they keep doing a bunch of stuff without putting it up to a vote , I’m already beginning to believe some of the conspiracy theories out there. It almost seems like it’s being done systematically, or on purpose. But whatever. Just going to wait until this thing breaks even and re sell it. Or just forget about it , wasting to much time on something that never gets fixed.

heh also I’ve gotten tons of notifications from Amazon and other apps / services that I use that my passwords (which are all diffrent ) are trying to be logged in from a bunch of Chinese ip addresses. So I mean it’s not heliums fault, but by having to disable a bunch of my firewall settings etc having the ports open on my modem is compromising my home network. Never had that issue until now.

I've been looking into why my hotspot (good setup, beacons never get more than 10 witnesses) and my neighbouring hex hotspot (good setup, never more than 10 witnesses) could fail to witness each other so often.

In looking through the last 24 hours of my HS logs, I found my HS had witnessed 24 times

Out of htose 24:

• 3 were good and resulted in rewards
• 11 Failed to connect to the Challenger (timed-out)
• 5 were sent successfully to the Challenger but the entire PoC is missing from explorer.
• 2 were sent to the Challenger but there were more than 14 witnesses (assume I lost the lottery)
• 1 was sent to the Challenger and was apparently ignored/lost
  (I get rewarded only for a small fraction of my actual witnessing #1238 (comment))

I turned on a debug level of logging to investigate the failures to connect to the Challenger and found that my hotpsot would always get failed to dial challenger "/p2p/...": not_found

Until this happens: @libp2p_multistream_client:negotiate_handler:{21,13} Negotiating handler for "miner_poc/1.0.0" using ["miner_poc/1.0.0"]

And then the next connection works.

When my HS times-out from too many retries, it never has that log from libp2p_multistream_client But this is only seen in the debug log.

Here is a typical witness:

2022-02-09 08:41:47.744 [info] <0.1953.0>@miner_onion_server:try_decrypt:{466,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM found poc. attempting to decrypt
2022-02-09 08:41:47.824 [info] <0.1953.0>@miner_onion_server:decrypt:{362,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:41:47.824 [info] <0.1953.0>@miner_onion_server:decrypt:{372,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM could not decrypt packet received via radio: treating as a witness
2022-02-09 08:41:47.824 [info] <0.12714.0>@miner_onion_server:send_witness:{188,13} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:41:47.955 [warning] <0.12714.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11ZwbKf2HKEqhEq21kcpkvekQAJvGns8VBXKFFs9nkEygYyWC8z": not_found
2022-02-09 08:41:58.219 [debug] <0.1958.0>@miner_poc_statem:requesting:{205,29} request not allowed @ <<207,100,43,251,171,18,1,160,172,242,57,239,90,57,130,56,227,196,141,170,133,97,53,40,38,142,252,216,228,207,175,17>>
2022-02-09 08:42:11.896 [info] <0.12542.0>@miner_onion_server:send_witness:{246,37} poc_id=XWaLY-z4a7bfrW64AVwNJ1H1hJSD4EKz_6MLP2mzupg re-sending witness at RSSI: -131, Frequency: 917.6, SNR: -11.0
2022-02-09 08:42:12.010 [warning] <0.12542.0>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/112KpGZKM1cS7eZwxaLfUWMnkXZtkFufVF1tDefL44yPHz5QDvsL": not_found
2022-02-09 08:42:17.956 [info] <0.12714.0>@miner_onion_server:send_witness:{246,37} poc_id=-uUgjDmuPbFbWung08PUCKsDPizcYGmREFPg9GFWIQM re-sending witness at RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:42:19.372 [debug] <0.12714.0>@libp2p_multistream_client:negotiate_handler:{21,13} Negotiating handler for "miner_poc/1.0.0" using ["miner_poc/1.0.0"]
2022-02-09 08:42:19.861 [info] <0.12714.0>@miner_onion_server:send_witness:{251,37} successfully sent witness to challenger "/p2p/11ZwbKf2HKEqhEq21kcpkvekQAJvGns8VBXKFFs9nkEygYyWC8z" with RSSI: -133, Frequency: 917.4, SNR: -14.5
2022-02-09 08:42:19.862 [info] <0.12742.0>@miner_poc_handler:handle_info:{74,5} client sending data: <<18,177,1,10,33,0,194,62,139,100,39,226,150,133,50,158,92,38,180,33,113,58,209,253,239,48,38,137,54,17,183,211,163,69,17,235,99,111,16,132,164,156,167,195,177,132,233,22,24,137,2,34,32,247,224,19,145,112,150,139,139,142,127,165,111,184,129,21,84,36,182,129,145,34,76,9,179,91,208,200,40,118,211,114,204,42,71,48,69,2,33,0,218,164,161,48,20,65,144,223,79,218,210,38,147,106,241,173,124,29,52,156,73,249,205,180,170,119,236,247,77,96,117,150,2,32,65,117,215,47,104,228,26,54,139,94,249,23,172,208,86,126,84,19,126,254,26,101,135,146,241,154,67,125,255,114,248,120,53,0,0,104,193,61,154,89,101,68,64,3,74,8,83,70,57,66,87,49,50,53>>

I also have this issue. Sometimes not the 10 hotspot minimum but that it just seems like PoC is broken. I read somewhere also that theirs a new PoC calculation of some sort, not sure if that’s official yet or in the works. Had something to do with rewards and witnesses , what I mean by witnesses is what has been posted about witness list, if you have x amount of witnesses in your list, the calculation does some weird thing that I didn’t understand about how much rewards you can get from beaconing / witnessing . I don’t know 🤷‍♀️ it was strange didn’t make much sense to me. But it’s almost like a CAP per hotspot in a sense . I know the beacons are like capped at 3-4 per day now. Also their was the blockchain outage that messed things up the other day. But I have friends who’s hotspots haven’t beaconed in weeks or months… they witness a lot but even with their 100s of witness rewards they are still doing worst then anyone that gets the 3-4 beacons per day. Which is less then 2.00$ per 24 hours , if you beacon your Atleast gaurenteed 1.50$ per day but then if your getting a max of 10 or less your losing out on the % from the difference of 14 . So I beaconed with 0 witnesses because the challanger didn’t receive any of the receipts, then that’s one beacon that you get screwed out of. I mentioned it before as a suggestion , that they shouldn’t let relayed miners participate in challanges. It should be a metric of some sort for creating challanges. Because it messes up everyone else that’s running their hotspots correctly .

*The multi stream thing you can see in the console log, but you got to filter or find all the errors or warnings etc… theirs also another error that occurred which is an IDENDiFY/numbers fails to handshake that’s strange also. And it closes the stream not sure if you’ve got that going on to

another thing I have noticed based on my light dig, was that all the hotspots in my area beacon at around the same time. So it seems like the network / beacon algo is doing something that’s geolocation specific, wether that’s always been that way or not, or is part of the anti gaming metrics , but it seems like when I beacon then I get witnesses and after a certain amount of time then the network goes dead around me. I don’t know maybe that was just on that day that I looked at which happened to be when the blockchain outage occurred .

[malder2508]

@serbyxp I understand your frustration! The network obviously has a lot of fundamental problems, and because we can't get the full picture we feel left in the dark and vulnerable to all the gamers that are taking advantage of it.

I get the impression the core developers are well aware of the problems with libp2p and the network in general. I don't know yet if they are up to the task of fixing it... but let's give them a chance.

I am trying to capture the failed transactions in detail so that it might help them debug their code.

[DCYeahThatsMe]

How long has this issue been going on? Why can't anyone spend some time and affect it as it is not specific to one miner, no blacklist involved - it's a great many people and it is completely wrecking the network. Yes, there's lots of other issues to fix... I get that. But we as a miner community are very frustrated and the issue is out of our hands. Please. Please. Please. PLEASE fix this libp2p "failed to dial challenger" witness issue. Thank you!

[serbyxp]

From my understanding the network is moving to gRPC so libp2p is going to be depreciated, that's probably why they aren't to focused on the issue, if they are focused on the gRPC release? not sure when that is supposed to come out , but that's when the network shifts from full miners to light hotspots. So... I mean kinda just spinning our wheels trying to debug things if they aren't even going to implement anything new to libp2p.

[florian-asche]

Does anyone know a way to change the retry time?

Here are my logs. And as you can see, it try to dial the challenger and fails. The miner then wait some time and try again and it is successful then. I want to change the time between because even after the successful message the witness still dont reach the end and i dont get a reward. I suspect that it took to much time.

2022-03-08 15:37:49.516 145 [info] <0.30647.3>@blockchain_sync_handler:handle_data:{176,29} sending blocks [1258187,1258188] to sync peer
2022-03-08 15:37:53.507 145 [info] <0.2305.0>@miner_onion_server:try_decrypt:{466,13} found poc. attempting to decrypt
2022-03-08 15:37:53.591 145 [info] <0.2305.0>@miner_onion_server:decrypt:{362,13} sending witness at RSSI: -116, Frequency: 868.5, SNR: -8.5
2022-03-08 15:37:53.591 145 [info] <0.2305.0>@miner_onion_server:decrypt:{372,13} could not decrypt packet received via radio: treating as a witness
2022-03-08 15:37:53.591 145 [info] <0.30655.3>@miner_onion_server:send_witness:{188,13} sending witness at RSSI: -116, Frequency: 868.5, SNR: -8.5
2022-03-08 15:37:53.718 145 [warning] <0.30655.3>@miner_onion_server:send_witness:{243,37} failed to dial challenger "/p2p/11WLxuNS8SBk1CBTsCJZUHyqVmMtBbKWWMRfMoZMzobo1cnq5VQ": not_found
2022-03-08 15:38:03.711 145 [info] <0.1716.0>@libp2p_group_gossip_server:drop_target:{612,5} dropping target for peerbook <0.1742.0>
2022-03-08 15:38:03.724 145 [info] <0.1716.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 1 new pids of type peerbook
2022-03-08 15:38:03.724 145 [info] <0.1716.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type seed
2022-03-08 15:38:03.725 145 [info] <0.1716.0>@libp2p_group_gossip_server:add_worker_pids:{641,27} 0 new pids of type inbound
2022-03-08 15:38:06.331 145 [info] <0.30679.3>@blockchain_sync_handler:handle_data:{176,29} sending blocks [1258188] to sync peer
2022-03-08 15:38:17.041 145 [warning] <0.2304.0>@miner_ebus:handle_message:{96,5} Unhandled dbus message "org.freedesktop.DBus.Introspectable.Introspect"
2022-03-08 15:38:18.439 145 [info] <0.2306.0>@miner_lora:handle_json_data:{564,5} got status #{<<"ackr">> => 100.0,<<"dwnb">> => 0,<<"rxfw">> => 1,<<"rxnb">> => 1,<<"rxok">> => 1,<<"temp">> => 30.0,<<"time">> => <<"2022-03-08 15:38:18 GMT">>,<<"txnb">> => 0}
2022-03-08 15:38:18.440 145 [info] <0.2306.0>@miner_lora:handle_json_data:{565,5} Gateway #gateway{mac=12273815315514654720,ip={127,0,0,1},port=50292,sent=0,received=7386,dropped=0,status=#{<<"ackr">> => 100.0,<<"dwnb">> => 0,<<"rxfw">> => 1,<<"rxnb">> => 1,<<"rxok">> => 1,<<"temp">> => 30.0,<<"time">> => <<"2022-03-08 15:38:18 GMT">>,<<"txnb">> => 0},rtt_samples=[],rtt=5000000}
2022-03-08 15:38:23.719 145 [info] <0.30655.3>@miner_onion_server:send_witness:{246,37} re-sending witness at RSSI: -116, Frequency: 868.5, SNR: -8.5
2022-03-08 15:38:25.153 145 [info] <0.30655.3>@miner_onion_server:send_witness:{251,37} successfully sent witness to challenger "/p2p/11WLxuNS8SBk1CBTsCJZUHyqVmMtBbKWWMRfMoZMzobo1cnq5VQ" with RSSI: -116, Frequency: 868.5, SNR: -8.5
2022-03-08 15:38:25.155 145 [info] <0.30696.3>@miner_poc_handler:handle_info:{74,5} client sending data: <<18,179,1,10,33,0,63,226,146,178,157,103,160,5,255,107,224,88,185,40,138,189,248,91,215,215,189,220,121,13,217,50,148,231,248,238,40,122,16,155,251,157,159,197,189,156,237,22,24,231,1,34,32,204,134,90,22,193,70,124,114,50,72,39,7,187,5,9,134,28,43,10,31,222,0,70,108,207,42,56,243,220,105,81,232,42,72,48,70,2,33,0,170,213,7,51,191,182,155,14,199,203,24,172,66,23,53,152,79,73,104,138,227,201,156,144,76,46,115,50,212,183,182,101,2,33,0,206,27,107,11,171,2,57,63,62,241,175,147,154,73,112,198,2,238,69,182,38,75,141,142,195,115,156,121,225,215,248,165,53,0,0,8,193,61,0,32,89,68,64,7,74,9,83,70,49,50,66,87,49,50,53>>
2022-03-08 15:38:26.218 145 [info] <0.1717.0>@libp2p_peerbook:notify_peers:{749,5} gossiping out 795 notify peers
2022-03-08 15:38:32.847 145 [info] <0.30710.3>@blockchain_sync_handler:handle_data:{176,29} sending blocks [1258188] to sync peer

[ke6jjj]

All:

This issue was most likely due to the hidden denylist that was in place at the time. I think we can close it now, as the denylist itself is public and almost certainly explains what was observed.