Something wrong with PUSH_UP, ACK

[JEBAHUMA]

After POC Offchain, the miners don't send the beacons through the middlmen.

[jibjab99]

How many miners are you running this with? I found the solution.

[JEBAHUMA]

what solution you found? i tried everything but without success.

[sblanchard]

@jibjab99 does your pr include the solution, I copied all your changes and reinstalled it but no luck.

[sblanchard]

possibly the way we identify beacons is not correct anymore
rxpk.get('size') == 52 and rxpk.get('datr') == 'SF9BW125'

[JEBAHUMA]

That part is the poc challenge, not for sending beacons data i guess

[sblanchard]

Name of helium packet forwarder commit "Update txpk_ack buffer size to accomodate new material"

[JEBAHUMA]

with your modification is working now?

[sblanchard]

How many miners are you running this with? I found the solution.

You have the solution?

[proxynetul]

someone fixed it ?

[simeononsecurity]

Seems we have a bunch of undesirables here. If you don't have something to contribute, keep off this issue.

@jibjab99 the fork you're requesting to merge isn't that updated from the original. This repo is almost entirely unmaintained at this point. If you want to test curiousforkers repo, switch to it entirely. If you want to test it inside of a docker container. use mine.
I've made modifications that randomize the SNR even better as well as a few other improvements.

I'm looking into the situation to see if I can solve it as well. However it's looking like it isn't so straight forward.

Also, if anyone has a good grasp on the gateway.rs changes and or is pretty good with rust. Message me.

[simeononsecurity]

https://github.com/simeononsecurity/helium-DIY-middleman

docker run \
    --net='bridge' \
    -p 1681:1681/udp \
    -e middleman_port=1681 \
    -e middleman_tx_adjust='--tx-adjust 0' \
    -e middleman_rx_adjust='--rx-adjust 0' \
    -e gateway_ID=AA555A0000000000 \
    -e server_address=localhost \
    -e serv_port_up=1680 \
    -e serv_port_down=1680 \
    --name diymiddleman -P -td simeononsecurity/helium_diy_middleman:latest

[JEBAHUMA]

i think i got where is the problem, but i don't have the skills to do it.
the problem i think is that the middlman don't send a non-empty txpk_ack (Like: txpk_ack={"error": "NONE"}) to the miner to confirm the succesufull transmition of the fake push_data.

[simeononsecurity]

i think i got where is the problem, but i don't have the skills to do it. the problem i think is that the middlman don't send a non-empty txpk_ack (Like: txpk_ack={"error": "NONE"}) to the miner to confirm the succesufull transmition of the fake push_data.

I've already got a fix for this untested if it solves the issue in it's entirety

It's in my github repo

    def get_stat(self):
        """
        return data, address where data is raw bytearray to send from socket and address is the destination (port, ip)
        if no message should be sent returns None, None
        :return:
        """
        payload = dict(
            stat=dict(
                time=dt.datetime.utcnow().isoformat()[:19] + " GMT",
                rxnb=self.rxnb,
                rxok=self.rxnb,
                rxfw=self.rxnb,
                txnb=self.txnb,
                dwnb=self.txnb,
                ackr=100.0
            )
        )
        return self.__get_PUSH_DATA__(payload)

    def get_rxpks(self, msg):
        new_rxpks = []

        # next iterate through each received packet to see if it is a repeat from cached
        for rx in msg['data']['rxpk']:

            # modify metadata as needed
            modified_rx = self.rxmodifier.modify_rxpk(rx, src_mac=msg['MAC'], dest_mac=self.mac)

            # add rx payload to array to be sent to miner
            new_rxpks.append(modified_rx)

        if not new_rxpks:
            return None, None
        payload = dict(rxpk=new_rxpks)

        self.rxnb += len(new_rxpks)
        status, message = self.__get_PUSH_DATA__(payload)
        if status == 0:
            self.logger.debug(f"sending PUSH_DATA with {len(new_rxpks)} packets from vGW:{self.mac[-8:]} to miner {(self.server_address, self.port_up)}")
            return None, None
        else:
            self.logger.error(f"error sending PUSH_DATA to miner {(self.server_address, self.port_up)}: {message}")
            return message, None
            
    def __get_PUSH_DATA__(self, payload):
        """
        Sends PUSH_DATA message to miner with payload contents
        :param payload: raw payload
        :return: (status, message) where status is 0 for success and 1 for failure, and message is None if successful, error message otherwise
        """
        try:
            top = dict(
                _NAME_=MsgPushData.NAME,
                identifier=MsgPushData.IDENT,
                ver=2,
                token=random.randint(0, 2**16-1),
                MAC=self.mac,
                data=payload
            )
            payload_raw = encode_message(top)
            # send the message
            # ...
            # return success status and None for message
            return (0, None)
        except Exception as e:
            # return failure status and error message
            return (1, str(e))
    ```

[JEBAHUMA]

get [ERROR ] Error sending PULL_DATA message: 'VirtualGateway' object has no attribute 'socket'
comes from vgateway.py

[simeononsecurity]

Appologies. I see the error now.
I'm unable to do the level of testing that I need for this currently. (I'm no where near the hardware)
I'm pushing an update now. Remove the docker container and it's image and try again in about 15 minutes or whenever this automated build is done
https://github.com/simeononsecurity/helium-DIY-middleman/actions/runs/4159374466/jobs/7195372510

[JEBAHUMA]

no worries, i'm not using a docker for the moment to do fast testing, when it's solved i will run a docker then.

[simeononsecurity]

Roger. Well the code is updated. Just gotta pull it from the repo then.
Also fyi, docker is just way better. You should give it a try.

[simeononsecurity]

I reverted the changes. Ended up causing more problems than fixing.

[JEBAHUMA]

I see, with last update (before the reveral) the Error sending PULL_DATA message: 'VirtualGateway' object has no attribute 'socket' still there.

[simeononsecurity]

So if anyone has any bright ideas I can turn it into the code. I'm not sure I have a grasp on it just yet.

The middle man software currently generates a fake tx ack. I was led to believe that is the issue but I'm starting to think that isn't so.

[JEBAHUMA]

is generating a empty tx ack? is has to be a non empty ack if not, will not work i guess.

[simeononsecurity]

is generating a empty tx ack? is has to be a non empty ack if not, will not work i guess.

I beleive it omits it even if there are errors.
This is the documentation for how semtech handles it https://github.com/Lora-net/packet_forwarder/blob/d0226eae6e7b6bbaec6117d0d2372bf17819c438/PROTOCOL.TXT#L404

[JEBAHUMA]

i see, but i don't see where is the code when is pushing the fake tx ack to the miners when the middleman receive a downstream from the miners

[simeononsecurity]

I've looked into this further. It wasn't the tx_ack that was being faked.
I was able to increase verbosity quite a bit. However, fixing this is going to require a level of refactoring I do not have the skills or time for. I added some comments and some new debugging output in my repo. Feel free to take a look. But I'm not gonna spend anymore time trying to fixing it.
https://github.com/simeononsecurity/helium-DIY-middleman

[JEBAHUMA]

@simeononsecurity thank you.

[simeononsecurity]

To justify the 10 plus hours I spent reversing this repo, I went back and added a bunch of useful comments explaining what things do in my repo. Hopefully this makes it easier for the next guy. But definitely over my head.

Just updated. https://github.com/simeononsecurity/helium-DIY-middleman

[jibjab99]

"It wasn't the tx_ack that was being faked."

It's not being faked, it's just not being sent.

[simeononsecurity]

"It wasn't the tx_ack that was being faked."

It's not being faked, it's just not being sent.

Reguardless of which. How have you determined this is the issue with the beacons?
No one can tell me. When reviewing the gateway-rs code it doesn't look like it even really matters if it is acknowledged or not.

[jibjab99]

It is 100% the ACK.

If you care to continue looking at it you can use util_tx_continuous to test sending data to the gateway.

You are just sending an empty TX_ACK of 12 bytes back to the miner.

[simeononsecurity]

@simeononsecurity

let's assume you have:

added vgateway for miner at 192.168.1.2 port: 1680(up)/1680(dn)

You receive a TX_ACK from the packet forwarder => ('192.168.1.2', 35380). withe Data: {'ver': 2, 'token': XXXXXX, 'identifier': 5, 'NAME': 'TX_ACK', 'UNIX_TS': XXXXXXXX.XXXXXX, 'MAC': 'XX:XX:XX:XX:XX:XX:XX:XX', 'data': {'txpk_ack': {'error': 'NONE', 'tmst': XXXXXXXX}}}

So the TX_ACK should go to the miner => ('192.168.1.2', 1680)

Isn't my best work, but I think this should work..

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr):
        # Extract the token from the message
        token = msg['token']
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Update the JSON data with the correct token
        json_data = None
        if len(msg) > 12:
            json_data = msg[12:]
            json_obj = json.loads(json_data)
            json_obj['token'] = token
            json_data = json.dumps(json_obj).encode('utf-8')
        # Encode the message with the updated JSON data and send it back to all the virtual gateways
        for vgw in self.vgw.values():
            vgw_address = (vgw.ip, vgw.port)
            rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': vgw.mac, 'data': json_data})
            self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
            self.sock.sendto(rawmsg, vgw_address)
            # Check the error field in the JSON object to determine if the downlink request was accepted or rejected
            if json_data:
                json_obj = json.loads(json_data)
                error = json_obj.get('txpk_ack', {}).get('error', 'NONE')
                if error == 'NONE':
                    # Log a debug message indicating that the downlink request was accepted
                    self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")
                else:
                    # Log a debug message indicating that the downlink request was rejected
                    self.vgw_logger.debug(f"Downlink request rejected by gateway at {vgw_address}: {error}")
            else:
                # Log a debug message indicating that the downlink request was accepted
                self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")

It's updated on my dev branch

[simeononsecurity]

This is set up to forward to all the vgateways as well.

Also please confirm where the pull_ack and push_ack are supposed to go.
I'm assuming the vgateway as well? @powerthesa

[simeononsecurity]

https://github.com/simeononsecurity/helium-DIY-middleman/tree/dev

[JEBAHUMA]

@simeononsecurity

yes the pull_ack and push_ack suppose to go to mines as well, yes.

[simeononsecurity]

@simeononsecurity

yes the pull_ack and push_ack suppose to go to mines as well, yes.

Update available on my repo.

    # Handle PULL_ACK message
    def handle_PULL_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for vgw in self.vgateways_by_mac.values():
            vgw_address = (vgw.server_ip, vgw.port)
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PULL_ACK has been received
        self.vgw_logger.debug(f"PULL_ACK received from gateway at {msg['MAC']}")

    # Handle PUSH_ACK message
    def handle_PUSH_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for vgw in self.vgateways_by_mac.values():
            vgw_address = (vgw.server_ip, vgw.port)
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PUSH_ACK has been received
        self.vgw_logger.debug(f"PUSH_ACK received from packet forwarder at {msg.get('MAC', addr)}")
    ```

[simeononsecurity]

And I broke it again. hold on

[simeononsecurity]

@powerthesa

Ok try again

    # Handle TX_ACK message
    def handle_TX_ACK(self, msg, addr):
        # Extract the token from the message
        token = msg['token']
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Update the JSON data with the correct token
        json_data = None
        if len(msg) > 12:
            json_data = msg[12:]
            json_obj = json.loads(json_data)
            json_obj['token'] = token
            json_data = json.dumps(json_obj).encode('utf-8')
        # Encode the message with the updated JSON data and send it back to all the virtual gateways
        for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])
            rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac_address, 'data': json_data})
            self.vgw_logger.debug(f"Encoded Message: {rawmsg}")
            self.sock.sendto(rawmsg, vgw_address)
            # Check the error field in the JSON object to determine if the downlink request was accepted or rejected
            if json_data:
                json_obj = json.loads(json_data)
                error = json_obj.get('txpk_ack', {}).get('error', 'NONE')
                if error == 'NONE':
                    # Log a debug message indicating that the downlink request was accepted
                    self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")
                else:
                    # Log a debug message indicating that the downlink request was rejected
                    self.vgw_logger.debug(f"Downlink request rejected by gateway at {vgw_address}: {error}")
            else:
                # Log a debug message indicating that the downlink request was accepted
                self.vgw_logger.debug(f"Downlink request accepted by gateway at {vgw_address}")

    # Handle PULL_ACK message
    def handle_PULL_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for addr, vgw in self.vgateways_by_addr.items():
            vgw_address = (addr[0], addr[1])
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PULL_ACK has been received
        self.vgw_logger.debug(f"PULL_ACK received from gateway at {msg.get('MAC', addr)}")

    # Handle PUSH_ACK message
    def handle_PUSH_ACK(self, msg, addr):
        # Log the decoded message
        self.vgw_logger.debug(f"Decoded Message: {msg}")
        # Encode the message and send it back to all the virtual gateways
        rawmsg = messages.encode_message(msg)
        for addr, vgw in self.vgateways_by_addr.items():
            vgw_address = (addr[0], addr[1])
            self.sock.sendto(rawmsg, vgw_address)
        # Log a debug message indicating that a PUSH_ACK has been received
        self.vgw_logger.debug(f"PUSH_ACK received from packet forwarder at {msg.get('MAC', addr)}")

https://github.com/simeononsecurity/helium-DIY-middleman/tree/dev

[JEBAHUMA]

@simeononsecurity runing now, let see what happen when a tx_ack is received.

[simeononsecurity]

Take 231337 lol

Updated again. Caught another error. Fixed it.
All variables now resolve I believe.

Try again now @powerthesa

[JEBAHUMA]

@simeononsecurity
ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

[simeononsecurity]

@simeononsecurity
ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

Okay that's good. I can work with this I think. I'm going AFK for an hour or two and I will look at fixing this and let you know.

[simeononsecurity]

@simeononsecurity ok, get the tx_ack.

Error:

-   File "/home/middleman/gateways2miners.py", line 435, in <module>
-     main()
-   File "/home/middleman/gateways2miners.py", line 429, in main
-     gw2miner.run()
-   File "/home/middleman/gateways2miners.py", line 124, in run
-     self.handle_TX_ACK(msg, addr)
-   File "/home/middleman/gateways2miners.py", line 272, in handle_TX_ACK
-     rawmsg = messages.encode_message({'ver': 2, 'token': token, '_NAME_': 'TX_ACK', '_UNIX_TS_': time.time(), 'MAC': mac, 'data': json_data})
-   File "/home/middleman/src/messages.py", line 182, in encode_message
-     rawmsg = msg_obj.encode(message_object)
-  File "/home/middleman/src/messages.py", line 139, in encode
-    super().encode(message_object)
-   File "/home/middleman/src/messages.py", line 41, in encode
-     self.data = struct.pack("=BHB", message_object.get('ver', 2), message_object.get('token'), message_object.get('identifier'))
- struct.error: required argument is not an integer

Updated the code on the dev branch again for testing.
It's going to be difficult to get this working. Essentially I'm having to fake the TX_ACK as well.
The way we are handling tokens in this is completely random

token=random.randint(0, 2**16 - 1)

According to https://github.com/Lora-net/packet_forwarder/blob/master/PROTOCOL.TXT
The TX_ACK needs to have the same Token as the PULL_RESP.
I wasn't able to figure out how to do that exactly. All the handlers in this use a random token so.. So I'm also randomly assigning the token for TX_ACK.

[JEBAHUMA]

@simeononsecurity nop, nothing happen.
I'm done with that, i spent a lot of time trying to fix it. maybe i will have a look when i have more time.

[simeononsecurity]

Well at least the code is working ish.
What we need to do really is set it up to where the PULL_RESP doesn't include fake data and have it repond with the TX_ACK only to the appropriate device and not all devices. The TX_ACK is an acknowledgement. It shouldn't contain any actual data. It's basically just a handshake.

[simeononsecurity]

Updated the dev branch again. I need more testers.

[simeononsecurity]

Merged my dev into main. Best I can tell my code is working. YMMV.

https://github.com/simeononsecurity/helium-DIY-middleman/tree/master

[sblanchard]

I’ll give feedback on testing. Last version crashed without error in log and the service restarted every few packets (so it seemed)

[JEBAHUMA]

@simeononsecurity i just had some time to do some test. and first we have to solve the first problem i see. to go to the next

The

for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])

is a loop of all vgateways (ip, port) and is sending the same ack to all the miners instead of sending the ack to only to the miner form we get the ack.

if we solve this, i can handl to see the next issue we if we get any. i think we are so near to solve this.

[Joehoehoepi]

I will test the new revision and come back to you guys, thanks for trying to resolve this

[simeononsecurity]

@simeononsecurity i just had some time to do some test. and first we have to solve the first problem i see. to go to the next

The

for addr, vgw in self.vgateways_by_addr.items():
            mac_address = self.vgateways_by_mac[vgw.mac].mac
            vgw_address = (addr[0], addr[1])

is a loop of all vgateways (ip, port) and is sending the same ack to all the miners instead of sending the ack to only to the miner form we get the ack.

if we solve this, i can handl to see the next issue we if we get any. i think we are so near to solve this.

Good catch didn't remember to fix that. I've updated the dev branch of my repo.
Please confirm.

[simeononsecurity]

If anyone has gateway-rs logs available for the PULL_RESP - TX_ACK chain that went through the MM software please send them to me.

[simeononsecurity]

also please use -d for debug while executing so you see the debug statements before it errors out if at all.

[Joehoehoepi]

MM runs without any errors over here

[simeononsecurity]

You'll need to wait for a pull response to happen and it'll debug the tx_ack as part of that

[proxynetul]

@Joehoehoepi @JEBAHUMA any news guys ? is working for you ?

[sblanchard]

Only for witnessing but not beaconing

[proxynetul]

Beaconing is important

[Joehoehoepi]

same here no beacons only witnessing

[simeononsecurity]

@Joehoehoepi @JEBAHUMA any news guys ? is working for you ?

I've dropped interest in the fix. Currently my repo lays some of the groundwork for a future fix. But it'll require almost a complete rewrite of the software to get it working. The fix was beyond my skill. But it doesn't mean it won't work again eventually. It's just not worth beating my head against the desk for the coming months to get it working again. I already dropped like 40 hours into it.

[Joehoehoepi]

@simeononsecurity No progress for me, there is also a lut issue going on atm, but I don't get the tx power error so the tx_acks are not comming through. Strange thing is that when I leave out the mitm and just connect straight to "birdsound" ;) it is still not receiving a tx_ack or creating an tx power error. the problem could very well also be in the pktfwder. None the less thanks for the time you put in, I wish I was a better coder and that I could help you out more.