Server not responding to device sending FCtrl.ADR=true, should send LinkADRReq with faster DataRate & FCtrl.ADR=true

[buzzware]

I have a third party node which I can only configure its EUIs, OTAA/ABP and cause it to rejoin. It is in the same room, or within 50 meters of the Sensecap and TTS gateways.
It works fine on TTN/TTS, configured as AU915 sideband 2, LoRaWAN 1.0.3. It joins on SF12 then within a few messages it shifts to SF7 via ADR.
However on Helium it joins with SF7 then uplinks in SF12, which wastes energy at an unacceptable level.
Each uplink is normally repeated with a 3 second gap - perhaps a separate issue, that may be caused/made worse by SF12 and so I am focussing on fixing the SF first.
I have configured the profile for the device, with ADR and CF enabled, then caused it to rejoin. This doesn't change the behaviour.
I then tried decoding the Join-Accept packets, and found that TTS was specifying a CFList and Helium was not.
Then I tried looking into the router source, and found the following :

1. the file c_src/compile.sh line 20 contains "for REGION in US915 EU868 AS923 CN470; do" - should AU915 be there ?
2. the file src/lora/lorawan_mac_region.erl contains calls to mk_join_accept_cf_list for most other plans, but not AU915
3. are there other issues for Helium to fully support AU915 side band 2 ?

Please let me know if there is anything I can do to test changes. I don't have Erlang experience, but have deep experience in other more mainstream languages. This is critical for us to use Helium for our startup.

[buzzware]

Here is the console JSON converted to CSV with some extra columns
event-debug-7-trunc.csv

[buzzware]

This is a typical join sequence on TTS.

The join happens on SF12, and very soon the SF gets reduced via the "Link ADR" messages eg.
https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=YDycDSamAAAGA0EA%2FwHkA%2FJm&nwkskey=&appskey=

In reverse order :

[buzzware]

I'm making sense of this now.

On The Things Stack (TTS) the device is shifted to SF7 because it first asks for ADR in the first uplink FCtrl, then the server responds with a LinkADRReq DR=SF7 in the following downlink FOpts.
https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=YKLqDSalAAADUAD%2FAaYMaNs%3D&nwkskey=&appskey=

On Helium, I can't see the full packet to get FCtrl (#464), but I'm assuming it is the same as on TTS, note FCtrl.ADR = true
https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=gKLqDSaAAAAI6wTEv1TQtPOOH29vq1%2FqbgWbgZ9pzOE3LDChO2ds2VWw&nwkskey=&appskey=

In the downlink, Helium responds with FCtrl.ADR = false, FOpts = NULL
https://lorawan-packet-decoder-0ta6puiniaut.runkit.sh/?data=YAIAAEggAADzMag%2F&nwkskey=&appskey=

So Helium simply isn't fulfilling the request by the device to have its DR adjusted.
Not only that, I think the network setting FCtrl.ADR = false is actually telling the device to not do ADR.
The spec says "The ADR bit MAY be set and unset by the end-device or the Network on demand"
I believe the server should return the same ADR value (true) that the device sends, which TTS is doing.

Also see my decoding here https://docs.google.com/spreadsheets/d/11D7PxIv61O-DvIyuwxCzw75WMS4yijNYqSUFjJBME6A/edit#gid=0

[buzzware]

I should have highlighted ADR, not FCtrl

[buzzware]

This issue is business critical for me, our product release is largely waiting on it.
However, I've never written a line of Erlang, so I've provided the best material I can to assist.
I'll move onto other things for now, but will look out for any activity here.
Many thanks!

[mikev]

@buzzware - Check router_device_worker.erl:2027
We don't consider ADR at all if it's not enabled. ADR must be set in the device's profile.

I'm not sure how Console reconciles multiple labels with conflicting settings.
You might have ADR on in 1 place, then applied something else that turns it off.

Please provide your Device ID (ex. 3483050b-284d-47e2-8eda-e087fc03a753).
That will allow us to double-check your device profile settings. It will also allow us to initiate a trace.

In the spreadsheet row 14 says "uplink" but you've linked to a downlink packet. Can you double-check your packet decodes?

[buzzware]

@buzzware - Check router_device_worker.erl:2027 We don't consider ADR at all if it's not enabled. ADR must be set in the device's profile.

I'm not sure how Console reconciles multiple labels with conflicting settings. You might have ADR on in 1 place, then applied something else that turns it off.

I've only had one label applied at a time to my devices, and that profile has ADR on.
I think its a design flaw that every label affects the ADR setting either way - perhaps there should be a third state of "no affect" or something.

Please provide your Device ID (ex. 3483050b-284d-47e2-8eda-e087fc03a753). That will allow us to double-check your device profile settings. It will also allow us to initiate a trace.

The devices I was using are :

ec040720-d3ca-47ef-bf4c-03b48bb8a5fc
c31a7a5a-ce77-41d9-9644-6a6ec7229f55

In the spreadsheet row 14 says "uplink" but you've linked to a downlink packet. Can you double-check your packet decodes?

Yes, that was because I was decoding the payload, not the whole raw packet.
See point 3 here #464

Thanks for your attention to this.

[mikev]

The end-device on the Uplink sets the ADR bit. On the Downlink Ack we must (per the spec) also set the ADR bit to mirror the setting on the Uplink.

The spec allows the LNS leeway on setting the ADR bit, so this is not an entirely black & white issue fix. If the Network temporarily cannot estimate the best data rate owing to rapid changes of the radio channel, it clears the ADR bit and thus signals the end-device that the end-device should rely on it's own behavior heuristics, i.e. usually set maximum power. Our LNS code is probably not that smart yet.

@buzzware - Thanks for your thorough investigation and bug report

[buzzware]

@mikev thanks for your attention to this.
"maximum power" SF12 is not acceptable in most cases, being 25 times the airtime & battery consumption of SF7.
Are you saying your LNS won't do ADR only in special cases like
"rapid changes of the radio channel"?
With ADR enabled in the console and these changes just made, can I expect the router to send "LinkADRReq with faster DataRate & FCtrl.ADR=true" when my device is close to the hotspot?

[mikev]

@buzzware - Possibly you missed my code commit. The issue should be fixed now. The change committed will set the ADR bit in all cases now. Can you please test and verify? Thanks.

[buzzware]

Yes, it is now returning FCtrl.ADR=true, but that is the only change and there are two things here.
It isn't sending LinkADRReq like TTS does. I simply can't use Helium without this, because the battery will die 25 times faster than it should.
Does the current router ever send LinkADRReq? In what conditions? Could the problem be limited to AU915?
I am running a Sensecap hotspot.

[buzzware]

Update: Helium is sending LinkADRReq eventually. I need to test this more - it wanders around different SFs and doesn't get to SF7. I will test and raise another issue if required.