Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update MQTT.cpp #1904

Closed
wants to merge 3 commits into from
Closed

Update MQTT.cpp #1904

wants to merge 3 commits into from

Conversation

arduionoGP
Copy link
Contributor

// removing time and post_timestamp in the json seem to fix the transmission of json POSITION_APP messages to broker. (Don't know why it works but the protobuf in cleartext does not include time/timestamp either)

// removing time and post_timestamp in the json seem to fix the transmission of json POSITION_APP messages to broker. (Don't know why it works but the protobuf in cleartext does not include time/timestamp either)
@caveman99
Copy link
Sponsor Member

looks like it doesn't like the fixed32 type. time will only be included if the device has an actual GPS and timestamp needs to be enabled in the position flags.

@caveman99
Copy link
Sponsor Member

i'd rather prefer not to exclude these but fix the (potential) underlying problem in the json11 lib, which also lives in our github :-)

@github-actions
Copy link
Contributor

github-actions bot commented Nov 5, 2022

🤖 Pull request artifacts

file commit
pr1904-firmware-2.0.0.eec37e3.zip eec37e3

thebentern added a commit to meshtastic/artifacts that referenced this pull request Nov 5, 2022
@garthvh
Copy link
Member

garthvh commented Nov 6, 2022

Is this a manually set fixed position? The position packet is built from flags so those flags need to be enabled for them to show up in a packet, there may not be a time if there is not a GPS.

Copy link
Member

@garthvh garthvh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better to manage that these fields (and the other position flags including altitude) are optional and can be configured per device.

Copy link
Member

@garthvh garthvh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the field is just timestamp too now, the pos_ notation has been removed from the protobufs

@caveman99
Copy link
Sponsor Member

the field is just timestamp too now, the pos_ notation has been removed from the protobufs

pos_timestamp is just the name in the json, that should not trigger that bug. i still think it's the special datatype for these 2 fields that silently fails to convert into something JSON understands.

thebentern added a commit to meshtastic/artifacts that referenced this pull request Nov 6, 2022
@garthvh
Copy link
Member

garthvh commented Nov 8, 2022

Do you have a device with a GPS? Seems weird that your position packets don't have the time field. pos_timestamp should be removed but time should always be there if you have a GPS or NTP server connected

Copy link
Member

@garthvh garthvh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have two issues here, can you update this to remove pos_timestamp completely and un-comment out the time field?(this is an optional field that is seldomly going to be enabled)

The time field is potentially a separate issue related to setting fixed position that we will need to test through. It does seem like JSON11 does not love a fixed32 value of 0 for a date.

@arduionoGP
Copy link
Contributor Author

Sure, will try to work on it tomorrow. I was using a t-beam with GPS on.
My understanding is that you are using proto3 for the protobufs. Under proto3, optional items having default type values, like 0 for a fixed32 are not encoded into the protobuf. Those fields (like a fixed32 of 0) are then missing from what you feed to J11. library.

The other thing to consider is adding the originator device ID for the GPS data within the position protobuf intead of the envelope. It seems that if you have multiple devices sending positions via LORA to an MQTT connected device -- all those position packets get stripped out and re-eneveloped as coming from the MQTT connected device before being sent to the broker as JSON. This makes it impossible to do plotting/geofencing for multiple units on a mesh using MQTT data unless they all individually connect via MQTT and originate their own position data.

@arduionoGP
Copy link
Contributor Author

Sorry, but un-commenting the time, getting rid of the post-timestamp and updating everything else to the current repository 2.03 still results in no JSON going to the MQTT broker. Two t-beams with gps enabled on the mesh, one with mqtt and network enabled.

image

02:19:43 916 [Router] Incoming message was filtered 0x55c7312c
02:19:49 923 [Power] Battery: usbPower=1, isCharging=0, batMv=4166, batPct=100
02:19:54 928 [GPS] WANT GPS=1
02:19:54 928 [GPS] WANT GPS=0
02:19:54 928 [GPS] publishing pos@636db14b:2, hasVal=1, GPSlock=1
02:19:54 928 [GPS] New GPS pos@636db14b:3 lat=39.933067, lon=-86.285825, alt=258
, pdop=3.31, track=225.22, speed=0.00, sats=6
02:19:54 928 [GPS] onGPSChanged() pos@636db14b, time=1668133194, lat=399330668,
lon=-862858250, alt=258
02:19:54 928 [GPS] updatePosition LOCAL pos@636db14b, time=1668133194, latI=3993
30668, lonI=-862858250, alt=258
02:19:54 928 [GPS] Node status update: 2 online, 3 total
02:19:55 928 [PositionModule] Sending pos@636db14b:6 to mesh (wantReplies=0)
02:19:55 928 [PositionModule] cancelSending id=0x5cdc86e8, removed=0
02:19:55 928 [PositionModule] Providing time to mesh 1668133195
02:19:55 929 [PositionModule] Position reply: time=1668133195, latI=399330668, l
onI=--862858250
02:19:55 929 [PositionModule] Update DB node 0x55c7312c, rx_time=1668133195
02:19:55 929 [PositionModule] handleReceived(LOCAL) (id=0x5cdc86ea Fr0x2c To0xff
, WantAck0, HopLim3 Ch0x0 Portnum=3 rxtime=1668133195 priority=10)
02:19:55 929 [PositionModule] No modules interested in portnum=3, src=LOCAL
02:19:55 929 [PositionModule] Add packet record (id=0x5cdc86ea Fr0x2c To0xff, Wa
ntAck0, HopLim3 Ch0x0 Portnum=3 rxtime=1668133195 priority=10)
02:19:55 929 [PositionModule] Should encrypt MQTT?: 0
02:19:55 929 [PositionModule] publish msh/2/c/LongFast/!55c7312c, 71 bytes
02:19:55 929 [PositionModule] serialized json message: {"channel": 0, "from": 14
39117612, "id": 1557956330, "payload": {"altitude": 258, "latitude_i": 399330668
, "longitude_i": -862858250,
02:19:55 929 [PositionModule] JSON publish message to msh/2/json/LongFast/!55c73
12c, 232 bytes: {"channel": 0, "from": 1439117612, "id": 1557956330, "payload":
{"altitude": 258, "latitude_
02:19:55 929 [PositionModule] {"channel": 0, "from": 1439117612, "id": 155795633
0, "payload": {"altitude": 258 "latitude_i": 399330668, "longitude_i": -86285825
0, "time": 1668133195}, "sender": "!55c7312c", "timestamp": 1668133195, "to": -1
, "type": "position"}Expanding short PSK #1
02:19:55 929 [PositionModule] Using AES128 key!
02:19:55 929 [PositionModule] ESP32 crypt fr=55c7312c, num=5cdc86ea, numBytes=22
!
02:19:55 929 [PositionModule] enqueuing for send (id=0x5cdc86ea Fr0x2c To0xff, W
antAck0, HopLim3 Ch0x8 encrypted rxtime=1668133195 priority=10)
02:19:55 929 [PositionModule] txGood=13,rxGood=15,rxBad=0
02:19:55 929 [PositionModule] Using channel 0 (hash 0x8)
02:19:55 929 [PositionModule] Expanding short PSK #1
02:19:55 929 [PositionModule] Using AES128 key!
02:19:55 929 [PositionModule] ESP32 crypt fr=55c7312c, num=5cdc86ea, numBytes=22

---------------- but later on in the log "time" does show up in the log but still not in json at the broker

02:27:26 1380 [PositionModule] Update DB node 0x55c7312c, rx_time=1668133646
02:27:26 1380 [PositionModule] handleReceived(LOCAL) (id=0x5cdc86f0 Fr0x2c To0xf
f, WantAck0, HopLim3 Ch0x0 Portnum=3 rxtime=1668133646 priority=10)
02:27:26 1380 [PositionModule] No modules interested in portnum=3, src=LOCAL
02:27:26 1380 [PositionModule] Add packet record (id=0x5cdc86f0 Fr0x2c To0xff, W
antAck0, HopLim3 Ch0x0 Portnum=3 rxtime=1668133646 priority=10)
02:27:26 1380 [PositionModule] Should encrypt MQTT?: 0
02:27:26 1380 [PositionModule] publish msh/2/c/LongFast/!55c7312c, 71 bytes
02:27:26 1380 [PositionModule] serialized json message: {"channel": 0, "from": 1
439117612, "id": 1557956336, "payload": {"altitude": 266, "latitude_i": 39933107
5, "longitude_i": -862858838,
02:27:27 1380 [PositionModule] JSON publish message to msh/2/json/LongFast/!55c7
312c, 232 bytes: {"channel": 0, "from": 1439117612, "id": 1557956336, "payload":
{"altitude": 266, "latitude_
02:27:27 1380 [PositionModule] {"channel": 0, "from": 1439117612, "id": 15579563
36, "payload": {"altitude": 266 "latitude_i": 399331075, "longitude_i": -8628588
38, "time": 1668133646}, "sender": "!55c7312c", "timestamp": 1668133646, "to": -
1, "type": "position"}Expanding short PSK #1
02:27:27 1380 [PositionModule] Using AES128 key!
02:27:27 1380 [PositionModule] ESP32 crypt fr=55c7312c, num=5cdc86f0, numBytes=2
2!

@garthvh
Copy link
Member

garthvh commented Nov 18, 2022

Deleting the time field is not the right solution, we can merge it and test if just the pos_timestamp field that is never in the packet is removed.

@thebentern thebentern closed this Nov 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants