Derive InfluxDB database and measurement from dev_id and signal data from all gateways #2

thiasB · 2019-11-25T09:35:39Z

fix for ERROR: 'collections.OrderedDict' object has no attribute 'iteritems'
optional: dynamic InfluxDB database and measurement derived from dev_id, e.g. when more than one device registered in TTN App
store LoRa signal quality data from all receiving gateways

amotl · 2019-11-29T17:14:53Z

Thanks for your contributions, @thiasB. Much appreciated!

When looking at the changes, two things come to my mind when operating this piece in a generic / arbitrary TTN environment.

What happens when the InfluxDB database and measurement name can't be derived from the dev_id like currently implemented within this PR? Maybe we should be more graceful here and use the implemented split-scheme just when activated through a specific command line option.
When enumerating all gateways seen by the device, we might want to turn the telemetry fields within the downstream message payload into some stable scheme, e.g. by keying them with the gtw_id. I propose using these telemetry field names:
- gw_{gtw_id}_rssi
- gw_{gtw_id}_snr
- gw_{gtw_id}_latitude
- gw_{gtw_id}_longitude
- gw_{gtw_id}_altitude
The gtw_id should probably be interpolated in lowercase letters.

What do you think about this?

thiasB · 2019-12-03T11:03:53Z

yes, if you want to keep the original generic behavior then make the "hiveeyes scheme" optional. feel free to implement :)

Geographical data is really not needed here. Usually they don't change over time. Let's not artificially blow up databases. Instead spreading factor (SF) and bandwidth (BW) could be candidates for consideration.

amotl · 2020-01-15T13:54:48Z

ttnlogger/ttn_logger.py


        # Convert all numeric values to floats.
-        data = convert_floats(data)
+        data = convert_floats(data, integers="gtw_count,sf,bw")


You should pass a list here, like integers=["gtw_count", "sf", "bw"].

thiasB · 2020-01-16T06:02:33Z

Will do, though it's working.

…rom MQTT topic' option

amotl · 2020-01-28T12:33:40Z

Thanks for addressing the generic aspects of ttnlogger by introducing respective command line arguments to control these details.

thiasB · 2020-01-28T12:43:37Z

You're welcome.
Squash into single commit?

amotl · 2020-01-28T12:50:55Z

ttnlogger/ttn_logger.py

+        for i in range(num_gtws):
+            gtw_id = ttn_message.metadata.gateways[i].gtw_id
+            print('Gateway ' + str(i+1) + ' ID          : ' + gtw_id)
+            key_rssi = 'gw_' + gtw_id + '_rssi'
+            key_snr  = 'gw_' + gtw_id + '_snr'
+            data[key_rssi] = int(ttn_message.metadata.gateways[i].rssi)
+            data[key_snr] = float(ttn_message.metadata.gateways[i].snr)
+
+        data['sf'] = int(ttn_message.metadata.data_rate.split('BW')[0][2:])
+        data['bw'] = int(ttn_message.metadata.data_rate.split('BW')[1])


I don't quite see through that yet.

Messages can be received from several gateways at once and each gateway puts the individual signal quality data (RSSI and SNR) into the final message. Spreading factor (sf) and bandwidth (bw) are determinated by the sending device and thus equals for every receiver.

And, on multiple receiving gateways there's no defined order of appearance in the TTN message. Thus, always picking gw0 will point to random gateways. That's why I'm now looping through and forwarding data from all gateways

On multiple receiving gateways there's no defined order of appearance in the TTN message.

That's sad. But we are keying by gtw_id now, right? This contains stable identifiers like eui-3338333018004700.

Spreading factor (sf) and bandwidth (bw) are determinated by the sending device and thus equals for every receiver.

So, the last two lines actually decode the

"data_rate": "SF7BW125"

right?

amotl · 2020-01-28T12:52:37Z

Geographical data is really not needed here. Usually they don't change over time. Let's not artificially blow up databases.

I believe it won't do any harm. Unfortunately, if we don't have it within the database, we can't visualize it.

thiasB · 2020-01-28T13:32:46Z

55aae7b is yet untested

amotl · 2020-01-28T19:11:57Z

ttnlogger/ttn_logger.py

+            if not self.nogeo:
+                key_lon  = 'gw_' + gtw_id + '_lon'
+                key_alt  = 'gw_' + gtw_id + '_alt'
+                data[key_lat] = float(ttn_message.metadata.gateways[i].latitude)


key_lat might be undefined here.

amotl · 2020-01-28T19:13:19Z

Squash into single commit?

Do you mean I should squash it when merging this pull request?

amotl · 2020-01-28T22:55:01Z

Thanks!

thiasB · 2020-01-28T23:40:50Z

wasn't done testing and will push some more fixes :)

Matthias Büchner and others added 2 commits November 25, 2019 10:34

Dictionaries do not have iteritem methods in 3.x.

14b1704

derive InfluxDB data base and measurement from dev_id

f52bf1e

thiasB changed the title ~~Dictionaries do not have iteritem methods in 3.x.~~ Derive InfluxDB database and measurement from dev_id Nov 25, 2019

Matthias Büchner and others added 6 commits November 26, 2019 09:57

simplify dev_id quartering

b437bd3

update documentation

09f1adb

close http session and delete InfluxDB object after sending data

c0e4cff

store LoRa signal data from all receiving gateways

5bc2b57

minor polishing

0c66990

Merge branch 'gateway_details'

bd1c619

thiasB changed the title ~~Derive InfluxDB database and measurement from dev_id~~ Derive InfluxDB database and measurement from dev_id and signal data from all gateways Nov 28, 2019

rename field names of LoRa signal data

f2b3dd0

add Spreading Factor and Bandwidth

3d5a1bb

amotl reviewed Jan 15, 2020

View reviewed changes

thiasB added 3 commits January 16, 2020 14:50

pass integer variables as a list

1794594

add /build to gitignore

828051b

(re)introduce database and measurement from cmd options. new 'split f…

8c438ff

…rom MQTT topic' option

thiasB force-pushed the master branch from 5e37450 to 726ec26 Compare January 28, 2020 12:34

update README

ab6af63

thiasB force-pushed the master branch from 726ec26 to ab6af63 Compare January 28, 2020 12:35

minor optimization

7162c17

amotl reviewed Jan 28, 2020

View reviewed changes

thiasB added 2 commits January 28, 2020 13:54

check for required options

c44502d

optional: disable saving geolocation of gateways

55aae7b

thiasB force-pushed the master branch from fd01499 to 55aae7b Compare January 28, 2020 13:30

bump version

30f7bc4

thiasB requested a review from amotl January 28, 2020 13:42

harmonize README

0717c39

amotl reviewed Jan 28, 2020

View reviewed changes

fix missing key for gateway geolocation data

902efac

amotl merged commit cd828ef into mqtt-tools:master Jan 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Derive InfluxDB database and measurement from dev_id and signal data from all gateways #2

Derive InfluxDB database and measurement from dev_id and signal data from all gateways #2

thiasB commented Nov 25, 2019 •

edited

amotl commented Nov 29, 2019 •

edited

thiasB commented Dec 3, 2019

amotl Jan 15, 2020

thiasB commented Jan 16, 2020

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020 •

edited

amotl Jan 28, 2020

thiasB Jan 28, 2020 •

edited

thiasB Jan 28, 2020 •

edited

amotl Jan 28, 2020

amotl Jan 28, 2020

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020

amotl Jan 28, 2020

amotl commented Jan 28, 2020

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020

Derive InfluxDB database and measurement from dev_id and signal data from all gateways #2

Derive InfluxDB database and measurement from dev_id and signal data from all gateways #2

Conversation

thiasB commented Nov 25, 2019 • edited

amotl commented Nov 29, 2019 • edited

thiasB commented Dec 3, 2019

amotl Jan 15, 2020

Choose a reason for hiding this comment

thiasB commented Jan 16, 2020

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020 • edited

amotl Jan 28, 2020

Choose a reason for hiding this comment

thiasB Jan 28, 2020 • edited

Choose a reason for hiding this comment

thiasB Jan 28, 2020 • edited

Choose a reason for hiding this comment

amotl Jan 28, 2020

Choose a reason for hiding this comment

amotl Jan 28, 2020

Choose a reason for hiding this comment

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020

amotl Jan 28, 2020

Choose a reason for hiding this comment

amotl commented Jan 28, 2020

amotl commented Jan 28, 2020

thiasB commented Jan 28, 2020

thiasB commented Nov 25, 2019 •

edited

amotl commented Nov 29, 2019 •

edited

thiasB commented Jan 28, 2020 •

edited

thiasB Jan 28, 2020 •

edited

thiasB Jan 28, 2020 •

edited