Generic TCP: Additional Field "UserToken"?

[DottoreTozzi]

As mentioned here:
http://hobbybrauer.de/forum/viewtopic.php?p=224288#p224288

For the public server we are developing, an extra input field in the GenericTCP config section, representing a User Token in analogy to Ubidots would be nice-to-have.

The contents should be submitted within the JSON as an additional parameter in order to ensure backwards compatibility.

[universam1]

Sounds good to me!
Would it make sense to hard code the DNS already for better user convenience?

[DottoreTozzi]

ispindle.de and Port 9501 as defaults would be cool.
Later on I'm planning to semi-autoconfig the iSpindel by a URL shown by the server.
Good thing you insisted on keeping HTTP GET. :)

[DottoreTozzi]

We'd need the IP to remain configurable, though, for those who want to use a Raspi as a Relay, of course...

[universam1]

K cool, I am also thinking about implementing a conditional config fetch, lets say every n-hours. This won't cost as much battery lifetime as on every upload cycle and still provide remote configuration.

[DottoreTozzi]

I absolutely love this idea! That would be brilliant!�
It could be triggered by a special return character from the script, within the same TCP connection.
So, once every n timer intervals the iSpindel would actually listen before shutting down the connection.

[ckrack]

+1 for the token and yes, please keep the URL editable.
It'd be possible that users host this on their own server.

[ckrack]

For backwards compatibility on older firmware we can allow the token to be sent in the name field, too.

[DottoreTozzi]

Let's not touch the name field. People want to use it, especially in connection with Ubidots.
A separate field is perfectly backwards compatible with all versions of my local server, and if implemented the way Sam suggested (nothing sent if empty), would remain so even with any other homebrew (pun intended) solution.

[ckrack]

Ok. Is there any kind of state of the art method of authorization in TCP APIs?
Somethin like the Authorization Header in HTTP for Restul APIs..
If so, we should use this.

Don't get me wrong, it's just a for fun and hobby thing but IOT got the bad press about security issues already, so let's not add to this ;)

[DottoreTozzi]

Like I said before, there is no such thing as a TCP API unless it's being implemented on Application Level.
Let's discuss this via PN on the forum and not unneccessarily contribute to Sam's workload by bloating this thread here.
You'll receive a detailed reply from me within the hour. :)

[universam1]

If it comes to security, then the real step is to use SSL which again becomes really expensive in battery lifetime.
And as speaking about IoT without checking the key chain there is also very limit benefit of doing this.

Conclusion, if rock solid security is not achievable, why bother too much.

[DottoreTozzi]

Fully agreed. There's no sensitive data to be gathered, here, and the iSpindel can't be hijacked to become part of a bot net, since it only speaks and doesn't listen. Server side, I'll implement some counter measures against flooding and brute force. Definitely won't need SSL for fermentation data ;)

[universam1]

Bumping on to do list

[universam1]

firmware.zip

Can someone @DottoreTozzi @ckrack please try and give feedback? Thanks!

[jvels]

Hi

I have tried firmware, but it end with config portal:
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Handle root
*WM: Handle root
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Handle root
*WM: Handle root
*WM: Request redirected to captive portal
*WM: Handle root
*WM: G12A
*WM: -76
*WM: Sent config page
*WM: Request redirected to captive portal
*WM: Request redirected to captive portal
*WM: WiFi save
*WM: Parameter
*WM: name
*WM: velsdk002
*WM: Parameter
*WM: sleep
*WM: 15
*WM: Parameter
*WM: vfact
*WM: 191.80
*WM: Parameter
*WM:
*WM:
*WM: Parameter
*WM:
*WM:
*WM: Parameter
*WM: selAPI
*WM: 3
*WM: Parameter
*WM: token
*WM: PzMPHVxsq
*WM: Parameter
*WM: server
*WM: MYDOMAIN.com
*WM: Parameter
*WM: port
*WM: 80
*WM: Parameter
*WM: url
*WM: /beer/
*WM: Parameter
*WM:
*WM:
*WM: Parameter
WM: POLYN
WM: -0.00031tilt^2+0.557tilt-14.054
*WM: Sent wifi save page
*WM: Connecting to new AP
*WM: Connecting wifi with new parameters...
*WM: previous settings invalidated
*WM: After waiting...
*WM: 6.21
*WM: seconds
WM: Connection result:
WM: WL_CONNECTED
saving config...{"Name":"velsdk002","Token":"PzMPHVxsq","Sleep":15,"Server":"MYDOMAIN.com","API":3,"Port":80,"URL":"/beer/","Vfact":191.8,"SSID":"G12A","PSK":"MY_WIFI","POLY":"-0.00031tilt^2+0.557tilt-14.054","aX":-1513,"aY":97,"aZ":1657}saved successfully
Spl 0: 70.54
Spl 1: 70.54
Spl 2: 70.53
Spl 3: 70.50
Spl 4: 70.52
Spl 5: 70.48
Spl 6: 70.50

a: -15706 -6834 332 absTilt: 70.52 T: 38.43 V: 3.57 owT: 37.81 Gravity: 23.68
After waiting 0.00 s, result 3
192.168.1.44

calling DTHTTP
genericHTTP: posting
POST /beer/ HTTP/1.1
Host: MYDOMAIN.com
User-Agent: velsdk002
Connection: close
Content-Type: application/json
Content-Length: 119

{"name":"velsdk002","ID":"3109502","⸮G":"","angle":70.5174,"temperature":37.8125,"battery":3.566215,"gravity":23.68265}HTTP/1.1 200 OK
Date: Mon, 21 Aug 2017 17:54:19 GMT
Server: Apache
X-Powered-By: PHP/5.6.30
Vary: Accept-Encoding
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

1f
New record created successfully
0

Final-sleep: 1s; RT:90439
s$

FW 5.3.0
2.0.0(656edbf)
Worker run!
Boot-Mode: 5
mounting FS... mounted!
reading config file
opened config file

parsed json
applying offsets
parsed config:
{"Name":"velsdk002","Token":"PzMPHVxsq","Sleep":15,"Server":"MYDOMAIN.com","API":3,"Port":80,"URL":"/beer/","Vfact":191.8,"SSID":"G12A","PSK":"MY_WIFI","POLY":"-0.00031tilt^2+0.557tilt-14.054","aX":-1513,"aY":97,"aZ":1657}
woken from deepsleep, normal mode
Spl 0: 70.44
Spl 1: 70.45
Spl 2: 70.43
Spl 3: 70.48
Spl 4: 70.49
Spl 5: 70.46
Spl 6: 70.47

a: -15692 -6844 330 absTilt: 70.46 T: 38.28 V: 3.57 owT: 37.75 Gravity: 23.65
After waiting 1.50 s, result 3
192.168.1.44

calling DTHTTP

Exception (28):
epc1=0x4021b0f0 epc2=0x00000000 epc3=0x00000000 excvaddr=0x00ffffff depc=0x00000000

ctx: cont
sp: 3fff1360 end: 3fff1750 offset: 01a0

stack>>>
3fff1500: 3fff1558 3fff1540 3fff1614 40100688
3fff1510: 00ffffff 3fff1540 3ffea918 40202330
3fff1520: 00000050 3fff1540 3fff1670 4020c3f9
3fff1530: 3ffe9bc8 3fff1540 000000c8 00000034
3fff1540: 3fff1530 3fff1548 3fff1558 3ffea918
3fff1550: 00000002 3ffe8590 00000000 3fff1568
3fff1560: 00000002 3fff156c 3f004449 39303133
3fff1570: 00323035 bff89fd1 ec3f999a 4020db0c
3fff1580: 4020d390 3fff1660 3fff370c 00000000
3fff1590: 3fff34ac 3fff34b0 00000001 4020df8c
3fff15a0: 00000002 00000000 00000000 00000033
3fff15b0: 4020d390 00000001 3fff1660 00000001
3fff15c0: 3fff351c 3fff3520 00000001 4020df8c
3fff15d0: 00000002 00000000 3fff1660 401004d8
3fff15e0: 3fff192c 00000000 3fff164f 40215e10
3fff15f0: 3fff0478 00000000 3ffea338 40215e10
3fff1600: 00000030 0000000a 00000000 00000000
3fff1610: 00000000 00000000 00000000 00000000
3fff1620: 3fff34b4 0000000f 00000000 3fff22fc
3fff1630: 0000000f 00000000 00000001 4020bb5e
3fff1640: 002f727e 3fff066c 0000000f 402166ab
3fff1650: 3ffea338 00000000 00000001 00000024
3fff1660: 00000050 00000000 00000003 402190eb
3fff1670: 3ffe8590 3fff04d0 3fff16bc 03060050
3fff1680: 3fff0204 3fff347c 3fff370c 3ffe9a88
3fff1690: 00000000 00001388 00000003 00000000
3fff16a0: 00000000 00000000 3ffea338 40215e10
3fff16b0: 00000002 00000000 00000001 6565622f
3fff16c0: 3f002f72 3fff1f30 3fff066c 40201715
3fff16d0: 3ffea338 3fff1700 3fff1f30 33000024
3fff16e0: 35393031 3f003230 3fff066c 402018b0
3fff16f0: 3ffea338 3fff1f30 3fff066c 402194af
3fff1700: 3ffea358 2c01a8c0 feefeffe feefeffe
3fff1710: feefeffe feefeffe feefeffe feefeffe
3fff1720: feefeffe feefeffe feefeffe 3fff0724
3fff1730: 3fffdad0 00000000 3fff071c 402030a8
3fff1740: feefeffe feefeffe 3fff0730 40205290
<<<stack<<<

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16
tail 8
chksum 0x2d
csum 0x2d
v00000000
~ld

FW 5.3.0
2.0.0(656edbf)
Worker run!
Boot-Mode: 2

Double Reset detected
mounting FS... mounted!
reading config file
opened config file

parsed json
applying offsets
parsed config:
{"Name":"velsdk002","Token":"PzMPHVxsq","Sleep":15,"Server":"MYDOMAIN.com","API":3,"Port":80,"URL":"/beer/","Vfact":191.8,"SSID":"G12A","PSK":"MY_WIFI","POLY":"-0.00031tilt^2+0.557tilt-14.054","aX":-1513,"aY":97,"aZ":1657}
going to Config Mode
{l

[universam1]

thanks for testing! I unfortunately dont have the backend available to test against, so I wonder how to proceed here.

[DottoreTozzi]

On it. Previous update landed in my spam folder, unfortunately.

[DottoreTozzi]

Entering the Token parameter on the iSpindel portal worked, the value doesn't seem to be transmitted in the JSON though, as of yet:

waiting for connection... listening on port: 9501 ('192.168.178.56', 7971) received:'{' ('192.168.178.56', 7971) Input Str is now:{ ('192.168.178.56', 7971) received:'"name":"iSpindleTest01","ID":"263990","angle":85.6939,"temperature":27.5,"battery":4.285714,"gravity":31.40103}' ('192.168.178.56', 7971) Input Str is now:{"name":"iSpindleTest01","ID":"263990","angle":85.6939,"temperature":27.5,"battery":4.285714,"gravity":31.40103} ('192.168.178.56', 7971) iSpindleTest01 (ID:263990) : Data received OK. ('192.168.178.56', 7971) - closed connection (

[universam1]

which provider did you select? Do you have the terminal output of the ispindel handy?

[DottoreTozzi]

GenericTCP.

Just tried again, this is what the serial monitor says (Token corresponds to what I had entered, just some random string that I've pulled out of somewhere...).
It's not being sent however, as confirmed by serial output:

parsed json
offsets not available
parsed config:
{"Name":"iSpindleTest01","Token":"3f4897a52fa10562","Sleep":900,"Server":"192.168.178.43","API":6,"Port":9501,"URL":"","Vfact":191.8,"SSID":"TOZZINET","PSK":"xxxxxxxxxxxxxxxx","POLY":"-0.00031tilt^2+0.557tilt-14.054","aX":0,"aY":0,"aZ":0}
woken from deepsleep, normal mode
Spl 0: 87.04
Spl 1: 87.07
Spl 2: 87.06
Spl 3: 87.15
Spl 4: 87.08
Spl 5: 87.06
Spl 6: 87.04

a: 302 15428 740 absTilt: 87.06 T: 32.04 V: 4.33 owT: 29.62 Gravity: 32.09
After waiting 1.60 s, result 3
192.168.178.56

calling DTTCP
genericTCP: sending
server: 192.168.178.43 Port: 9501
{"name":"iSpindleTest01","ID":"263990","angle":87.06411,"temperature":29.625,"battery":4.327425,"gravity":32.09085}wait
�
Final-sleep: 900s; RT:3455

[universam1]

Thanks @DottoreTozzi @jvels for testing - in response I've completelly rewritten the upload process to have it more granular and flexible. From my testing the token is now successfully added to the json string.
Please check this out and report back.

[universam1]

Here @DottoreTozzi @jvels is the new compiled firmware. Thanks for testing!
firmware.zip

[DottoreTozzi]

Token is transmitted OK now, however the iSpindle's name is now missing from the JSON...

And while you're at it:
Changing the TCP port default value to 9501 would also make things easier for the end user.
Wouldn't want to open a separate issue for this one.

And: Thanks again for your hard work on this, highly appreciated!
I'm sure it will all be worth it in the long run.

[universam1]

Thanks for testing, I mixed it with the ID! Alright, the name is now transmitted as well:

calling iSPINDELde
{"name":"iSpindel000","ID":3958946,"token":"supersecrettoken","angle":74.91941,"temperature":25.1875,"battery":4.171429,"gravity":25.93611}
ERROR Sender: couldnt connect

Let me know if you need anything else!

In this case I've created a new preset for "iSpindel.de" for user convenience, predetermines the host name, port and access api. I hope it works out of the box, can't verify.

Feedback welcome!
firmware.zip

[ckrack]

If you use your hosts file and re-route ispindel.de to 127.0.0.1, you can test it.

How do we go about the preference of the name fields?
The workflow to add a spindle to the public-server is to generate a token and set a name to be displayed in the interface.
The name that is generated from the spindle-ui is not needed.
I can imagine people using the public-server more often than the spindle-ui, as you have to go into config mode before dropping it into your fermenter.
This would argument for prefering the name set in the public-server and omitting the one set in the spindle-ui.

[universam1]

@ckrack the iSpindel has no hosts file!

Regarding the name field, whatever you use to display in the web service is up to you, you can map to the ID or the name property, both are now transmitted.

[DottoreTozzi]

OK one final small quirk:
ID used to be transmitted in double-quotes, now these are ommitted.
If I fix this on my side (server), the new firmware would break existing installations...

@ckrack let's try to stay backward-compatible regarding the local server. Also, people will want to keep using the spindle name as a means to identify spindle and/or fermentation.
But let's not bother Sam with this.

[jvels]

@universam1 I can confirm it works :-)

[universam1]

Thanks @jvels ! 👍

You @DottoreTozzi are actually correct, while working on it it came to my mind that the ID is an integer serial number I've been transmitting as a String which kind of was nonsense... So I've changed it now to its native data type.
I'm not so much a fan of keeping the faults of the past just because of history. But if you really feel strongly about that matter let me know I will do you the favor 😉

[DottoreTozzi]

Thank you Sam!

Well, I am kinda torn by this, on the one hand I'd really like to keep backwards compatibility, on the other hand I absolutely agree that keeping incoherencies from the past and work around them is not the way to go in the long run.

I do not know how many people are actually using the local server at the moment, but I guess it's in the lowest 2-digit range.

So, now will probably be the best time to set things right, rather than later.

I will incorporate a fix and warn people that they will have to update the script and database in order to be able to use the new firmware and will make sure I get the fix out ASAP.
Need to do a major review and some nasty GIT merges over the next few days anyway, as well as put together a new Raspi image and create a repository for the public server, so you won't be bothered by that anymore.

Thanks again for your work, really nice job, and I guess we can close this issue now.

[universam1]

Awesome @DottoreTozzi great work! So I'll wait for the merge until you got the change in, so I can mention that required update in one go.
Looking forward!

[DottoreTozzi]

Actually, you might as well merge it right away... :)
As it turns out, older versions of my script do work fine after all, it's just the debug output that's screwed up.
By a lucky mistake of mine the script would send an NAK (which the firmware doesn't listen to), but else behave just normally and write the data anyway. That's a bug I'd almost like to call a feature...
Python 2.7 is apparently way more tolerant when it comes to these things than I had expected.

But I'll submit a pull request anyway. Quite a few changes I'd like to release now.

[DottoreTozzi]

Looks like in the latest version, iSpindel "Name" is no longer being transmitted.
This breaks compatibility.

On another note:
I have created a separate repository for the server, so, if you like, you can remove the tree "tools/genericTCP" now.

[universam1]

Well, it's in there https://github.com/universam1/iSpindel/blob/master/pio/src/iSpindel.cpp#L433 but lower case, might the be the issue?

I can't test right now as my iSpindel are currently busy producing beer ... 😄

Do you have a link I can mention?

[DottoreTozzi]

I assembled a new iSpindel last night and directly flashed 5.4.0 on it (newly downloaded firmware.zip from the link above, version 28.08.2017).
Token is transmitted OK, but "Name" field is missing.
Seems to be a separate issue though, as the version from August 16th worked fine.

The link to my local server repository is:
https://github.com/DottoreTozzi/iSpindel-TCP-Server

[universam1]

Could you give 5.5 a try and post the json output from console? I’ll try to reproduce ASAP.

[DottoreTozzi]

5.5 works. Sorry for the confusion...

[universam1]

Cool thanks for testing