Non-string variables returned as "strings" - part deux!

[pb66]

#273

When using the feed api to return the last value it is returned as a string eg "2222", I have been using a bash script to interact with emoncms and found the api call results are causing issues.eg

RESULT=$(curl -s "http://myemoncmsserver.com/feed/value.json?id=123&apikey=abc..123")
echo $((RESULT+2))

returns

-bash: "2222": syntax error: operand expected (error token is ""2222"")

Where as

RESULT="2222"
echo $((RESULT+2))

returns 2224 (as does setting RESULT=2222 without any quotes), the "included" quotes prevent the result being seen as numeric and then the quotes are accepted as part of the string resulting in double quotes.

Since the api is calling the last VALUE shouldn't that be numeric (or null) not a string???

[chaveiro]

Should be fixed, see 4e2671e

[pb66]

No Change, not fixed, please re-open this issue!

Maybe due to the fix only being applied to MySQLTimeseries??

Querying PHPFina or PHPTimeseries or even mySQL results in a string containing double quotes to be returned as shown above, I've tested on at least one of every type of feed I have and none of them return actual numeric data.

(I have fully updated to v9.7.4 - master branch)

[chaveiro]

What api did you test? Please past the url.

[Paul-Reed]

Isn't it posted in post#1 ?
If I understand Paul's question correctly, he's asking why a string is returned instead of a number, when it's always going to be a numerical value (correct me Paul if I've got this wrong)

Paul

[pb66]

An example api would be "http://myemoncmsserver.com/feed/value.json?id=123&apikey=abc..123", I used it in the bash example in the first post to show how to experience the issue first hand.

Using that api returns quote "123.45" unquote (when the feed value is 123.45) ie a string of 8 chars including the leading and trailing double quotes. It's therefore not considered numeric and the returned value cannot be used directly in math functions without removing the quotes and casting to a numeric datatype. Since the quotes are just useless baggage and provide no benefit, I see no reason for the value to be a quoted string.

I think it should return
123.45
not
"123.45"

@chaveiro please check #273, you have previously corrected other api's to return numeric data types rather than a string representation of a value.

@Paul-Reed, no correction needed, you got it.

[Paul-Reed]

Having thought about this @pb66 ...
Isn't it because the same api returns non-numeric data too..
eg "http://myemoncmsserver.com/feed/list.json&apikey=abc..123"

There needs to be some consistency.

Paul

[pb66]

/feed/list is a different endpoint to /feed/value, one returns a list and the other a value.

This was fixed before, again check #273.

Having re read that thread again, it was actually Trystan that submited a fix since then it has been changed a couple of times, more recently by @chaveiro ac0407b#diff-a6d269aac8d41ae47febc430faf033ceL58 on the 2nd of June 16, a couple of weeks before I posted this issue, soon after I pulled in the changes.
There were obviously some changes in between these events as the method of casting to a float has altered, The original fix used "(float)" but at some point that changed to "1*" (multiply by 1 to cast to a number), this was removed again in June.

Could it be there is some confusion between returning a valid null value and a "null" string?

null doesn't need the quotes either, in fact I'm pretty sure it shouldn't have quotes.

for example the input api accepts null values eg 1,2,null,4,5. I do not think it would respond well to 1,2,"null",4,5 because that is not a null value, that value is a string of 4 letters n u l and l.

[pb66]

The api I was specifically using in this example was the "feed/value" api
https://github.com/emoncms/emoncms/blob/master/Modules/feed/feed_controller.php#L97

however the original discussion on this topic was surrounding the "feed/fetch" api
https://github.com/emoncms/emoncms/blob/master/Modules/feed/feed_controller.php#L49-L61

The "feed/value" should return a single value for a single feed (eg id=123), where as the "feed/fetch" returns CSV with a value for each of the specified feeds(eg ids=123,124,125). It should be in the form
[12.3,45.6,78.9]
which it did for a while, but instead now it currently returns
["12.3","45.6","78.9"]
which is not how I intended it to work.

Mid Dec '14, I submitted the fetch API and Trystan merged. #269
Late Dec '14 @chaveiro agreed the returned value should be a numeric datatype #273
Feb '15 Trystan submitted a fix
July '15 @chaveiro removed that fix and added "// null is a valid response" (to the value api too)
May '16 Trystan adds the "1*" to the fetch api
June '16 chavario removed the "1*" from the fetch api

So only the "fetch" api was ever actually fixed, but that was removed and reapplied and removed again.

I see the wish to return a null value is the cause here, but do not understand why a null value needs to be returned as a string, by very definition it is not null if it's "null". The need to return "false" if not found is not really required either, the latest value will only ever be null if the feed doesn't exist or it has never been written too, even a stale 2 years out of date value is still the "latest value" for that feed.

Any errors should really be returned as something other than a 200 response rather than dictating that all successful requests are returned incorrectly as strings.

Do we need another pair of feed api's "feed/numeric_fetch" and "feed/numeric_value" just to get just to get numeric data?

EDIT - Just for some history/context here, although I use these api's in various places, the "fetch" is specifically intended for emonhub to be able to "fetch" an array of variables and distribute as defined by the user, either transmit over RFM for emonGLCD or emonTx configuration etc, or publish the latest feed totals to MQTT or perhaps upload them to a solarpv website etc etc, The quotes add nothing but complication, the unquoted array can be read straight into Python (among other langs).

[chaveiro]

Ok, so the problem is that php is not a strong typed language, meaning that it 'loses' track if the variable is numeric or string. The change trystan made by casting to float or *1 will fix the issue with numbers but not when the value is NULL as it will be converted to 0. Null is valid for some engines for processlist logic.
That proposed change at #273 (with JSON_NUMERIC_CHECK) is tricky because it makes php guess by looking at variable content, i see one behaviour, the removing of leading zeros that could cause issues. But this could be tried out and see if there are complains.

On the other end the 'conservative' fix should be made at the engine files level, instead of the controller and care taken if the engine can return a null. This fix is simple we need to hint php (by casting all know numeric variables to int or float) so that the json_encode function knows how to represent it.
The tricky part is that we need to do the cast is many places.
Also note that it's not only the value but also other numeric fields that are currently being returned as strings on other APIs (the time, the feed sizes, some ids).

[chaveiro]

Lets see if this fixed anything on your side ? #568
(i use mysql only, cant test other engines)

[pb66]

Sorry I haven't had a chance to try this yet, but what you say makes sense.

If null values being cast to 0 is the issue would it be simpler to test for a null value (rather than test for a numeric value) and if false cast to numeric? that could possibly be done in just the one place before the api returns, although it is probably more correct to get the data in the correct datatypes upstream, it may leave holes for other data types to slip through the net, it not only needs implementing in multiple places but ongoing it needs maintaining in multiple places, plus any new engines or functions will need to comply too.

I hope to try this out as soon as i can, but I'm just cautious about doing "quick pulls" on a live server in case there are issues and no time to fix, i don't currently have a redundant "experimental" server instance, maybe at the weekend if not before.

[glynhudson]

I hope to try this out as soon as i can, but I'm just cautious about doing "quick pulls" on a live server in case there are issues and no time to fix, i don't currently have a redundant "experimental" server instance, maybe at the weekend if not before.

Have you tried the Emoncms docker instance? It can be fired up in minutes on any machine (assuming docker is installed) in minuites. It's great for testing as you can launch into a clean enviroment. I would love to get more people using it: https://github.com/emoncms/emoncms-docker

Another alternative (that I've started using recently) is cloud9 online IDE: https://community.openenergymonitor.org/t/emoncms-dev-in-a-browser-using-cloud9-ide/1756

Off-topic: I woul really like to create a testing enviroment as part of the emoncms-docker which automaticlly loads in a decent amount of test data and some complex dashboards at launch for testing and also a service to hit the API hard with a large volume of incoming data. However this is not the righ location to discuss this....