Feed not updating once 'clear' has been used

[borpin]

I cleared the data in a feed and now the feed will not update with new values.

Server Information

Emoncms	Version	9.9.3
	Modules	Administration : App v1.1.1 : Backup v1.1.4 : Dashboard v1.3.0 : Device v1.1.0 : EventProcesses : Feed : Input : CoreProcess : Schedule : Time : User : Visualisation
	Git URL	https://github.com/emoncms/emoncms.git
	Git Branch	* master
Server	OS	Linux 4.9.0-6-amd64
	Host	DietPi-EmoncmsVM DietPi-EmoncmsVM (192.168.7.47)
	Date	2018-11-09 16:19:30 UTC
	Uptime	16:19:30 up 24 days, 21:35, 0 users, load average: 0.00, 0.00, 0.00
HTTP	Server	lighttpd/1.4.45 HTTP/1.1 CGI/1.1 80
MySQL	Version	5.5.5-10.1.26-MariaDB-0+deb9u1
	Host	localhost (127.0.0.1)
	Date	2018-11-09 16:19:30 (UTC 00:00‌​)
	Stats	Uptime: 1845036 Threads: 2 Questions: 1897363 Slow queries: 0 Opens: 41 Flush tables: 1 Open tables: 35 Queries per second avg: 1.028
Redis	Version	3.2.6
	Host	localhost:6379 (127.0.0.1)
	Size	6017 keys (1.81M)
	Uptime	21 days
MQTT Server	Version	Mosquitto
	Host	192.168.7.50:1883 (192.168.7.50)
Memory	RAM	Used: 32.46% Total: 492.5 MB Used: 159.86 MB Free: 332.64 MB
	Swap	Used: 0.00% Total: 1.52 GB Used: 0 B Free: 1.52 GB
Disk	Mount	Stats
	/	Used: 38.56% Total: 7.81 GB Used: 3.01 GB Free: 4.7 GB
PHP	Version	7.0.30-0+deb9u1 (Zend Version 3.0.0)
	Modules	apcu v5.1.8 : calendar v7.0.30-0+deb9u1 : cgi-fcgi : Core v7.0.30-0+deb9u1 : ctype v7.0.30-0+deb9u1 : curl v7.0.30-0+deb9u1 : date v7.0.30-0+deb9u1 : dom v20031129 : exif v7.0.30-0+deb9u1 : fileinfo v1.0.5 : filter v7.0.30-0+deb9u1 : ftp v7.0.30-0+deb9u1 : gd v7.0.30-0+deb9u1 : gettext v7.0.30-0+deb9u1 : hash v1.0 : iconv v7.0.30-0+deb9u1 : json v1.4.0 : libxml v7.0.30-0+deb9u1 : mbstring v7.0.30-0+deb9u1 : mcrypt v7.0.30-0+deb9u1 : mosquitto v0.4.0 : mysqli v7.0.30-0+deb9u1 : mysqlnd vmysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $ : openssl v7.0.30-0+deb9u1 : pcre v7.0.30-0+deb9u1 : PDO v7.0.30-0+deb9u1 : pdo_mysql v7.0.30-0+deb9u1 : Phar v2.0.2 : posix v7.0.30-0+deb9u1 : readline v7.0.30-0+deb9u1 : redis v3.1.6 : Reflection v7.0.30-0+deb9u1 : session v7.0.30-0+deb9u1 : shmop v7.0.30-0+deb9u1 : SimpleXML v7.0.30-0+deb9u1 : sockets v7.0.30-0+deb9u1 : SPL v7.0.30-0+deb9u1 : standard v7.0.30-0+deb9u1 : sysvmsg v7.0.30-0+deb9u1 : sysvsem v7.0.30-0+deb9u1 : sysvshm v7.0.30-0+deb9u1 : tokenizer v7.0.30-0+deb9u1 : wddx v7.0.30-0+deb9u1 : xml v7.0.30-0+deb9u1 : xmlreader v7.0.30-0+deb9u1 : xmlwriter v7.0.30-0+deb9u1 : xsl v7.0.30-0+deb9u1 : Zend OPcache v7.0.30-0+deb9u1 : zip v1.13.5 : zlib v7.0.30-0+deb9u1

Client Information

HTTP	Browser	Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36 Vivaldi/2.1.1337.36
Screen	Resolution	1920 x 1080
Window	Size	1482 x 877

[Paul-Reed]

What type of feed was it - phpfina... etc
Does the corresponding feed .meta file still exist?

[borpin]

PHPFINA.

Does the corresponding feed .meta file still exist?

How will I know which is which?

I needed to use a feed so ended up deleting the original completely.

I have noted some oddities in renaming feeds etc as well although it is too difficult to replicate or remember exactly what I was doing, but the same apparent steps do not always have the same result.

I'm still using the original interface.

[Paul-Reed]

How will I know which is which?

Because the data file is suffixed .dat and the meta file is suffixed .meta

So for example feed 22 comprises of 22.dat & 22.meta
The data file contains the actual datapoints, whilst the meta file contains information which allows the feed data to be processed correctly.

although it is too difficult to replicate or remember exactly what I was doing

Without some further information Brian it's going to be difficult to investigate the issue further, as I've tried clearing feeds here, and it appears to work fine (for me).
Is there anything further that you wish to add?

[borpin]

So I have recreated the issue, this is an MQTT value feed updated at 2 min interval.

The meta file simply has an x in it

Log extract (grep on =30)

2018-11-10 19:21:19.647|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541877679 feedtime=1541877679 value=56.6 arg=
2018-11-10 19:21:19.647|INFO|PHPFina.php|post() id=30 timestamp=1541877679 value=56.6 padding=
2018-11-10 19:23:15.689|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541877795 feedtime=1541877795 value=56.7 arg=
2018-11-10 19:23:15.689|INFO|PHPFina.php|post() id=30 timestamp=1541877795 value=56.7 padding=
2018-11-10 19:25:17.615|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541877917 feedtime=1541877917 value=56.9 arg=
2018-11-10 19:25:17.615|INFO|PHPFina.php|post() id=30 timestamp=1541877917 value=56.9 padding=
2018-11-10 19:25:36.808|INFO|PHPFina.php|get_data() feed=30 st=1541273138 end=1541877938 int=756 sk=1 lm=1 pts=854 st=1541775480
2018-11-10 19:25:42.208|INFO|PHPFina.php|get_data() feed=30 st=1541723907 end=1541877938 int=193 sk=1 lm=1 pts=854 st=1541775480
2018-11-10 19:25:58.900|INFO|PHPFina.php|get_data() feed=30 st=1541273160 end=1541877960 int=756 sk=1 lm=1 pts=854 st=1541775480
2018-11-10 19:26:20.603|INFO|feed_model.php|feed model: clear() feedid=30
2018-11-10 19:26:32.359|INFO|PHPFina.php|get_data() feed=30 st=1541394120 end=1541998920 int=756 sk=1 lm=1 pts=0 st=0
2018-11-10 19:27:01.565|INFO|PHPFina.php|get_data() feed=30 st=1541891680 end=1541998920 int=134 sk=1 lm=1 pts=0 st=0
2018-11-10 19:27:21.630|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541878041 feedtime=1541878041 value=56.9 arg=
2018-11-10 19:27:21.630|INFO|PHPFina.php|post() id=30 timestamp=1541878041 value=56.9 padding=
2018-11-10 19:27:34.277|INFO|PHPFina.php|get_data() feed=30 st=1541838060 end=1542052540 int=268 sk=1 lm=1 pts=855 st=0
2018-11-10 19:27:39.013|INFO|PHPFina.php|get_data() feed=30 st=1541993878 end=1542052540 int=120 sk=1 lm=1 pts=855 st=0
2018-11-10 19:29:11.614|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541878151 feedtime=1541878151 value=56.6 arg=
2018-11-10 19:29:11.615|INFO|PHPFina.php|post() id=30 timestamp=1541878151 value=56.6 padding=
2018-11-10 19:30:22.820|INFO|PHPFina.php|get_data() feed=30 st=1541791824 end=1541878224 int=120 sk=1 lm=1 pts=856 st=0
2018-11-10 19:30:36.619|INFO|PHPFina.php|get_data() feed=30 st=1541748624 end=1541921424 int=216 sk=1 lm=1 pts=856 st=0
2018-11-10 19:31:11.664|INFO|feed_model.php|insert_data() feedid=30 updatetime=1541878271 feedtime=1541878271 value=57.1 arg=
2018-11-10 19:31:11.664|INFO|PHPFina.php|post() id=30 timestamp=1541878271 value=57.1 padding=

[TrystanLea]

Spent all morning on this one, the issue was caused by caching of the feed meta data in a local variable in feedwriter, clearing the feed cleared the file on the disk but not the local variable cache. I have removed the local variable cache as it's not really needed. I've also added further error checking, in particular checking for attempts to write huge amount of padding which could cause the feedwriter to crash.

Now merged into master: #1122. Please restart feedwriter as well to test.

[borpin]

Please restart feedwriter as well to test.

Will do.

[borpin]

Seems to have fixed it. 😄