FR: URL parameters from & to for time range

[jinnatar]

I would love to have deepdive links from Grafana pointing towards Echofish. Grafana will generate a URL similar to this:
index.php?from=1486289218479&to=1486302980904&r=syslog/archive/admin

What would be needed from Echofish is supporting from & to to match a time range.

[gadamo]

@Artanicus grafana not yet in my stack. That said, can you provide a more accurate description of what we are trying to achieve here?

My guess is that Grafana can generate custom links (pointing to echofish) for specific timeranges (from-to specified as unix timestamps). Based on this assumption:

• Is this link supposed to be a regular page launching in a new browser window? What would be the expected output (e.g. the standard HTML output, or JSON, or XML)?

• Is the generation of the links customisable from grafana, and in what ways? Can it be something like /index.php?r=syslog/archive/admin&Archive[customfrom]=20170124102001&Archive[customto]=20170124102059, or does it need to be from & to, strictly containing unix timestamps? Probably timestamps would be the way to go anyhow, just asking to learn what flexibility we are given here.

We are always interested in working well (or integrating) with other tools; Thank you for bringing this FR,

[jinnatar]

Grafana has an option called "Drilldown" which will essentially generate a clickable link from the time period shown in the graph. For example, I could add a drilldown link from a load average graph to go to Echofish so I can easily search for a potential reason for an anomaly.

There's a few options available for these right now:

• So whether it opens in a new tab or not is controllable but it doesn't get embedded in any case.
• Expected output is a normal Echofish page, html output. This would just be a way to "set the stage" already as part of URL parameters
• The generation is somewhat customizable, but not as much as I'd wish. For example, the from & to parameters are static since that's what Grafana itself uses for time range parameters. Thankfully they're names that make sense also in this context.

[gadamo]

@Artanicus This functionality has been implemented by @proditis with merge #58

Pull the latest echofish master branch and try the following link format:

• index.php?r=syslog/archive&from=1487073653000&to=1487073653000&divider=1000
• index.php?r=syslog/logs/admin&from=1487073653000&to=1487073653000&divider=1000

Because kibana's "timestamps" include milliseconds, we have introduced the divider parameter; thus, for kibana timestamps make sure you include divider=1000.

Based on your specs, this should integrate correctly. Please test with your setup and let us know.

[jinnatar]

So the URL gets formed correctly I think:
index.php?from=1486219028878&to=1486305829979&r=syslog/archive&divider=1000

Ends up after a long load being an Error 500 though:

2017/02/14 21:55:42 [error] [php] Trying to get property of non-object (/var/www/yii/framework/yiilite.php(864) : eval()'d code:1)
Stack trace:
#0 /var/www/yii/framework/zii/widgets/grid/CGridColumn.php(231): TbDataColumn->getDataCellContent()
#1 /var/www/yii/framework/zii/widgets/grid/CGridColumn.php(154): TbDataColumn->renderDataCellContent()
#2 /var/www/html/protected/extensions/booster/widgets/TbExtendedGridView.php(563): TbDataColumn->renderDataCell()
#3 /var/www/yii/framework/zii/widgets/grid/CGridView.php(578): TbExtendedGridView->renderTableRow()
#4 /var/www/yii/framework/zii/widgets/grid/CGridView.php(488): TbExtendedGridView->renderTableBody()
#5 /var/www/yii/framework/zii/widgets/CBaseListView.php(181): TbExtendedGridView->renderItems()
#6 unknown(0): TbExtendedGridView->renderSection()
#7 /var/www/yii/framework/zii/widgets/CBaseListView.php(164): preg_replace_callback()
#8 /var/www/html/protected/extensions/booster/widgets/TbExtendedGridView.php(240): TbExtendedGridView->renderContent()
#9 /var/www/yii/framework/zii/widgets/CBaseListView.php(149): TbExtendedGridView->renderContent()
#10 /var/www/yii/framework/yiilite.php(3526): TbExtendedGridView->run()
#11 /var/www/html/protected/modules/syslog/views/archive/admin.php(92): ArchiveController->widget()
#12 /var/www/yii/framework/yiilite.php(3493): require()
#13 /var/www/yii/framework/yiilite.php(3472): ArchiveController->renderInternal()
#14 /var/www/yii/framework/yiilite.php(3904): ArchiveController->renderFile()
#15 /var/www/yii/framework/yiilite.php(3872): ArchiveController->renderPartial()
#16 /var/www/html/protected/modules/syslog/controllers/ArchiveController.php(84): ArchiveController->render()
#17 unknown(0): ArchiveController->actionAdmin()
#18 /var/www/yii/framework/yiilite.php(4121): ReflectionMethod->invokeArgs()
#19 /var/www/yii/framework/yiilite.php(4138): CInlineAction->runWithParamsInternal()
#20 /var/www/yii/framework/yiilite.php(3652): CInlineAction->runWithParams()
#21 /var/www/yii/framework/yiilite.php(6895): ArchiveController->runAction()
#22 /var/www/yii/framework/yiilite.php(6904): CFilterChain->run()
#23 /var/www/yii/framework/yiilite.php(4030): CAccessControlFilter->filter()
#24 /var/www/yii/framework/yiilite.php(6937): ArchiveController->filterAccessControl()
#25 /var/www/yii/framework/yiilite.php(6892): CInlineFilter->filter()
#26 /var/www/yii/framework/yiilite.php(3642): CFilterChain->run()
#27 /var/www/yii/framework/yiilite.php(3627): ArchiveController->runActionWithFilters()
#28 /var/www/yii/framework/yiilite.php(1761): ArchiveController->run()
#29 /var/www/yii/framework/yiilite.php(1681): CWebApplication->runController()
#30 /var/www/yii/framework/yiilite.php(1202): CWebApplication->processRequest()
#31 /var/www/html/index.php(13): CWebApplication->run()
REQUEST_URI=/index.php?from=1486219028878&to=1486305829979&r=syslog/archive&divider=1000

The archive right now is ~1mil entries and that query would have matched a 24h span, so quite a lot. Same error happens with a 2h span though so might not be related to span size.

[gadamo]

@Artanicus I believe you reached some time-out (e.g. php max_execution_time). Beware that archive table is unindexed data, so this query will be pretty slow as your dataset grows.

[gadamo]

@Artanicus Did you try increasing max_execution_time in php.ini?

On another issue you mentioned your setup; On your setup, the file should be located in /etc/php/7.0/fpm/php.ini; I propose increasing max_execution_time (perhaps memory_limit too) -- make sure you restart php7.0-fpm service -- and see what happens.

I believe you may be able to speed-up your queries to archive by adding an index, e.g.:

MariaDB [ETS_echofish]> CREATE INDEX receivedts_idx ON archive (`received_ts`);
Query OK, 0 rows affected (0.04 sec)
Records: 0  Duplicates: 0  Warnings: 0

Please test and let us know as always.

[jinnatar]

With the added index and a 20 minute time window of archived data it took 33 seconds to load. Jumping to another page of the results takes another 30 seconds. But it did get the right data, so there's that!