cron.php - Errno 8: Undefined index: logstash

[profilernz]

I get this error if I run cron.php from the terminal:

/var/www/411/bin/cron.php
[+] Scheduler: 1476998611
[+] Maintenance
[+] Search Health
except [NONE] Errno 8: "Undefined index: logstash" at [/var/www/411/phplib/Search/Elasticsearch.php:38] 0:[FOO\Elasticsearch_Search::getConfig() called at [/var/www/411/phplib/Search/Elasticsearch.php:66]] 1:[FOO\Elasticsearch_Search->isWorking() called at [/var/www/411/phplib/Scheduler.php:341]] 2:[FOO\Scheduler->health() called at [/var/www/411/phplib/Scheduler.php:146]] 3:[FOO\Scheduler->processSite() called at [/var/www/411/bin/cron.php:45]]
except [NONE] Errno 8: "Undefined index: logstash" at [/var/www/411/phplib/ESClient.php:50] 0:[FOO\ESClient::getClient() called at [/var/www/411/phplib/Search/Elasticsearch.php:67]] 1:[FOO\Elasticsearch_Search->isWorking() called at [/var/www/411/phplib/Scheduler.php:341]] 2:[FOO\Scheduler->health() called at [/var/www/411/phplib/Scheduler.php:146]] 3:[FOO\Scheduler->processSite() called at [/var/www/411/bin/cron.php:45]]
[+] Rollups
[+] Searches
[+] Reports
[+] Summary
[+] Autoclose
[+] Cleanup

My config.php has no mention of a logstash index (since we don't use that default).
The only ocurrence of "logstash" as an index seems to be on: phplib/Search/Logstash.php
Also, not sure if related, but currently alerts don't fire unless someone presses execute in the webui.
Any help much appreciated. Thanks!

[kiwiz]

Fixed! This should also resolve alerts not firing.

[profilernz]

Thanks!
But maybe there's been a misunderstanding?
We do use/need logstash searches, we just don't use the default "logstash-" index naming.

i.e. We use stats-, ossec- etc, but no logstash-:
'ossec' => [
'hosts' => ['http://localhost:9200'],
'index_hosts' => [],
'ssl_cert' => null,
'index' => 'ossec-',
'date_based' => false,
'date_field' => '@timestamp',
'src_url' => null,
],
'stats' => [
'hosts' => ['http://localhost:9200'],
'index_hosts' => [],
'ssl_cert' => null,
'index' => 'stats-',
'date_based' => false,
'date_field' => '@timestamp',
'src_url' => null,
],

It seems that with this commit, logstash searches have been disabled all together?
I'm running v1.0.1 with your phplib/Search/Elasticsearch.php from the commit above.
The error has disappeared after creating a new data.db, but now when we try to create a new search, we get "Null" instead of "Logstash" in the combo box.
Thanks again!

[kiwiz]

Ah. The process for doing this is terrible atm: #3.

The alternative is to use the ECL search type. Ex:
es:ossec somekey:somevalue.

[profilernz]

Thanks for the ECL workaround.
But back to the issue, I don't want or need to add another dropdown option in the combo box.
All I want is to add more indices (and none happen to be called logstash-).
I had followed the instructions here:
https://github.com/etsy/411/blob/master/docs/HowTo/NewESSource.md
Which lead me to believe all I had to do is add another logstash block and call "things" something else.
If there's no simple way to add a list of indices, maybe I should just go with 'index' => null, which should query them all?
Thanks again!

[profilernz]

I'm now getting:
/var/www/411/bin/worker.php
[+] Worker
[+] Worker: 1477430423
[+] Job count: 249
PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000]: General error: 1 near "LIMIT": syntax error' in /var/www/411/phplib/DB.php:78
Stack trace:
#0 /var/www/411/phplib/DB.php(78): PDO->prepare('UPDATE jobs S...')
#1 /var/www/411/phplib/Job.php(167): FOO\DB::query('UPDATE jobs S...', Array, 0)
#2 /var/www/411/phplib/Worker.php(77): FOO\JobFinder::getAndLock('1', 1477430423)
#3 /var/www/411/bin/worker.php(34): FOO\Worker->processSite(Object(FOO\Site), 1477430423)
#4 {main}
thrown in /var/www/411/phplib/DB.php on line 78
err [411_Worker] Worker error site:[1] ret:[65280]

[profilernz]

Found #23

[kiwiz]

Oh, I see. You can't currently provide a list of indices, but 'index' => null would work. Feel free to open an issue for specifying multiple indices in a config block and we'll take a look when we have time.

[profilernz]

Issue #23 suggests replacing sqlite with mysql, but i wanted to stick with sqlite.
For the record, I was able to fix the issue on Centos 7 by doing roughly the following:

yumdownloader --source sqlite
or
wget http://vault.centos.org/7.1.1503/updates/Source/SPackages/sqlite-3.7.17-6.el7_1.1.src.rpm
(the source rpm for sqlite-3.7.17-8 was not available at the time for some reason, so i grabbed version "-6" above via wget)

yum install rpm-build redhat-rpm-config ncurses-devel readline-devel autoconf
rpm -i sqlite-3.7.17-6.el7_1.1.src.rpm
nano ~/rpmbuild/SPECS/sqlite.spec

at line 128 add -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 to the CFLAGS, i.e.:

export CFLAGS="$RPM_OPT_FLAGS -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_ENABLE_COLUMN_METADATA=1 -DSQLITE_DISABLE_DIRSYNC=1 -DSQLITE_ENABLE_FTS3=3 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_SECURE_DELETE=1 -DSQLITE_ENABLE_UNLOCK_NOTIFY=1 -Wall -fno-strict-aliasing"

rpmbuild -ba ~/rpmbuild/SPECS/sqlite.spec
rpm -U --oldpackage ~/rpmbuild/RPMS/x86_64/sqlite-3.7.17-6.el7.centos.1.x86_64.rpm

[profilernz]

I can confirm the issue above also affects Fedora 24 (requiring a similar workaround), so Debian or Ubuntu might be an easier choice for those wanting to run 411 + sqlite.

[kiwiz]

I don't remember why I put that LIMIT 1 in the query, but it shouldn't be necessary as the UPDATE query is scoped to the PK... Fixed in e56a76f!

[profilernz]

Nice! :)

[kiwiz]

Is this all good?

[kiwiz]

Closing. Feel free to reopen if this is not resolved!