"Failed to parse query" for blacklist rule when file contains more than 1024 entries

[tfriesen]

When trying to create a blacklist rule that parses a file that contains more than 1024 lines, I get the following error:

elastalert_error - {'message': 'Error running query: [\'Failed to parse query [winlog.event_data.md5:"65a27335241f56963655d472704cae8e" OR .... winlog.event_data.md5:"cf85c2ced0556ce7... (60500 characters removed)', 'traceback': ['Traceback (most recent call last):', ' File "/usr/local/lib/python3.6/site-packages/elastalert/elastalert.py", line 398, in get_hits', ' raise ElasticsearchException(errs)', 'elasticsearch.exceptions.ElasticsearchException: [\'Failed to parse query [winlog.event_data.md5:"65a2...

To reproduce:

1. Create a blacklist rule (probably works with other rules) that reads a file for blacklist terms
2. enter more than 1024 items into the blacklist file
3. Attempt to test rule via elastalert-test-rule --config config.yaml myrule.yaml

Is this a limitaiton with elastalert, or with one of the frameworks it depends on, or with Elasticsearch, or...?

[tfriesen]

Turns out it's a (default) limitation in ElasticSearch, the indices.query.bool.max_clause_count term. Increasing this would fix my problem, but is not recommended.

Best solution would be to rejigger the get_query() or enhance_filter() function to break up queries with more 1024 terms into multiple queries. Or maybe make a LongBlacklist rule class.

[rayaar]

Yes. Having the same issue with large blacklists.