Unable to query data from elasticsearch

[kinzhong]

Describe the bug
Hi, I am trying to follow the tutorial from the documentation hub using an ELK stack. However, I am getting a KestrelSyntaxError when querying. I tried it with Python 3.6 and 3.9; both have the same error results.

Details of the bug

• What is the hunt flow/script you are executing?
  Hunt flow from the tutorial.
• What is the command that failed?

var = GET process FROM stixshifter://host101

• What is the error message?

[ERROR] KestrelSyntaxError: invalid token "" at line 1 column 24. rewrite the failed statement.

To Reproduce
Steps to reproduce the behavior:

1. Setup Symon & Elasticsearch
2. Create API key on Elasticsearch for access
3. Test Elasticsearch access using API key
4. Configure environment variables

$ export STIXSHIFTER_HOST101_CONNECTOR=elastic_ecs
$ export STIXSHIFTER_HOST101_CONNECTION='{"host":"REDACTED.elastic-cloud.com", "port":9243, "indices":"winlogbeat-7.14.0-2021.08.04-000001"}'
$ export STIXSHIFTER_HOST101_CONFIG='{"auth":{"id":"REDACTED", "api_key":"REDACTED"}}'

5. Test using stix-shifter:

$ stix-shifter transmit elastic_ecs '{"host":"REDACTED.elastic-cloud.com", "port":9243, "indices":"winlogbeat-7.14.0-2021.08.04-000001"}' '{"auth":{"id":"REDACTED", "api_key":"REDACTED"}}' ping

{

    "success": true,

    "data": "{\n  \"cluster_name\" : \"66a63ad60eae4e2b9fb38f524b8defcc\",\n  \"status\" : \"green\",\n  \"timed_out\" : false,\n  \"number_of_nodes\" : 3,\n  \"number_of_data_nodes\" : 2,\n  \"active_primary_shards\" : 86,\n  \"active_shards\" : 172,\n  \"relocating_shards\" : 0,\n  \"initializing_shards\" : 0,\n  \"unassigned_shards\" : 0,\n  \"delayed_unassigned_shards\" : 0,\n  \"number_of_pending_tasks\" : 0,\n  \"number_of_in_flight_fetch\" : 0,\n  \"task_max_waiting_in_queue_millis\" : 0,\n  \"active_shards_percent_as_number\" : 100.0\n}\n"

}

6. Run jupyter notebook with command

var = GET process FROM stixshifter://host101
[ERROR] KestrelSyntaxError: invalid token "" at line 1 column 24. rewrite the failed statement.

Expected behavior
Results from query

Environment (please complete the following information):

• OS: Ubuntu 20.04
• Python version: Python 3.9.5, Python 3.6.9
• Python install environment: Python virtual environment
• STIX-Shifter version: 3.5.0

[subbyte]

You may want to try with a WHERE clause to describe the processes you'd like to get. And for the first GET to a data source, we strongly recommend to add START/STOP. More information is in the syntax doc: https://kestrel.readthedocs.io/en/latest/language.html#get

[pcoccoli]

The WHERE clause is required. That's the part that gets translated to an elasticsearch query. A better error message would be nice.

[subbyte]

better syntax error message support in ced4da7 and released into v1.0.13

[kinzhong]

Thank you for the clarification! It works on my end.