This repo contains information reguarding elasticsearch and the elasticstack when running on your local machine

In order to properly utilize the resources provided here you'll need to download elasticsearch, logstash, and kibana. You can find these online at the elasticsearch website.

• Elasticsearch: https://www.elastic.co/elasticsearch
• Kibana: https://www.elastic.co/kibana
• Logstash: https://www.elastic.co/logstash
• Additionally, you may want to download curl, a cli tool, which interacts with the elasticstack: https://curl.se/windows/

WARNING!!!
THIS DOCUMENT IS INTENDED FOR WINDOWS, COMMANDS HAVE ONLY BEEN TESTED ON WINDOWS COMMAND PROMPT, BE AWARE

To upload a log to elasticsearch via logstash:

• create a file in your logstash\config folder named logstash.conf
• configure logstash.conf
• open the terminal
• cd C:/path/to/logstash/bin
• logstash -f C:\path\to\logstash\config\logstash.conf <- Make sure you use backslashes, forwardslashes can be seen as 'end line' characters

Depending on what you want to upload, you will have to configure your logstash.conf file differently. There is an example logstash.conf file located in this repo which is designed for Azure DevOps build logs.

Curl Commands to interact with Elasticsearch

Before you run any curl commands:

Make sure you are in the proper folder in your terminal cd C:/path/to/curl Make sure elasticsearch is up and running. You can navigate cd C:/path/to/elasticsearch/bin and run elasticsearch.bat

Depending on your system setup, in order to interact with elasticsearch you will need to configure the .yml file. Go to elasticsearch/config/elasticsearch.yml and change the xpack.security.enabled to false

• To create a new index in your local elasticsearch session:

  curl -XPUT http://localhost:9200/[index-name-here]?pretty" <- if necessary you can remove the ?pretty, it's only here for formatting

• To delete an index:

  curl -XDELETE http://localhost:9200/[index-name-here]?pretty

• To view all of your indecies:

  curl -XGET http://localhost:9200/_cat/indices

• To view the contents of the whole index:

  curl -XGET "http://localhost:9200/[index-name-here]/_search?size=1000&pretty"<- size value is number of documents displayed

• To upload a single document:

  You don't need to create a new index before uploading a document, it'll do it for you, but make sure your file is in a json format and only has one json object curl -XPOST http://localhost:9200/[index-name-here]/_doc?pretty -H "Content-Type: application/json" -d @C:/path/to/file.json

WARNING!!!
UPLOADING DOCUMENTS WITH CURL IS VERY INCONSISTENT, IT'S HIGHLY RECCOMENDED THAT YOU USE THE LOGSTASH METHOD LOCATED ABOVE. IF YOU NEED TO USE THE CURL METHOD, MAKE SURE IT'S IN A JSON FORMAT. FOR AZURE DEVOPS BUILD LOGS THERE IS A PROVIDED FORMATTING PROGRAM NAMED bulk-conversion.py IN THIS REPO THAT WILL CHANGE TAKE THE LOGS AND FORMAT THEM FOR .JSON

• To upload multiple documents:

  You can use bulk-conversion.py to format azure devops logs for bulk upload curl -XPOST http://localhost:9200/[index-name-here]/_bulk?pretty --data-binary @C:/path/to/file.json -H "Content-Type: application/json"

• To search each document in an index, replace [field 1] and [field 2] (make sure to remove the brackets but keep all other formatting) and replace them with your search terms

  curl -XGET "http://localhost:9200/[index-name-here]/_search?pretty" -H "Content-Type: application/json" -d"{\"query\":{\"match\":{\"[field 1]\":\"[field 2]\"}}}"