Unreal Log parser project overview

A backend service for parsing and analizing the log files that are generated by the Unreal Engine build process.
With a Grafana dashboard to better visualize parsed issues their status and severity.

Setup.

Create a virtual environment for the project and install dependencies.

You can use the following command to create the virtual environment.

python3 -m venv .venv

Activate the .venv using:

source .venv/bin/activate

Install dependencies:

pip install -r requirements.txt

The project is based on docker-compose so make sure you have the appropriate software installed.
During the developement version of docker-compose 2.29.6 and Python 3.12.3 was used.

Starting the project:

To start the project use the following command:

docker-compose up --build

This will build all the images used by the app.
Note: after successfull image build, please give some time for the app to start properly before inserting logfiles. (around 1 minute should be sufficient)

Grafana will be available at http://localhost:3000
The API will be available at http://localhost:8000

Inserting the Logfiles

You can inset logfiles using the code provided in build.py by running:

python3 build.py --insert-logfile=<logfile_path>

You can insert many logfiles by providing paths to the files separated by comma ,

python3 build.py --insert-logfile=<logfile_path1>,<logfile_path2>

Each of the files must be MAX 10 MB of size, otherwise it will be rejected by the backend.

What happens after upload?

Upon uploading the files they will be parsed. Found Warnings, Errors and Tracebacks will be inserted to PostgreSQL database.
Whole unmodified lines from the file will be inserted to the Elasticsearch for future reference and access.

The found issues are deduplicated so by the message and only single instance of a particular error/warning is present in the database at a time.

Issues have generated id hash (log_entry_id) that is the same for entries in the Postgres, Elasticsearch and the results saved in the parsed file for ease of referencing the interesing lines.

Available API endpoints

You can test the API with standard tools like curl

POST	/logs	                            Process the logfile 

Issues (PostgreSQL)

GET	    /issues	                            Returns a list of issues with a open status (additionaly you can filter using ?status=open)
GET	    /issues/{issue_id}	                returns the id based on the issue
POST	/issues	                            Inserts an issue by hand
PATCH	/issues/{issue_id}	                Updates the issue status (eg. open -> closed)
DELETE	/issues/{issue_id}	                Deletes a issue

Logs (Elasticsearch)

GET	    /logs/{log_entry_id}	            Returns specific log entry based on provided hash
GET	    /logs/{log_entry_id}/line_number	Returns a original line number from the logfile
GET	    /logs/{log_entry_id}/datetime	    Returns a timestamp associated with the log

Visualization

There are two dashboards available that ware created in Grafana.

• Issues Dashboard is Dashboard visualizing the parsed issues - Errors, Warnings and Tracebacks.
• Logfile Browser is for browsing the inserted logfiles.

Testing available endpoints

You can test current endpoints using the curl command.
Creating new issues (issue_id will be returned as a response):

curl -X POST "http://localhost:8000/issues" \
     -H "Content-Type: application/json" \
     -d '{"message":"Error #1 message","category":"LogEngine","status":"open","severity":"Error"}'

curl -X POST "http://localhost:8000/issues" \
     -H "Content-Type: application/json" \
     -d '{"message":"Error #2 message","category":"API","status":"open","severity":"Error"}'

We expect to recieve the issue_id"

List of all issues:

curl "http://localhost:8000/issues"

List of filtered issues with status "open":

curl "http://localhost:8000/issues?status=open"

Requesting an defails about a specific issue based on the issue_id:

curl "http://localhost:8000/issues/<issue_id>"

Patching an existing issue based on issue_id eg. issue status from open to closed:

curl -X PATCH "http://localhost:8000/issues/<issue_id>" \
     -H "Content-Type: application/json" \
     -d '{"new_status":"closed"}'

Removing issue:

curl -X DELETE "http://localhost:8000/issues/<issue_id>"

Gathering full logline from Elasticsearch by the log_entry_id hash (eg. 94rYyV6QlU_uCCUV71s2):

curl "http://localhost:8000/logs/<log_entry_id>"

Getting date and time of log:

curl "http://localhost:8000/logs/<log_entry_id>/datetime"

Getting the line number from log:

curl "http://localhost:8000/logs/<log_entry_id>/line_number"