Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Nginx] Add benchmark template for error logs #8777

Closed
wants to merge 3 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions packages/nginx/_dev/benchmark/rally/error-benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
description: Benchmark 20000 nginx error logs
data_stream:
name: error
corpora:
generator:
total_events: 20000
template:
type: gotext
path: ./error-benchmark/template.ndjson
config:
path: ./error-benchmark/config.yml
fields:
path: ./error-benchmark/fields.yml
16 changes: 16 additions & 0 deletions packages/nginx/_dev/benchmark/rally/error-benchmark/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
fields:
- name: 'timestamp'
period: -24h # one day
- name: agent.id
value: "da6cb4c8-c84c-4c5f-97c7-f8586a098af4"
- name: cloud.region
enum: ["us-east-1", "us-east-2", "us-west-1", "us-west-2", "ap-south-1", "ap-northeast-3", "ap-northeast-2", "ap-southeast-1", "ap-southeast-2", "ap-northeast-1", "ca-central-1", "eu-central-1", "eu-west-1", "eu-west-2", "eu-west-3", "eu-north-1", "sa-east-1", "af-south-1", "ap-east-1", "ap-south-2", "ap-southeast-3", "eu-south-2", "eu-central-2", "me-south-1", "me-central-1"]
cardinality: 100
- name: host.name
cardinality: 100
- name: client.ip
cardinality: 100
- name: process.pid
cardinality: 65000
- name: process.thread.id
cardinality: 4
16 changes: 16 additions & 0 deletions packages/nginx/_dev/benchmark/rally/error-benchmark/fields.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
- name: timestamp
type: date
- name: client.ip
type: ip
- name: message
type: text
- name: host.name
type: keyword
- name: agent.id
type: keyword
- name: cloud.region
type: keyword
- name: process.pid
type: long
- name: process.thread.id
type: long
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{- $timestamp := generate "timestamp" }}
{{- $ip := generate "client.ip" }}
{{- $hostname := generate "host.name" }}
{{- $agentId := generate "agent.id" }}
{{- $processPid := generate "process.pid" }}
{{- $processThreadId := generate "process.thread.id" }}
{
"@timestamp": "{{ $timestamp.Format "2006-01-02T15:04:05.999999Z07:00" }}",
"log.file.path": "/var/log/nginx/error.log",
"data_stream": {
"namespace": "ep",
"type": "logs",
"dataset": "nginx.error"
},
"message": "{{$timestamp | date "2006/01/02 15:04:05"}} [error] {{$processPid}}#{{$processThreadId}}: *1 open() \"/usr/local/Cellar/nginx/1.10.2_1/html/favicon.ico\" failed (2: No such file or directory), client: {{$ip}}, server: localhost, request: \"GET /favicon.ico HTTP/1.1\", host: \"localhost:8080\", referrer: \"http://localhost:8080/\"",
"event": {
"dataset": "nginx.error",
"timezone": "CET"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using this to get the right timezone. But this should be dynamic to where the data is generated from to make sure it is always accurate.

My initial plan was to use timestamp.Zone and use the first return param but turns out it is not possible in templates to use functions with more then one return param.

@aspacca As this is a common problem, it would be nice if for example timezone as variable would always be available? What other solutions are there?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can use $timestamp.Format "MST"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aliabbas-elastic Could you try this out in your PR?

},
"host": {
"name": "{{$hostname}}"
},
"agent": {
"id": "{{$agentId}}",
"name": "{{$hostname}}",
"type": "filebeat",
"version": "8.8.0",
"ephemeral_id": "{{$agentId}}"
},
"tags": [
"preserve_original_event"
]
}