Quickstart example to create your parser development project.
On the punch you develop parsers as part of simple Makefile based project.
Together the punchlets parse, normalise and enrich the incoming logs, whatever their format is. The punch comes equipped with many parsers. Should you write your own, this project is for you.
The layout for punch parser is straightfowrward:
.
├── INFO
├── LICENSE
├── Makefile
├── metadata
│ └── metadata.yml
├── README.md
├── src
│ ├── main
│ │ └── punchlang
│ │ └── com
│ │ └── analytics
│ │ └── web
│ │ └── apache
│ │ ├── enrichment.punch
│ │ ├── http_codes.json
│ │ └── parser.punch
│ └── test
│ ├── kooker
│ │ └── punchline.yaml
│ ├── puncher
│ │ └── unit.json
│ └── resources
│ └── ecs-fields.csv
└── target
├── apache-2.0.0-artifact.zip
└── tmp
├── apache-2.0.0.zip
└── metadata.yml
Where:
- 'src/main/punchlang' is the root folder containing your parser, patterns and resource files.
- the recommended layout is to use folders com/analytics/web/apache that conform to your group and classifiers.
parser.punch
andenrichment.punch
are sample punchlets. They illustrates the basics. This is where you write the actual logic of your log parsing.groks/pattern.grok
is a sample grok pattern. The punch comes with many patterns directly loaded, but here is how you can add your own.http_codes.json
is a sample resource files. Check how it is used in the enrichment.punch punchlet.
Tests are provided in the 'src/test' folder. In there:
- puncher is the root folder for unit test executed using the punch provided puncher tool.
- resources and kooker are optional folder to provide extra test facilities.
You must conform to the src/main/punchlang and src/test/puncher root folders.
To test your parsers with your unit tests file simply run :
make test
It will launch a punch tool called puncher
in a docker container which is in charge of running your tests. Be sure to
have docker installed.
Once your parser is ready and packaged, you can simply refer to it in any punchline.
Beofr going to a real K8 platform, you can start by executing a foreground docker image of a punch pipeline that embeds your parser.
Remember a punchline is a log processing pipeline where you chain your parser. Check out the test/kooker/punchline.yaml example. A fake log is generated using the punch generator source, then the parser is invoked, and last a show sink is used to print the result.
Try it using:
make run
You will notice that punchline exits after the log has been fully processed.
First, deliver your parser to the target kubernetes cluster artifact registry. If you do not know what that is, we advise you to use kooker. Kooker is an small open source project to bootstrap a complete K8 cluster automatically equipped with the additional punch service, including the artifact server.
Once kooker is started; simply type in:
make upload
That will upload your parser to the artifact server running in Kooker.
From there you can You can apply your punchline, start a service and run the simulator by running :
make apply
When the simulator starts, you can exit this command using Ctrl+c
and run :
make logs
This should show your punchline logs.
To delete everything, simply run :
make delete
The Makefile is only here to help. The shell underlying shell commands are the following:
To upload your parser package to the artifact server:
curl -X POST "http://artifacts-server.kooker:4245/v1/artifacts/upload" -F artifact=@target/parsers-1.0.0-artifact.zip -F override=true
To start your punchline on kubernetes, de not forget to check the artifact service name in punchline.yaml
file before
executing this command :
kubectl apply -f punchline.yaml
In one window, view your punchline log :
kubectl logs -f --tail -1 -l punchline-name=test-parser-punchline