traefik parser

[buixor]

I am running a couple of services using docker using traefik as a reverse proxy. I would like to enable an access log for traefik and have crowdsec check those whic hwould help secure all services behind the reverse proxy.

afaik traefik can write the access logs in json or in CLF format

(x-post from crowdsecurity/crowdsec#343)

[ovizii]

traefik can reverse proxy http, tcp and udp. I only use it for http traffic though.

here is a sample anonymized log file for an access log in CLF format:

79.231.111.111 - - [01/Nov/2020:16:08:14 +0000] "PUT /updater/check?download=0&X-Plex-Product=Plex%20Web&X-Plex-Version=4.43.4&X-Plex-Client-Identifier=xyz&X-Plex-Platform=Chrome&X-Plex-Platform-Version=86.0&X-Plex-Sync-Version=2&X-Plex-Features=external-media%2Cindirect-media&X-Plex-Model=bundled&X-Plex-Device=Windows&X-Plex-Device-Name=Chrome&X-Plex-Device-Screen-Resolution=1920x938%2C1920x1080&X-Plex-Token=abc&X-Plex-Language=en-GB HTTP/2.0" 200 0 "-" "-" 4397 "plex@docker" "http://172.29.0.5:32400" 210ms
79.231.111.111 - - [01/Nov/2020:16:08:15 +0000] "GET /web/favicon.ico HTTP/2.0" 200 5430 "-" "-" 4403 "plex@docker" "http://172.29.0.5:32400" 0ms
79.231.111.111 - - [01/Nov/2020:16:08:15 +0000] "GET /api/settings/public HTTP/2.0" 200 533 "-" "-" 4404 "portainer@docker" "http://172.29.0.6:9000" 0ms
79.231.111.111 - - [01/Nov/2020:16:08:15 +0000] "GET /photo/:/transcode?url=https%3A%2F%2Fplex.tv%2Fusers%2F1ecab781cb922f98%2Favatar%3Fc%3D1585328998%26size%3D160&width=160&height=160&minSize=1&X-Plex-Token=xyz HTTP/2.0" 200 6457 "-" "-" 4401 "plex@docker" "http://172.29.0.5:32400" 209ms
79.231.111.111 - - [01/Nov/2020:16:08:15 +0000] "GET /photo/:/transcode?url=https%3A%2F%2Fplex.tv%2Fusers%2F1ecab781cb922f98%2Favatar%3Fc%3D1585328998%26size%3D240&width=240&height=240&minSize=1&blur=20&opacity=50&background=000000&X-Plex-Token=xyz HTTP/2.0" 200 1916 "-" "-" 4402 "plex@docker" "http://172.29.0.5:32400" 267ms
79.231.111.111 - - [01/Nov/2020:16:08:16 +0000] "GET /api/users/admin/check HTTP/2.0" 204 0 "-" "-" 4405 "portainer@docker" "http://172.29.0.6:9000" 0ms
79.231.111.111 - - [01/Nov/2020:16:08:14 +0000] "GET /:/websockets/notifications?X-Plex-Token=xyz HTTP/1.1" 0 0 "-" "-" 4399 "plex@docker" "http://172.29.0.5:32400" 1973ms
79.231.111.111 - ovi [01/Nov/2020:16:08:18 +0000] "GET /api/overview HTTP/2.0" 200 444 "-" "-" 4406 "api@docker" "-" 0ms
79.231.111.111 - ovi [01/Nov/2020:16:08:23 +0000] "GET /api/overview HTTP/2.0" 200 444 "-" "-" 4407 "api@docker" "-" 0ms

Here is one in json format:

{"ClientAddr":"79.231.111.111:49573","ClientHost":"79.231.111.111","ClientPort":"49573","ClientUsername":"ovi","DownstreamContentSize":444,"DownstreamStatus":200,"Duration":451431,"OriginContentSize":444,"OriginDuration":324552,"OriginStatus":200,"Overhead":126879,"RequestAddr":"traefik.local.domain.tld","RequestContentSize":0,"RequestCount":27,"RequestHost":"traefik.local.domain.tld","RequestMethod":"GET","RequestPath":"/api/overview","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"api@docker","StartLocal":"2020-11-01T17:16:13.265526314+01:00","StartUTC":"2020-11-01T16:16:13.265526314Z","entryPointName":"https","level":"info","msg":"","time":"2020-11-01T17:16:13+01:00"}
{"ClientAddr":"79.231.111.111:49573","ClientHost":"79.231.111.111","ClientPort":"49573","ClientUsername":"ovi","DownstreamContentSize":444,"DownstreamStatus":200,"Duration":477420,"OriginContentSize":444,"OriginDuration":330824,"OriginStatus":200,"Overhead":146596,"RequestAddr":"traefik.local.domain.tld","RequestContentSize":0,"RequestCount":28,"RequestHost":"traefik.local.domain.tld","RequestMethod":"GET","RequestPath":"/api/overview","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"api@docker","StartLocal":"2020-11-01T17:16:18.267053481+01:00","StartUTC":"2020-11-01T16:16:18.267053481Z","entryPointName":"https","level":"info","msg":"","time":"2020-11-01T17:16:18+01:00"}
{"ClientAddr":"79.231.111.111:49632","ClientHost":"79.231.111.111","ClientPort":"49632","ClientUsername":"-","DownstreamContentSize":0,"DownstreamStatus":0,"Duration":1168091349,"OriginContentSize":0,"OriginDuration":1168034892,"OriginStatus":0,"Overhead":56457,"RequestAddr":"plex.local.domain.tld","RequestContentSize":0,"RequestCount":52,"RequestHost":"plex.local.domain.tld","RequestMethod":"GET","RequestPath":"/:/websockets/notifications?X-Plex-Token=xyz","RequestPort":"-","RequestProtocol":"HTTP/1.1","RequestScheme":"https","RetryAttempts":0,"RouterName":"plex@docker","ServiceAddr":"172.29.0.5:32400","ServiceName":"plex@docker","ServiceURL":{"Scheme":"http","Opaque":"","User":null,"Host":"172.29.0.5:32400","Path":"","RawPath":"","ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2020-11-01T17:18:51.113204941+01:00","StartUTC":"2020-11-01T16:18:51.113204941Z","entryPointName":"https","level":"info","msg":"","time":"2020-11-01T17:18:52+01:00"}

[johackim]

Any news about this ?

[gmelodie]

I'll give this one a shot!

So as far as I understand this would require me to write a new pattern to filter logs (maybe two, including JSON). Should I create it only inside the parser .yaml file (under pattern_syntax) or do you think it's better to add it to config/patterns so it could be reused? In the latter case, which pattern file should I use to add it?

As a reference, I'm basing myself on this traefik log documentation:

<remote_IP_address> - <client_user_name_if_available> [<timestamp>] "<request_method> <request_path> <request_protocol>" <origin_server_HTTP_status> <origin_server_content_size> "<request_referrer>" "<request_user_agent>" <number_of_requests_received_since_Traefik_started> "<Traefik_router_name>" "<Traefik_server_URL>" <request_duration_in_ms>ms

[gmelodie]

Here's a sample parser I created (only for CLF for now).

debug: true
filter: "evt.Parsed.program startsWith 'traefik'"
onsuccess: next_stage
pattern_syntax:
  TRAEFIK_ROUTER: '%{USER}@%{URIHOST}'
  # for json just use TIMESTAMP_ISO8601
name: crowdsecurity/traefik-logs
description: "Parse Traefik access logs"
#
grok:
  pattern: '%{NGINXACCESS} %{NUMBER:number_of_requests_received_since_traefik_started} "(%{TRAEFIK_ROUTER:traefik_router_name}|\-)" "(%{URI:traefik_server_url}|\-)" %{NUMBER:request_duration_in_ms}ms'
  apply_on: message

[blotus]

Hello @gmelodie,

It's better to add the patterns inside the parser directly, it allows for easier updates (we can just update the hub instead of forcing the user to update their crowdsec)

[AlteredCoder]

Hello,

We have released the traefik parser in this PR : #296

Thanks to @gmelodie for its help on this :)