Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

caddy-logs parser not working, but apache2-logs parser works #9

Closed
jzemla opened this issue Dec 15, 2021 · 4 comments
Closed

caddy-logs parser not working, but apache2-logs parser works #9

jzemla opened this issue Dec 15, 2021 · 4 comments

Comments

@jzemla
Copy link

jzemla commented Dec 15, 2021 •
edited

Hello,

Environment:

crowdsec v1.2.1 (docker)
caddy v2.4.6 (docker xcaddy build includes:  caddy-l4 , format-encoder , realip , caddy2-proxyprotocol , caddy-crowdsec-bouncer/http@main , caddy-crowdsec-bouncer/layer4@main)

I'm having trouble getting this to parse my caddy access.log. I am using the suggested config from the example, but crowdsec is unable to parse the file. I apologize in advance for being a github/devops newbie -- if there is something I missed or can provide more insight into, please let me know!

Caddy - config.json:

"logging": {
        "logs": {
            "default": {
                "level": "DEBUG",
                "writer": {
                  "output": "stderr"
                }
            },
            "access": {
              "level": "DEBUG",
              "writer": {
                "output": "file",
                "filename": "/var/log/caddy/access.log"
              },
              "encoder": {
                "format": "formatted",
                "template": "{common_log} \"{request>headers>Referer>[0]}\" \"{request>headers>User-Agent>[0]}\""
              },
              "include": [
                "http.log.access.access"
              ]
          }
        }
    },

Failed grok parse via caddy-logs:

# crowdsec -dsn file:///var/log/caddy/access.log -type caddy -no-api -trace
---snip---
TRAC[15-12-2021 11:58:22] INPUT '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'
TRAC[15-12-2021 11:58:22] node stage : s00-raw, current stage : s00-raw
TRAC[15-12-2021 11:58:22] Processing node 0/12 -> sparkling-waterfall   node-name=sparkling-waterfall stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node                           id=sparkling-waterfall name=crowdsecurity/docker-logs stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=sparkling-waterfall name=crowdsecurity/docker-logs stage=s00-raw
TRAC[15-12-2021 11:58:22] node (sparkling-waterfall) ret : false        node-name=sparkling-waterfall stage=s00-raw
TRAC[15-12-2021 11:58:22] Processing node 1/12 -> dawn-feather          node-name=dawn-feather stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node                           id=dawn-feather name=crowdsecurity/syslog-logs stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=dawn-feather name=crowdsecurity/syslog-logs stage=s00-raw
TRAC[15-12-2021 11:58:22] node (dawn-feather) ret : false               node-name=dawn-feather stage=s00-raw
TRAC[15-12-2021 11:58:22] Processing node 2/12 -> little-hill           node-name=little-hill stage=s00-raw
TRAC[15-12-2021 11:58:22] Event entering node                           id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0                       id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] State after nodes : true                      id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] + Processing 4 statics                        id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Parsed[message] = '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'  id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Parsed[program] = 'caddy'                    id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Meta[datasource_path] = '/var/log/caddy/test.log'  id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] .Meta[datasource_type] = 'file'               id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:58:22] Event leaving node : ok                       id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] node is successful, check strategy
DEBU[15-12-2021 11:58:22] move Event from stage s00-raw to s01-parse    id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] Node successful, continue                     id=little-hill name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:58:22] node (little-hill) ret : true                 node-name=little-hill stage=s00-raw
DEBU[15-12-2021 11:58:22] node successful, stop end stage s00-raw       node-name=little-hill stage=s00-raw
TRAC[15-12-2021 11:58:22] node stage : s01-parse, current stage : s01-parse
TRAC[15-12-2021 11:58:22] Processing node 3/12 -> spring-water          node-name=spring-water stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=spring-water name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=spring-water name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (spring-water) ret : false               node-name=spring-water stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 4/12 -> patient-pond          node-name=patient-pond stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0                       id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter                    id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] ! No grok pattern : 0x0                       id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter                    id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern :  : 0xc0006a8f70     id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [common_log]
DEBU[15-12-2021 11:58:22] [common_log] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '%{NOT...' didn't return data on ''    id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false                     id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko                       id=ancient-shape name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22]       sub-node (ancient-shape) ret : false (strategy:)  id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter                    id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern :  : 0xc0004251f0     id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [request remote_addr]
DEBU[15-12-2021 11:58:22] [request remote_addr] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '%{IPO...' didn't return data on ''    id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false                     id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko                       id=black-dream name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22]       sub-node (black-dream) ret : false (strategy:)  id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Node has not filter, enter                    id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing grok pattern :  : 0xc000425840     id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] extract path [request headers User-Agent]
DEBU[15-12-2021 11:58:22] [request headers User-Agent] doesn't exist
DEBU[15-12-2021 11:58:22] + Grok '\["%{...' didn't return data on ''    id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false                     id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko                       id=wispy-dew name=child-child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22]       sub-node (wispy-dew) ret : false (strategy:)  id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false                     id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko                       id=solitary-snow name=child-crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22]       sub-node (solitary-snow) ret : false (strategy:next_stage)  id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] State after nodes : false                     id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko                       id=patient-pond name=crowdsecurity/caddy-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (patient-pond) ret : false               node-name=patient-pond stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 5/12 -> cold-cherry           node-name=cold-cherry stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=cold-cherry name=crowdsecurity/modsecurity stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=cold-cherry name=crowdsecurity/modsecurity stage=s01-parse
TRAC[15-12-2021 11:58:22] node (cold-cherry) ret : false                node-name=cold-cherry stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 6/12 -> hidden-snowflake      node-name=hidden-snowflake stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=hidden-snowflake name=crowdsecurity/nginx-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=hidden-snowflake name=crowdsecurity/nginx-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (hidden-snowflake) ret : false           node-name=hidden-snowflake stage=s01-parse
TRAC[15-12-2021 11:58:22] Processing node 7/12 -> young-waterfall       node-name=young-waterfall stage=s01-parse
TRAC[15-12-2021 11:58:22] Event entering node                           id=young-waterfall name=crowdsecurity/sshd-logs stage=s01-parse
DEBU[15-12-2021 11:58:22] Event leaving node : ko (failed filter)       id=young-waterfall name=crowdsecurity/sshd-logs stage=s01-parse
TRAC[15-12-2021 11:58:22] node (young-waterfall) ret : false            node-name=young-waterfall stage=s01-parse
DEBU[15-12-2021 11:58:22] Log didn't finish stage s01-parse
DEBU[15-12-2021 11:58:22] Discarding line {Type:0 ExpectMode:1 Whitelisted:false WhitelistReason: Stage:s01-parse Line:{Raw:176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" Src:/var/log/caddy/test.log Time:2021-12-15 11:58:22.200949223 -0700 MST m=+4.603803966 Labels:map[type:caddy] Process:true Module:file} Parsed:map[message:176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" program:caddy] Enriched:map[] Overflow:{Mapkey: BucketId: Whitelisted:false Reprocess:false Sources:map[] Alert:<nil> APIAlerts:[]} Time:2021-12-15 11:58:22.209799114 -0700 MST m=+4.612653893 StrTime: MarshaledTime: Process:false Meta:map[datasource_path:/var/log/caddy/test.log datasource_type:file]}
---snip---

I found that I can force crowdsec to use the apache2-logs parser by modifying /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml to:

filter: "evt.Parsed.program startsWith 'caddy'"

...which then gets me this...

Successful grok parse via apache2-logs:

# crowdsec -dsn file:///var/log/caddy/access.log -type caddy -no-api -trace
---snip---
TRAC[15-12-2021 11:49:28] INPUT '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'
TRAC[15-12-2021 11:49:28] node stage : s00-raw, current stage : s00-raw
TRAC[15-12-2021 11:49:28] Processing node 0/11 -> lingering-dew         node-name=lingering-dew stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node                           id=lingering-dew name=crowdsecurity/docker-logs stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ko (failed filter)       id=lingering-dew name=crowdsecurity/docker-logs stage=s00-raw
TRAC[15-12-2021 11:49:28] node (lingering-dew) ret : false              node-name=lingering-dew stage=s00-raw
TRAC[15-12-2021 11:49:28] Processing node 1/11 -> silent-sea            node-name=silent-sea stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node                           id=silent-sea name=crowdsecurity/syslog-logs stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ko (failed filter)       id=silent-sea name=crowdsecurity/syslog-logs stage=s00-raw
TRAC[15-12-2021 11:49:28] node (silent-sea) ret : false                 node-name=silent-sea stage=s00-raw
TRAC[15-12-2021 11:49:28] Processing node 2/11 -> polished-wood         node-name=polished-wood stage=s00-raw
TRAC[15-12-2021 11:49:28] Event entering node                           id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] ! No grok pattern : 0x0                       id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] State after nodes : true                      id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] + Processing 4 statics                        id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Parsed[message] = '176.53.221.38 - - [15/Dec/2021:17:39:52 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"'  id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Parsed[program] = 'caddy'                    id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Meta[datasource_path] = '/var/log/caddy/test.log'  id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] .Meta[datasource_type] = 'file'               id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
DEBU[15-12-2021 11:49:28] Event leaving node : ok                       id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] node is successful, check strategy
DEBU[15-12-2021 11:49:28] move Event from stage s00-raw to s01-parse    id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] Node successful, continue                     id=polished-wood name=crowdsecurity/non-syslog stage=s00-raw
TRAC[15-12-2021 11:49:28] node (polished-wood) ret : true               node-name=polished-wood stage=s00-raw
DEBU[15-12-2021 11:49:28] node successful, stop end stage s00-raw       node-name=polished-wood stage=s00-raw
TRAC[15-12-2021 11:49:28] node stage : s01-parse, current stage : s01-parse
TRAC[15-12-2021 11:49:28] Processing node 3/11 -> throbbing-field       node-name=throbbing-field stage=s01-parse
TRAC[15-12-2021 11:49:28] Event entering node                           id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] ! No grok pattern : 0x0                       id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Event entering node                           id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Node has not filter, enter                    id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:28] Processing grok pattern :  : 0xc000678000     id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28] line 45.90.62.143 - - [15/Dec/2021:17:39:54 +0000] "GET / HTTP/1.1" 308 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"  oneshot=/var/log/caddy/test.log type="file:///var/log/caddy/test.log"
DEBU[15-12-2021 11:49:28] + Grok '(%{IP...' returned 13 entries to merge in Parsed  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28]       .Parsed['response'] = '308'                  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28]       .Parsed['bytes'] = '0'                       id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28]       .Parsed['verb'] = 'GET'                      id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28]       .Parsed['timestamp'] = '15/Dec/2021:17:39:52 +0000'  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:28]       .Parsed['http_user_agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36'  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['target_fqdn'] = ''                  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['request'] = '/'                     id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['auth'] = '-'                        id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['httpversion'] = '1.1'               id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['referrer'] = '-'                    id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['clientip'] = '176.53.221.38'        id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['ident'] = '-'                       id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29]       .Parsed['rawrequest'] = ''                   id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] event against holder 0/28                     cfg=red-butterfly file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902
DEBU[15-12-2021 11:49:29] Event leaving node : ko (filter mismatch)     cfg=red-butterfly file=/etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml name=crowdsecurity/f5-big-ip-cve-2020-5902
TRAC[15-12-2021 11:49:29] event against holder 1/28                     cfg=lingering-shadow file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
DEBU[15-12-2021 11:49:29] .Meta[log_type] = 'http_access-log'           id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] setting target StrTime to 15/Dec/2021:17:39:52 +0000
DEBU[15-12-2021 11:49:29] evt.StrTime = '15/Dec/2021:17:39:52 +0000'    id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[service] = 'http'                       id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[source_ip] = '176.53.221.38'            id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[http_status] = '308'                    id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] .Meta[http_path] = '/'                        id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] State after nodes : true                      id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] ! No node statics                             id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] Event leaving node : ok                       id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node is successful, check strategy
DEBU[15-12-2021 11:49:29] move Event from stage s01-parse to s02-enrich  id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] Node successful, continue                     id=weathered-fire name=child-crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29]       sub-node (weathered-fire) ret : true (strategy:next_stage)  id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] child is success, OnSuccess=next_stage, skip  id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] State after nodes : true                      id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] ! No node statics                             id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
DEBU[15-12-2021 11:49:29] Event leaving node : ok                       id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node is successful, check strategy
DEBU[15-12-2021 11:49:29] node reached the last stage : s02-enrich      id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] Node successful, continue                     id=throbbing-field name=crowdsecurity/apache2-logs stage=s01-parse
TRAC[15-12-2021 11:49:29] node (throbbing-field) ret : true             node-name=throbbing-field stage=s01-parse
DEBU[15-12-2021 11:49:29] node successful, stop end stage s01-parse     node-name=throbbing-field stage=s01-parse
---snip---

Did I configure something incorrectly?
@jzemla
Copy link
Author

jzemla commented Dec 21, 2021

So, I am openly calling myself out on knowing just enough to be dangerous and not having a full understanding of what I'm doing. Hello, world!

In short, this is resolved by changing:

"encoder": {
                "format": "formatted",
                "template": "{common_log} \"{request>headers>Referer>[0]}\" \"{request>headers>User-Agent>[0]}\""
              },

to:

"encoder": {
              "format": "json"
            },

...and thus benefitting from caddy's structured log files which is the purpose of this bouncer to begin with.

@jzemla jzemla closed this as completed Dec 21, 2021
@hslatman
Copy link
Owner

hslatman commented Dec 21, 2021 •
edited

@jzemla: great that you found out yourself!

I should probably update the example for the logs in config.json. Back when I included it for the first time I had to output Caddy logs in the Apache format for CrowdSec to parse it. It seems support for the Caddy structured format was added to CrowdSec not too long ago, so it's nice that it now works out of the box 😄.

I've always considered the example for the logs as a kind of extra. It's not required to ingest the Caddy logs into CrowdSec to make the bouncer work, but it's a good thing to do, nonetheless.

Have opened #10 to track this. Your example will help me test this. Thanks!

@hslatman hslatman mentioned this issue Dec 21, 2021
Open
@jdeath
Copy link

jdeath commented Nov 20, 2022 •
edited

Sorry to open an old issue. Could anyone get the caddy-logs parser to work in 2.6.2? I also tried using a grok debugger to find what changed, but couldn't get it to work with both console and json log formats. I was able to get it to work by downloading caddy with the transform plugin and outputting in the common_log format and use the apache2 collection.

log {
	   format transform "{common_log}"
	}

Also, I had to change the apache2-logs.yaml file to look for the logs coming from caddy instead of apache (I use homeassistant, so needed to use the plugin name)
filter: "evt.Parsed.program startsWith 'addon_c80c7555_caddy-2'"

Direct caddy logs would be nicer, but this method works. The bouncer works fine!

@rserbitar
Copy link

rserbitar commented Apr 25, 2024 •
edited

So, what is the correct config file currently? And how can I test that?
Edit: I also succeeded with transformation to apache2 like jdeath did. Would be ncie if native caddy format could be suported again . . .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hslatman @rserbitar @jzemla @jdeath