Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parsing of caddy log fails with cannot fetch headers from <nil>, no decisions for nikto test #2921

Open
arminus opened this issue Mar 29, 2024 · 12 comments
Open

Parsing of caddy log fails with cannot fetch headers from <nil>, no decisions for nikto test #2921

arminus opened this issue Mar 29, 2024 · 12 comments
Labels
kind/bug Something isn't working triage/accepted version/1.6.0

Comments

@arminus
Copy link

arminus commented Mar 29, 2024

What happened?

I'm trying to test with nikto if my crowdsec setup for caddy logs works. Basically, that test produced a bunch of caddy log entries like this:

{"level":"error","ts":1711730614.6231081,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.x.x.x","remote_port":"33774","proto":"HTTP/1.1","method":"GET","host":"home.test.xyz","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"home.scopeforge.de"}},"user_id":"","duration":0.000116418,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}

(remote IP anonymized here only)

crowdesc then produces this warning

time="2024-03-29T16:26:33Z" level=warning msg="failed to run RunTimeValue : cannot fetch headers from <nil> (1:58)\n | evt.Meta.http_status == '401' && evt.Unmarshaled.request.headers.Authorization startsWith 'Basic ' ? 'auth_fail' : ''\n | .........................................................^" id=polished-paper name=crowdsecurity/caddy-logs stage=s01-parse

(the timestamp might be slightly off, I'm not sure I picked the right warning line from the list of hundreds)

When I run the same nikto test against another box which crowdsec and apache logs, I get blocked, not with the caddy logs though, which is kind of in line with the warning that it cannot fetch headers - and so can't recognize the IP for the 401 ?

What did you expect to happen?

crowdsec to fully parse the caddy log line with the 401 errors

How can we reproduce it (as minimally and precisely as possible)?

acquis.yaml:

---
filenames:
 - /var/log/access.log
labels:
  type: caddy

Anything else we need to know?

No response

Crowdsec version

$ cscli version
2024/03/29 17:05:00 version: v1.6.0-4192af30
2024/03/29 17:05:00 Codename: alphaga
2024/03/29 17:05:00 BuildDate: 2024-01-31_12:35:08
2024/03/29 17:05:00 GoVersion: 1.21.6
2024/03/29 17:05:00 Platform: docker
2024/03/29 17:05:00 libre2: C++
2024/03/29 17:05:00 Constraint_parser: >= 1.0, <= 3.0
2024/03/29 17:05:00 Constraint_scenario: >= 1.0, <= 3.0
2024/03/29 17:05:00 Constraint_api: v1
2024/03/29 17:05:00 Constraint_acquis: >= 1.0, < 2.0

OS version

# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.19.1
PRETTY_NAME="Alpine Linux v3.19"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
$ uname -a
Linux 5dcff4feca5c 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) x86_64 Linux

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

$ cscli hub list -o raw
name,status,version,description,type
crowdsecurity/caddy-logs,enabled,0.7,Parse caddy logs,parsers
crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers
crowdsecurity/geoip-enrich,enabled,0.3,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/sshd-logs,enabled,2.3,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios
crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios
crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios
crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios
crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.3,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/http_base,enabled,0.2,,contexts
crowdsecurity/base-http-scenarios,enabled,0.8,http common : scanners detection,collections
crowdsecurity/caddy,enabled,0.1,caddy support : parser and generic http scenarios,collections
crowdsecurity/http-cve,enabled,2.6,Detect CVE exploitation in http logs,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections```

</details>


### Acquisition config

<details>
```console
# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
---
filenames:
 - /var/log/access.log
labels:
  type: caddy

cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory
# On Windows:
C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
# paste output here
</details>


### Config show

<details>

```console
$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://0.0.0.0:8080/
  - Login                   : localhost
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000```

</details>


### Prometheus metrics

<details>

```console
$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
@arminus arminus added the kind/bug Something isn't working label Mar 29, 2024
@github-actions GitHub Actions
Copy link

@arminus: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Mar 29, 2024 •
edited

So here is the parsed details

$ cscli explain --log '{"level":"error","ts":1711730614.6231081,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.x.x.x","remote_port":"33774","proto":"HTTP/1.1","method":"GET","host":"home.test.xyz","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"home.scopeforge.de"}},"user_id":"","duration":0.000116418,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}' --type caddy -v
line: {"level":"error","ts":1711730614.6231081,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.x.x.x","remote_port":"33774","proto":"HTTP/1.1","method":"GET","host":"home.test.xyz","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"home.scopeforge.de"}},"user_id":"","duration":0.000116418,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
	├ s00-raw
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		└ update evt.ExpectMode : %!s(int=0) -> 1
	|		└ update evt.Stage :  -> s01-parse
	|		└ update evt.Line.Raw :  -> {"level":"error","ts":1711730614.6231081,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.x.x.x","remote_port":"33774","proto":"HTTP/1.1","method":"GET","host":"home.test.xyz","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"home.scopeforge.de"}},"user_id":"","duration":0.000116418,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
	|		└ update evt.Line.Src :  -> /tmp/cscli_explain568595764/cscli_test_tmp.log
	|		└ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-03-29 19:05:55.247835527 +0000 UTC
	|		└ create evt.Line.Labels.type : caddy
	|		└ update evt.Line.Process : %!s(bool=false) -> true
	|		└ update evt.Line.Module :  -> file
	|		└ create evt.Parsed.message : {"level":"error","ts":1711730614.6231081,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"152.x.x.x","remote_port":"33774","proto":"HTTP/1.1","method":"GET","host":"home.test.xyz","uri":"/","headers":{"Connection":["Keep-Alive"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"home.scopeforge.de"}},"user_id":"","duration":0.000116418,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
	|		└ create evt.Parsed.program : caddy
	|		└ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2024-03-29 19:05:55.247859495 +0000 UTC
	|		└ create evt.Meta.datasource_type : file
	|		└ create evt.Meta.datasource_path : /tmp/cscli_explain568595764/cscli_test_tmp.log
	├ s01-parse
	|	└ 🟢 crowdsecurity/caddy-logs (+12 ~2)
	|		└ update evt.Stage : s01-parse -> s02-enrich
	|		└ create evt.Parsed.request : /
	|		└ create evt.Parsed.verb : GET
	|		└ create evt.Parsed.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
	|		└ create evt.Parsed.http_version : 1.1
	|		└ create evt.Unmarshaled.caddy : map[duration:0.000116418 level:error logger:http.log.access msg:handled request request:map[headers:map[Connection:[Keep-Alive] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36]] host:home.test.xyz method:GET proto:HTTP/1.1 remote_ip:152.x.x.x remote_port:33774 tls:map[cipher_suite:4865 proto: resumed:false server_name:home.scopeforge.de version:772] uri:/] resp_headers:map[Server:[Caddy] Www-Authenticate:[Basic realm="restricted"]] size:0 status:401 ts:1.7117306146231081e+09 user_id:]
	|		└ update evt.StrTime :  -> 1711730614
	|		└ create evt.Meta.log_type : http_access-log
	|		└ create evt.Meta.http_user_agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36
	|		└ create evt.Meta.service : http
	|		└ create evt.Meta.target_fqdn : home.test.xyz
	|		└ create evt.Meta.http_path : /
	|		└ create evt.Meta.http_status : 401
	|		└ create evt.Meta.http_verb : GET
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
	|		├ create evt.Enriched.MarshaledTime : 2024-03-29T16:43:34Z
	|		├ update evt.Time : 2024-03-29 19:05:55.247859495 +0000 UTC -> 2024-03-29 16:43:34 +0000 UTC
	|		├ update evt.MarshaledTime :  -> 2024-03-29T16:43:34Z
	|		├ create evt.Meta.timestamp : 2024-03-29T16:43:34Z
	|	├ 🟢 crowdsecurity/http-logs (+6)
	|		├ create evt.Parsed.impact_completion : true
	|		├ create evt.Parsed.static_ressource : false
	|		├ create evt.Parsed.file_dir : /
	|		├ create evt.Parsed.file_frag : 
	|		├ create evt.Parsed.file_ext : 
	|		├ create evt.Meta.http_args_len : 0
	|	├ 🟢 crowdsecurity/jellyfin-whitelist (unchanged)
	|	├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-dos-swithcing-ua

So it is being poured to a bucket what scenario is triggering on apache, the warning you are seeing is because there was not authentication header sent by the client.

Here are the filter around 40X response codes

$ grep filter /etc/crowdsec/scenarios/http-generic-bf.yaml
filter: "evt.Meta.service == 'http' && evt.Meta.sub_type == 'auth_fail'"
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.verb == 'POST' && evt.Meta.http_status == '401'"
filter: "evt.Meta.log_type == 'http_access-log' && evt.Parsed.verb == 'POST' && evt.Meta.http_status == '403'"

Okay looks like the "auth_fail" for Caddy will be more complicated, I guess what we should do is look at the respone headers and also check if www-authenticate was requested

@LaurenceJJones
Copy link
Contributor

Doing some testing

{"level":"error","ts":1711741798.0391326,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"46944","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:9080","uri":"/","headers":{"Authorization":[],"User-Agent":["curl/7.88.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.782670468,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"error","ts":1711741827.9626286,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"54462","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:9080","uri":"/","headers":{"User-Agent":["curl/7.88.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.000033987,"size":0,"status":401,"resp_headers":{"Server":["Caddy"],"Www-Authenticate":["Basic realm=\"restricted\""]}}
{"level":"info","ts":1711741864.947103,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"43498","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:9080","uri":"/","headers":{"Authorization":[],"User-Agent":["curl/7.88.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"Bob","duration":0.794682124,"size":18630,"status":200,"resp_headers":{"Content-Type":["text/html; charset=utf-8"],"Last-Modified":["Fri, 08 Dec 2023 00:28:15 GMT"],"Accept-Ranges":["bytes"],"Content-Length":["18630"],"Server":["Caddy"],"Etag":["\"s5bnz3edi\""]}}

First log is invalid credentials, Second is empty authentication and third is successful

@LaurenceJJones LaurenceJJones mentioned this issue Mar 29, 2024
Merged
@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Apr 2, 2024 •
edited

Can you try updating the caddy collection to see if this is now properly handled?

cscli hub update && cscli hub upgrade

@LaurenceJJones LaurenceJJones added question Further information is requested triage/accepted and removed needs/triage labels Apr 2, 2024
@arminus
Copy link
Author

arminus commented Apr 2, 2024

Thanks for the quick help!

I just ran the update, restarted the crowdsec container, then ran nikto again from a remote box, now the error is different:

time="2024-04-02T13:40:37Z" level=error msg="unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/http-generic-bf: scope is Ip but Meta[source_ip] doesn't exist"

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Apr 2, 2024 •
edited

Thanks for the quick help!

I just ran the update, restarted the crowdsec container, then ran nikto again from a remote box, now the error is different:

time="2024-04-02T13:40:37Z" level=error msg="unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/http-generic-bf: scope is Ip but Meta[source_ip] doesn't exist"

Also ensure your caddy installation is up to date to the github releases as using an old version may cause this error

https://github.com/caddyserver/caddy/releases/tag/v2.7.6

@arminus
Copy link
Author

arminus commented Apr 2, 2024

Ok, going to caddy 2.8 fixed the problem.

time="2024-04-02T14:04:55Z" level=info msg="(localhost/crowdsec) crowdsecurity/http-generic-bf by ip x.x.x.x (AT/197540) : 4h ban on Ip x.x.x.x"
time="2024-04-02T14:04:55Z" level=info msg="127.0.0.1 - [Tue, 02 Apr 2024 14:04:55 UTC] \"POST /v1/alerts HTTP/1.1 201 22.47711ms \"crowdsec/v1.6.0-4192af30\" \""
time="2024-04-02T14:04:59Z" level=info msg="Signal push: 1 signals to push"

There's one more, though I think:

time="2024-04-02T14:05:04Z" level=warning msg="failed to run RunTimeValue : invalid operation: int(<nil>) (1:1)\n | int(evt.Unmarshaled.caddy.status)\n | ^" id=red-sunset name=crowdsecurity/caddy-logs stage=s01-parse

@LaurenceJJones
Copy link
Contributor

Ok, going to caddy 2.8 fixed the problem.

time="2024-04-02T14:04:55Z" level=info msg="(localhost/crowdsec) crowdsecurity/http-generic-bf by ip x.x.x.x (AT/197540) : 4h ban on Ip x.x.x.x"
time="2024-04-02T14:04:55Z" level=info msg="127.0.0.1 - [Tue, 02 Apr 2024 14:04:55 UTC] \"POST /v1/alerts HTTP/1.1 201 22.47711ms \"crowdsec/v1.6.0-4192af30\" \""
time="2024-04-02T14:04:59Z" level=info msg="Signal push: 1 signals to push"

There's one more, though I think:

time="2024-04-02T14:05:04Z" level=warning msg="failed to run RunTimeValue : invalid operation: int(<nil>) (1:1)\n | int(evt.Unmarshaled.caddy.status)\n | ^" id=red-sunset name=crowdsecurity/caddy-logs stage=s01-parse

Do you have the caddy logs, be surprised if there was no status response?

@arminus
Copy link
Author

arminus commented Apr 2, 2024 •
edited

I have the full caddy log, but it's kind of hard to dig the relevant line out of it since the timestamps there are in ms vs the zulu times in the crowdsec log. I'll try later.

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Apr 2, 2024 •
edited

I have the full caddy log, but it's kind of hard to dig the relevant line out of it since the timestamps there are in ms vs the zulu times in the crowdsec log. I'll try later.

havent tried it but jq?

jq '. | select( .status | . == null or . == "")' < /path/to/caddy/log

It depends if you have other things logging to the file such as debug logs for plugins they will also be attempted to be parsed

@arminus
Copy link
Author

arminus commented Apr 2, 2024

Ok, that produces a bunch of lines with this pattern (primarily for gitea and jira for which caddy is a proxy here):

{
  "level": "error",
  "ts": 1712066552.3486984,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting with incomplete response",
  "upstream": "172.19.0.8:3000",
  "duration": 0.00405423,
  "request": {
    "remote_ip": "x.x.x.x",
    "remote_port": "56973",
    "client_ip": "x.x.x.x",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "git.mydomain.com",
    "uri": "/user/events",
    "headers": {
      "Cache-Control": [
        "no-cache"
      ],
      "Cookie": [],
      "Sec-Fetch-Dest": [
        "empty"
      ],
      "Te": [
        "trailers"
      ],
      "Pragma": [
        "no-cache"
      ],
      "X-Forwarded-Proto": [
        "https"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
      ],
      "X-Forwarded-Host": [
        "git.mydomain.com"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Sec-Fetch-Mode": [
        "cors"
      ],
      "Accept": [
        "text/event-stream"
      ],
      "Accept-Language": [
        "de,en-US;q=0.7,en;q=0.3"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "X-Forwarded-For": [
        "x.x.x.x"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "git.mydomain.com"
    }
  },
  "error": "reading: context canceled"
}

@LaurenceJJones
Copy link
Contributor

Ok, that produces a bunch of lines with this pattern (primarily for gitea and jira for which caddy is a proxy here):

{
  "level": "error",
  "ts": 1712066552.3486984,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting with incomplete response",
  "upstream": "172.19.0.8:3000",
  "duration": 0.00405423,
  "request": {
    "remote_ip": "x.x.x.x",
    "remote_port": "56973",
    "client_ip": "x.x.x.x",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "git.mydomain.com",
    "uri": "/user/events",
    "headers": {
      "Cache-Control": [
        "no-cache"
      ],
      "Cookie": [],
      "Sec-Fetch-Dest": [
        "empty"
      ],
      "Te": [
        "trailers"
      ],
      "Pragma": [
        "no-cache"
      ],
      "X-Forwarded-Proto": [
        "https"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
      ],
      "X-Forwarded-Host": [
        "git.mydomain.com"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Sec-Fetch-Mode": [
        "cors"
      ],
      "Accept": [
        "text/event-stream"
      ],
      "Accept-Language": [
        "de,en-US;q=0.7,en;q=0.3"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "X-Forwarded-For": [
        "x.x.x.x"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "git.mydomain.com"
    }
  },
  "error": "reading: context canceled"
}

Makes sense context is canceled so there is no status code, hmmm let me think about this

@LaurenceJJones LaurenceJJones removed the question Further information is requested label Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working triage/accepted version/1.6.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@arminus @LaurenceJJones