`cscli hub upgrade` attempts to update a local parser/enricher

[Athanasius]

What happened?

1. System picked up the upgrade to 1.6.5
2. The Debian postinst script ran cscli hub upgrade
3. This attempted to check a local enricher, which fails because there's no URL for it.

In addition, because cscli hub upgrade is run from the Debian postinst script it makes dpkg/APT think the installation failed.

What did you expect to happen?

cscli hub upgrade should continue to ignore local parsers of any kind.

How can we reproduce it (as minimally and precisely as possible)?

1. Have any local, not installed from hub, enricher. Mine is <config dir>/parsers/s02-enrich/whitelists_fysh.yaml
2. Run cscli hub upgrade

Anything else we need to know?

This was all working fine with v1.6.4. I just downgraded to that version and running cscli hub upgrade completes without errors.

Crowdsec version

Details

$ cscli version
# paste output here
version: v1.6.5-debian-pragmatic-amd64-d8dcdc91
Codename: alphaga
BuildDate: 2025-02-07_14:53:23
GoVersion: 1.23.6
Platform: linux
libre2: C++
User-Agent: crowdsec/v1.6.5-debian-pragmatic-amd64-d8dcdc91-linux
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0
Built-in optional components: cscli_setup, datasource_appsec, datasource_cloudwatch, datasource_docker, datasource_file, datasource_http, datasource_journalctl, datasource_k8s-audit, datasource_kafka, datasource_kinesis, datasource_loki, datasource_s3, datasource_syslog, datasource_victorialogs, datasource_wineventlog

OS version

Details

# On Linux:
$ cat /etc/os-release
# paste output here
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
$ uname -a
# paste output here
Linux river.fysh.org 6.1.125-fysh-kvmguest #1 SMP PREEMPT_DYNAMIC Fri Jan 17 15:07:39 UTC 2025 x86_64 GNU/Linux


# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Enabled collections and parsers

Details

$ cscli hub list -o raw
# paste output here
Loaded: 134 parsers, 10 postoverflows, 751 scenarios, 8 contexts, 4 appsec-configs, 93 appsec-rules, 131 collections
Unmanaged items: 1 local, 0 tainted
name,status,version,description,type
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/geoip-enrich,enabled,0.5,"Populate event with geoloc info : as, country, coords, source range.",parsers
crowdsecurity/mysql-logs,enabled,0.4,Parse MySQL logs,parsers
crowdsecurity/sshd-logs,enabled,2.9,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
fysh/whitelists,"enabled,local",,,parsers
crowdsecurity/mysql-bf,enabled,0.2,Detect mysql bruteforce,scenarios
crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-cve-2024-6387,enabled,0.2,Detect exploitation attempt of CVE-2024-6387,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios
crowdsecurity/bf_base,enabled,0.1,,contexts
crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections
crowdsecurity/mysql,enabled,0.1,mysql support : logs and brute-force scenarios,collections
crowdsecurity/sshd,enabled,0.5,sshd support : parser and brute-force detection,collections

Acquisition config

(NB: I removed the opening <detail> as github wasn't rendering the pasted text properly)

On Linux:

$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*

paste output here

#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: mysql) / files : /var/log/mysql/error.log
filenames:
  - /var/log/mysql/error.log
labels:
  type: mysql
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages
filenames:
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/messages
labels:
  type: syslog
---
cat: '/etc/crowdsec/acquis.d/*': No such file or directory

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

Details

$ cscli config show
# paste output here
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://192.168.41.2:8080/
  - Login                   : <redacted>
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 192.168.41.2:8080
  - Listen Socket           : 
  - Profile File            : /etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

Details

$ cscli metrics
# paste output here
WARNING while fetching metrics: executing GET request for URL "http://192.168.41.2:6060/metrics" failed: Get "http://192.168.41.2:6060/metrics": dial tcp 192.168.41.2:6060: connect: connection refused 
╭───────────────────────────────────────────────────────────────────────╮
│ Bouncer Metrics (cs-firewall-bouncer-1696258531) since 2025-01-31 19: │
│ 33:25 +0000 UTC                                                       │
├────────┬──────────────────┬───────────────────┬───────────────────────┤
│ Origin │ active_decisions │      dropped      │       processed       │
│        │        IPs       │  bytes  │ packets │   bytes   │  packets  │
├────────┼──────────────────┼─────────┼─────────┼───────────┼───────────┤
│ ipset  │           23.66k │  55.50M │ 967.70k │         - │         - │
├────────┼──────────────────┼─────────┼─────────┼───────────┼───────────┤
│  Total │           23.66k │  55.50M │ 967.70k │     8.85G │    17.15M │
╰────────┴──────────────────┴─────────┴─────────┴───────────┴───────────╯

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Details

The custom enricher is:

name: fysh/whitelists
description: "Whitelist fysh-specific IPs"
data:
  - dest_file: "fysh-whitelist-ips.txt"
    type: string

whitelist:
  reason: "Known good local IPs"
  ip: 
    - "<redacted>" # Athan's Aquiss static IPv4 address
  cidr:
    - "192.168.41.0/2" # Internal VM NAT
  expression:
  #   - "'foo.com' in evt.Meta.source_ip.reverse" 
    - evt.Meta.source_ip in File('fysh-whitelist-ips.txt')

[github-actions]

@Athanasius: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[LaurenceJJones]

Hey 👋

This is the output I got

Nothing to do, the hub index is up to date.
 parsers:laurencejjones/dovecot-pam - not downloading local item
 parsers:my/whitelists - not downloading local item
 scenarios:laurencejjones/dovecot-pam - not downloading local item
 scenarios:laurencejjones/ufw-block - not downloading local item
 appsec-configs:my/appsec-config - not downloading local item
 appsec-rules:laurencejjones/myrules - not downloading local item
 downloading https://hub-data.crowdsec.net/web/log4j2_cve_2021_44228.txt
 downloading https://hub-data.crowdsec.net/web/bad_user_agents.regex.txt
 downloading https://hub-data.crowdsec.net/web/xss_probe_patterns.txt
 Get started with CrowdSec:
  * Go further by following our post installation steps : https://docs.crowdsec.net/u/getting_started/next_steps
 ====================================================================================================================
  * Install a remediation component to block attackers: https://docs.crowdsec.net/u/bouncers/intro
 ====================================================================================================================
  * Find more collections, parsers and scenarios created by the community with the Hub: https://hub.crowdsec.net
 ====================================================================================================================
  * Subscribe to additional blocklists, visualize your alerts and more with the console: https://app.crowdsec.net
 You can always run the configuration again interactively by using '/usr/share/crowdsec/wizard.sh -c'
 Log ended: 2025-02-07  17:01:17

mywhitelist is a local enricher and it didn't seem to fail the installation is there anything custom in this enricher such as a data property?

Edit: missed the parser in the last details

[Athanasius]

mywhitelist is a local enricher and it didn't seem to fail the installation is there anything custom in this enricher such as a data property?

Yes, the entire content of the enricher is in the Related custom configs versions section.

[LaurenceJJones]

We created a fix, however, we are unsure how wide spread the issue is and if it warrents a rush release of 1.6.6.

For your data items does it actually have a remote url where you store the data? as i didnt realize that we introduced a feature where local items can now actually download remote files.

[Athanasius]

For your data items does it actually have a remote url where you store the data? as i didnt realize that we introduced a feature where local items can now actually download remote files.

No, there is no source URL for this. I did originally have source_url: '' as part of the data: section, but removed it in case explicitly stating that was causing the issue. The issue remained.

Having struggled to move this enrichment into a subdirectory, crowdsec insisting on still trying to load the then non-existent file, is it possible something has cached it, including the source_url: '' and that is the root cause given I've removed that statement from the file ?

[LaurenceJJones]

For your data items does it actually have a remote url where you store the data? as i didnt realize that we introduced a feature where local items can now actually download remote files.

No, there is no source URL for this. I did originally have source_url: '' as part of the data: section, but removed it in case explicitly stating that was causing the issue. The issue remained.

Having struggled to move this enrichment into a subdirectory, crowdsec insisting on still trying to load the then non-existent file, is it possible something has cached it, including the source_url: '' and that is the root cause given I've removed that statement from the file ?

typically no, if the file is still under s02-enrich it will still be loaded if it has the .yaml extension. Currently as per the fix, there is a bug that if the data structure is configured it expects there to be a source_url and throws an error if it doesnt exists.

[Athanasius]

Just to confirm that 1.6.6 looks to have this issue fixed. Thanks !

[LaurenceJJones]

Just to confirm that 1.6.6 looks to have this issue fixed. Thanks !

Thank you for that swift update! awesome anything else that happens that doesnt seem right open a new issue 👍🏻