[EPIC] Crowdsec integration to James

[quantranhong1999]

CrowdSec is a free, modern & collaborative behavior detection engine, coupled with a global IP reputation network, based on AI behaviour refinement.

Develop a third-party plugin in James for questionning CrowdSec:

• Create a SMTP EHLO hook questionning via a REST call the CrowdSec local agent
• Create a mailet questionning via a REST call the CrowdSec local agent
• Create a mailet to provision local CrowdSec database (for highest level of spam for instance)
• Think about the interfaces we would need to question CrowdSec upon incoming IMAP connections
• Externalize behaviour linked to failed login attempts (sleep, 3 failure connection closure) as configurable extensible - plugins.
• Use it to manage IP reporting to crowdSec, especially upon failed authentications (~fail2ban).

rf: https://issues.apache.org/jira/browse/JAMES-3897, linagora/tmail-backend#803

cc @guimard @chibenwa

[quantranhong1999]

@thanhbv200585 please research the topic and make a proposal about the list of tasks (could think more or less than the above tasks if needed) with basic ticket details first.

[vttranlina]

query mode ip API looks like what we want
https://doc.crowdsec.net/docs/local_api/bouncers#query-mode--ip

[thanhbv200585]

Task proposal:

• ADR
• Create a SMTP EHLO hook questionning via a REST call the CrowdSec local agent
• Create a mailet questionning via a REST call the CrowdSec local agent
• Create a mailet to provision local CrowdSec database (for highest level of spam for instance) (https://doc.crowdsec.net/docs/local_api/bouncers#query-mode--ip)
• Think about the interfaces we would need to question CrowdSec upon incoming IMAP connections (check a mail used for spam)
• Externalize behaviour linked to failed login attempts (sleep, 3 failure connection closure) as configurable extensible - plugins.
• Use it to manage IP reporting to crowdSec, especially upon failed authentications (~fail2ban).

[guimard]

Task proposal:

* ADR
* Create a SMTP EHLO hook questionning via a REST call the CrowdSec local agent

Not only SMTP, IMAP/POP also

* Create a mailet questionning via a REST call the CrowdSec local agent
* Create a mailet to provision local CrowdSec database (for highest level of spam for instance) (https://doc.crowdsec.net/docs/local_api/bouncers#query-mode--ip)
* Think about the interfaces we would need to question CrowdSec upon incoming IMAP connections (check a mail used for spam)
* Externalize behaviour linked to failed login attempts (sleep, 3 failure connection closure) as configurable extensible - plugins.
* Use it to manage IP reporting to crowdSec, especially upon failed authentications (~fail2ban).

[quantranhong1999]

Task proposal:

Basic details for each task, and are there any possible tasks?

[guimard]

Two distincts plugins:

• one to check all connections (call Crowdsec)
• (later) one to push information to Crowdsec (~fail2ban)

[Arsnael]

I didn't have time to really have a look myself tbh, but I would think you would need first to implement a client to communicate with crowdsec, which would force you to implement a docker extension for testing too

[quantranhong1999]

(later) one to push information to Crowdsec (~fail2ban)

Do you suggest letting CrowdSec access the James logs, or we let James push directly alerts/decisions via CrowdSec HTTP API?

[guimard]

It's easy to write the first plugin: Lemonldap-NG plugin as example

[quantranhong1999]

(later) one to push information to Crowdsec (~fail2ban)

I succeeded to report IP via HTTP API directly
Screencast from 21-09-2023 14:35:08.webm

[chibenwa]

or we let James push directly alerts/decisions via CrowdSec HTTP API?

This one eliminate the need to plug James logs to crowdsec and might save headaches with fluentbit / loki...

[quantranhong1999]

Rene:

This one eliminate the need to plug James logs to crowdsec and might save headaches with fluentbit / loki...

You mean implement the magic on our side about if we should ban an IP or not and just report to crowdsec? I feel it would be more of an headache than just letting crowdsec scraping our logs and doing the magic itself that he is supposed to know to do well already, no? It would mean you need to track all failed attempts (like with Redis?)

I would be more interested to know if we can just report via HTTP API to crowdsec IPs that are failing to login and if crowdsec is able to detect many reports of failed attempts from a same IP from us and act in consequence?

(later) one to push information to Crowdsec (~fail2ban)

Can we have more detail on that? I'm not sure if it means we do the magic implementation of determining ourselves which IPs to ban, or if we just tell crowdsec this IP failed to login and let crowdsec take the decision after a couple of similar reports... (joins my previous point)

Looks confusing to me (and the team)

[quantranhong1999]

Propose other tasks:
- Crowdsec Docker extension
- Crowdsec configuration + Crowdsec HTTP client
... above tasks: #4874 (comment)
- Integrate Crowdsec modules into TMail backend
- Helm chart for Crowdsec (crowdsec already provides helm chart)
- Deploy Crowdsec in preprod/sandbox
- Setup Prometheus + Grafana dashboard for CrowdSec (CrowdSec does provide /metrics endpoint)
- Performance test TMail with CrowdSec

[quantranhong1999]

For @thanhbv200585 trying to parse James' logs and build a CrowdSec attack scenario - would need to write a log parser https://docs.crowdsec.net/docs/concepts#parsers to parse James' logs.

Seems not easy to me, but let's try :)

[quantranhong1999]

FYI, CrowdSec supports

• Ingress Nginx Bouncer / OpenResty Bouncer (which APISIX based on) / NGINX bouncer -> cover JMAP, and even other Linagora services
• HAProxy Bouncer -> cover IMAP

Missing SMTP though.
If we want a no-code solution, and not only for TMail but for other services as well, we have the choice :-)

[chibenwa]

Integrate Crowdsec modules into TMail backend

Why not James?