[FR]: GitLab, application_json.log, application.log (deprecated)

[SDuesterhaupt]

GitLab marked the application.log as deprecated in GitLab 15.10 and introduced the application_json.log.

Please I need support to update the filter regular expression. That's definitely my weak side in Linux.

Thank you. I appreciate your work. :)

Environment:

• Fail2Ban version : 1.0.2-7.el9
• OS, including release name/version : Rocky Linux 9.2 (Blue Onyx)

Service, project or product which log or journal should be monitored

• Name of filter or jail in Fail2Ban (if already exists) : gitlab
• Service, project or product name, including release name/version : gitlab

Log or journal information

• Log file name(s) : application_json.log

Any additional information

• Reference: https://docs.gitlab.com/ee/administration/logs/#applicationlog-deprecated

Relevant lines from monitored log files:

failures in sense of fail2ban filter (fail2ban must match):

{"severity":"INFO","time":"2023-08-25T11:25:19.979Z","correlation_id":"01H8P721YTW1YH5Q7YCSJPVJ5T","meta.caller_id":"SessionsController#new","meta.remote_ip":"x.x.x.x","meta.feature_category":"system_access","meta.client_id":"ip/x.x.x.x","message":"Failed Login: username=**** ip=x.x.x.x"}

legitimate messages (fail2ban should not consider as failures):

{"severity":"INFO","time":"2023-08-25T11:55:10.525Z","correlation_id":"01H8P8RPMVW53W0KWEN2H2DNF4","meta.caller_id":"SessionsController#create","meta.remote_ip":"x.x.x.x","meta.feature_category":"system_access","meta.user":"****","meta.user_id":2,"meta.client_id":"user/2","message":"Successful Login: username=**** ip=x.x.x.x method=standard admin=true"}

[sebres]

At the moment we don't have real JSON parsing in fail2ban (see #3526)...
However writing more or less good filter is possible, just would imply some restrictions (few tags before time, that are not remote_ip and message, the order of tags is important, etc):

[Definition]

_groupre = (?:"(?!(?:time|meta\.remote_ip|message)\b)\w[^"]+":(?:"[^"]+"|\w+)\s*[,\}]\s*)
datepattern = ^\{%(_groupre)s*"time":"%%Y-%%m-%%dT%%H:%%M:%%S\.%%f%%z"\s*,\s*
failregex = ^%(_groupre)s*"meta\.remote_ip":"<ADDR>"\s*,\s*%(_groupre)s*"message":"Failed Login: username=<F-USER>(?:\S*|[^"]*?)(?= ip=)</F-USER>

[SDuesterhaupt]

Excellent. The filter works as expected.

Thank you very much. From my point of view, this issue can be closed.