Json Request Body Parser doesn't respect charset=iso8859-1

[UweBeckert]

Describe the bug

We are running nginx with ModSecurity 3.0.9 and rule set 3.2.0.

Our user are sending POST requests with content-type
application/fhir+json
application/fhir+json; charset=utf-8
application/fhir+json; charset=iso8859-1

To enable the Json request body parser we added this rule

SecRule REQUEST_HEADERS:Content-Type "^application/.+[+]json.*$"
"id:'100',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"

But somehow the charset is not considered by the Json parser.
If a POST is done with
content-type: application/fhir+json; charset=iso8859-1
and with some iso8859-1 characters in the json body, ModSecurity can't parse the requests and returns an error

[14] tcp.0: [[1706742332.054460688, {}], {"date"=>1706742329.475638, "log"=>"2024/01/31 23:05:29 [error] 3588367#3588367: *26426370 [client 128.65.209.32] ModSecurity: Access denied with code 400 (phase 2). Matched "Operator `Eq' with parameter `0' against variable `REQBODY_ERROR' (Value: `1' ) [file "/etc/nginx/modsec/modsecurity.conf"] [line "54"] [id "200002"] [rev ""] [msg "Failed to parse request body."] [data "JSON parsing error: lexical error: invalid bytes in UTF8 string.\x0a"] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"]

ModSecurity Json parser considers the body as UTF8.

How to convince the Json parser to parse it as iso8859-1 as stated in the content-type?

[airween]

Hi @UweBeckert,

thanks for reporting - but it seems like not a CRS issue rather a ModSecurity problem (this is why I removed the bug label).

If you take a look to ModSecurity's issue list, you can see there are few similar error.

And yes, it sounds like the engine does not support the non-UTF8 JSON requests. Just take a look to the comment of the engine's body processor - I think the author had knew this when he created the code. But the intention was in v2's parser too.

YAJL (Yet Another Json Library) supports the processing of non-UTF8 structure, but the documentation says it has a bit higher cost (see yajl_dont_validate_strings option, which is not used yet): "When set, this option makes parsing slightly more expensive (~7% depending on processor and compiler in use)".

Unfortunately you should use UTF8 encoded JSON.

I'm going to close this issue.

Few other notes: I'm not sure the iso8859-1 is a valid charset - did you mean iso-8859-1? The rule with id 100 misses a \ from the end of the first line. Seems that libmodsecurity3's parser allows this syntax but if you want to use it on Apache+mod_security2 that won't work. The correct syntax is:

SecRule REQUEST_HEADERS:Content-Type "^application/.+[+]json.*$" \
"id:'100',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"

See the extra \ at the end of the first line.

I know the problem is not one of these, it's just a note.