StackOverflow for JsonBodyFilter

[Sam-Kruglov]

Using a debugger, I built a function that will reproduce it (this is inside com.monese.card.service.config.logbook.JsonBodyFilters#replace inside delegate lambda):

public static void main(String[] args) {
    //com.monese.card.service.config.logbook.JsonBodyFilters#replace
    String STRING = "(?:\"(?:[^\"\\\\]|\\\\.)*\")";
    String value = STRING;
    final Pattern pattern = compile("(?<key>\"(?<property>.*?)\"\\s*:\\s*)(" + value + "|null)");
    String body = "{\"variables\":{},\"query\":\"\\n    query IntrospectionQuery {\\n      __schema {\\n        queryType { name }\\n        mutationType { name }\\n        subscriptionType { name }\\n        types {\\n          ...FullType\\n        }\\n        directives {\\n          name\\n          description\\n          locations\\n          args {\\n            ...InputValue\\n          }\\n        }\\n      }\\n    }\\n\\n    fragment FullType on __Type {\\n      kind\\n      name\\n      description\\n      fields(includeDeprecated: true) {\\n        name\\n        description\\n        args {\\n          ...InputValue\\n        }\\n        type {\\n          ...TypeRef\\n        }\\n        isDeprecated\\n        deprecationReason\\n      }\\n      inputFields {\\n        ...InputValue\\n      }\\n      interfaces {\\n        ...TypeRef\\n      }\\n      enumValues(includeDeprecated: true) {\\n        name\\n        description\\n        isDeprecated\\n        deprecationReason\\n      }\\n      possibleTypes {\\n        ...TypeRef\\n      }\\n    }\\n\\n    fragment InputValue on __InputValue {\\n      name\\n      description\\n      type { ...TypeRef }\\n      defaultValue\\n    }\\n\\n    fragment TypeRef on __Type {\\n      kind\\n      name\\n      ofType {\\n        kind\\n        name\\n        ofType {\\n          kind\\n          name\\n          ofType {\\n            kind\\n            name\\n            ofType {\\n              kind\\n              name\\n              ofType {\\n                kind\\n                name\\n                ofType {\\n                  kind\\n                  name\\n                  ofType {\\n                    kind\\n                    name\\n                  }\\n                }\\n              }\\n            }\\n          }\\n        }\\n      }\\n    }\\n  \"}";
    pattern.matcher(body).find();
}

[PascalSchumacher]

This is proably the same proablem as the one reported in #686

[Sam-Kruglov]

I think it failed on rc6 as well tho

[whiskeysierra]

@PascalSchumacher It's related, yes. But the issue with long was fixed (my test uses a property with 10,000 characters or something like that. I believe the issue now is that there are many " in there which require backtracking.

[whiskeysierra]

@Sam-Kruglov Thanks for the issue and the test/input! 🎉

[whiskeysierra]

@Sam-Kruglov Can you share your configuration, i.e. set of property names and replacement? My local test with that input works. Which version are you running?

[Sam-Kruglov]

It's rc8. What exactly do you need apart what's on the screenshot?

[whiskeysierra]

The predicate or set of property names. Can you try to write a test that reproduces this locally? Seems to work for me... 🤔

[Sam-Kruglov]

So the main function doesn't throw an error for you when you run it? For me it fails

[PascalSchumacher]

@PascalSchumacher It's related, yes. But the issue with long was fixed (my test uses a property with 10,000 characters or something like that. I believe the issue now is that there are many " in there which require backtracking.

I think you are mixing up #681 and #686. #681 was for long property values. Thanks again for fixing this in RC.8. I successfully tested values to up to 300.000 chars (didn't bother testing even longer values) using RC.8.

#686 is for long property values containg (a lot of) escaped double qoutes. The unit test in the issue will allow you to reproduce the problem.

[whiskeysierra]

Good point. Seems I was overwhelmed with StackOverflowErrors recently...

[whiskeysierra]

This is proably the same proablem as the one reported in #686

The sample input has no double quotes. So I'm kind of lost why this one fails, especially since I can't reproduce it.

[whiskeysierra]

@Sam-Kruglov I added your test input as an additional test case in #712 (aiming to fix #686). It already passed with on my machine before the changes though. Maybe you could verify if that branch builds locally for you. I'm out of ideas how to reproduce that, to be honest.

[Sam-Kruglov]

I'm using jdk11, are you? I will try tomorrow

[whiskeysierra]

Usually not, but I just tried and current master works as well with 11.

[whiskeysierra]

@Sam-Kruglov Did you have a chance to try?

[Sam-Kruglov]

I tried using open jdk 11.0.1 and 11.0.3 that I have installed. I launched it from intellij and got the stack overflow. Not sure how it works for you...

[Sam-Kruglov]

oracle jdk 1.8.0_192 on a clean project with no dependencies - same stack overflow. Try copy-pasting the main method from my first message

[whiskeysierra]

I took your code snippet and turned it into a test case without changing the values. It's linked above and already in the master.

…

On Fri, 14 Feb 2020, 16:40 Sam Kruglov, ***@***.***> wrote: oracle jdk 1.8.0_192 on a clean project with no dependencies - same stack overflow. Try copy-pasting the main method from my first message — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#689?email_source=notifications&email_token=AADI7HO24XJFTJEJKU3WMZTRC23NPA5CNFSM4KHHV3LKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELZNYOI#issuecomment-586341433>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADI7HLOUZMNWWCAPWLYHFDRC23NPANCNFSM4KHHV3LA> .

[whiskeysierra]

So I took another look and your code sample inlined patterns that are no longer in use.

There is a test case in the master that covers your sample input and it passes ✔️

logbook/logbook-json/src/test/java/org/zalando/logbook/json/JsonBodyFiltersTest.java

Lines 173 to 182 in 54054d6

	@Test
	void supportsVeryLargeEmbeddedJsonValues() throws IOException {
	final byte[] bytes = Files.readAllBytes(Paths.get("src/test/resources/huge-json-value.json"));
	final String original = new String(bytes, UTF_8);
	final BodyFilter unit = replaceJsonStringProperty(singleton("variables"), "XXX");
	final String actual = unit.filter(contentType, original);
	assertThat(actual, is(original));
	}

logbook/logbook-json/src/test/resources/huge-json-value.json

Lines 1 to 4 in 54054d6

	{
	"variables": {},
	"query": "\n query IntrospectionQuery {\n __schema {\n queryType { name }\n mutationType { name }\n subscriptionType { name }\n types {\n ...FullType\n }\n directives {\n name\n description\n locations\n args {\n ...InputValue\n }\n }\n }\n }\n\n fragment FullType on __Type {\n kind\n name\n description\n fields(includeDeprecated: true) {\n name\n description\n args {\n ...InputValue\n }\n type {\n ...TypeRef\n }\n isDeprecated\n deprecationReason\n }\n inputFields {\n ...InputValue\n }\n interfaces {\n ...TypeRef\n }\n enumValues(includeDeprecated: true) {\n name\n description\n isDeprecated\n deprecationReason\n }\n possibleTypes {\n ...TypeRef\n }\n }\n\n fragment InputValue on __InputValue {\n name\n description\n type { ...TypeRef }\n defaultValue\n }\n\n fragment TypeRef on __Type {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n ofType {\n kind\n name\n }\n }\n }\n }\n }\n }\n }\n }\n "
	}

[Sam-Kruglov]

Thanks for taking a deep look