Investigate possibilities for improving start-up performance

[dev-chirag]

I have a Microservice that contains a Jersey REST service and that accepts JSON body. I use jersey's Jackson support to deserialize the request into POJOs. This is a complex JSON (sometime upto 250kbs).

This microservice runs behind Envoy loadbalancer on a K8 cluster. So I can easily scale the number of instances.

Now the problem is, when I scale my application and Envoy starts distributing load (round-robin), my new instance has a really slow startup. For this new instance, it's like a burst of requests before JVM and service warmup. And during this time, I see that all my requests on this instance fail with a 503.

Since this new instance experienced a burst without Jackson and Jersey warmup, all threads are busy classloading. I can see this in my CPU profiling that almost 100% CPU is consumed by C1 and C2 compiler threads and loaded classes count shoots up.

My assumption here is that, all the requests are executed in parallel and all threads handling them are waiting for Jackson to load the POJO classes. Once the classloading is completed, the classes are cached, freeing up CPU and allowing threads to process the request instead.

I understand this from #1970 that this could be the problem. Is my assumption correct? Is there any thing I can do to load these classes before the first request?

Any leads would be much appreciated. Thanks

[plokhotnyuk]

@dev-chirag a good option is switching to Scala and using jsoniter-scala. It does reflection in compile time only and is more efficient in runtime. Also it can be compiled to GraalVM native image to exclude impact of JIT compiler at all.

[cowtowncoder]

@plokhotnyuk Switching to Scala is typically not a good idea for performance issues. :-p

[cowtowncoder]

@dev-chirag You can certainly "warm up" things by eagerly calling serialize/deserialize on types that are likely used, as long as ObjectMapper being used is then shared for actual load. Beside JVM warmup aspect the first read and write of given POJO does a lot of reflection which will not be done again as long as ObjectMapper used keeps generated serializers/deserializers cached .

[cowtowncoder]

One note on class loading: class loading itself would be done by JVM; Jackson does not really do any of that (with the exception of 3 modules: Afterburner and Blackbird generate optimized handlers, and Mr Bean can materialize interface/abstract class implementations).
But there is definitely lots of introspection and (de)serialize construction when a POJO instance is encountered for the first time. If there is concurrent access, it is possible that there is a lot of duplicated work at this point -- doing pre-emptive readValue()/writeValue() calls for warmup, before service is indicated as live, would allow avoiding this.

[ptorkko]

We recently ran into this, or a very similar issue with our spring boot app.

After the service starts up and a loadbalancer marks it healthy, it starts receiving quite a bit of traffic from the clients.

In my local testing this seems cause issues in the cpu constrained environment (e.g. 10 parallel nodes, 1vcpu each) when parallel threads start making api calls to services using the same POJO types, causing severe lock contention in DeserializerCache#_createAndCacheValueDeserializer (v2.13.4.2).

I tested creating a simple warmup sequence for spring beans that use http services and complex classes as bodies, e.g.:

@Override
public void warmup(ObjectMapper objectMapper) throws Exception {
    objectMapper.readValue("{}", new TypeReference<FancyRequestBody>() {});
    objectMapper.readValue("{}", new TypeReference<SomeOtherBody>() {});
    objectMapper.readValue("[]", new TypeReference<List<FancyRequestBody>>() {});
}

This seems to basically remove the issue as the deserializer caches are done before accepting any traffic.

Question is, which parts are actually necessary? Do I need to .readValue for the root object only? What about permutations of List<T>, Map<K, T> or even deeper nestings?

Also is objectMapper.writeValueAsString(new SomeResponseObject()) enough for SerializerCache? And same as above, what about types nested in the root type, and generic collections etc?

[cowtowncoder]

I'd have to double-check this, but I think that for deserialization, cache pre-warming should work well for transitive dependencies, there not being much difference between root-level values and branch/leaf level. For serialization there is bit more difference due to more dynamic nature of handling. Still, there's probably most value in doing readValue()/writeValue() for a small set of most widely used types.

objectMapper.writeValueAsString() should indeed be sufficient. Caching exists both at root value level and dependencies.

On doing List (and Map, array) variants: it can help of course, but may not be necessary: element value (de)serializers getting cached likely gets most benefits, as Introspection needed for List/Map/array (de)serializers is quite a bit less work than that for POJOs.
Similarly most other JDK type (de)serializers have less reliance on Introspection so not as necessary to pre-warm (but many do get exercised indirectly as POJO properties anyway).

I hope this helps.

[ptorkko]

Definitely helps, thanks.

Will the deserialization cache be populated with types of members of the root type if the value to deserialize is e.g. {}?

record RootType(ChildType member) {}
record ChildType(String value) {}

should both be cached via .readValue separately or does the RootType introspection achieve both? I'm assuming the cache is flat, not hierarchical starting from the RootType.

[JooHyukKim]

If you use debugger, you can check it out yourself via mapper.getDeserializationContext()._cache.
I am not yet sure whether or not there's proper way of accessing the cache tho.

[cowtowncoder]

@ptorkko For deserializers, yes, value deserializers are fetched eagerly and not on-demand. For serializers fetching is dynamic unless type is final.