After fixing a similar problem to #16803 in Spring AMQP (AMQP-701), I took a look to see if the same problem exists in the JMS BPP and noticed there is already a fix there by caching nonAnnotatedClasses.
I believe this Set could be cleared in afterSingletonsInstantiated(), freeing up the heap space for these references. It would be at the cost of 1 extra parse for scoped beans that have already been referenced during context initialization.
I took a slightly difference approach in Spring AMQP, by caching the annotated methods and their annotations (with non-annotated classes getting an empty MultiValueMap method list), so I had rather more data after context initialization than just a reference to most bean classes; hence I clear the cache when complete.
#16803 Scheduled/JmsListenerAnnotationBeanPostProcessor needlessly scans every scoped instance