Dynamic Integration flow Registration causing performance degradation

[pinkeshsagar-harptec]

Time taken by dynamically registering new integration flow is increasing as more flows are being registered. Each flow has unique flow id passed.

Lets say we have a flow which contains 20 different beans,

Initially, when application starts, registering new flows takes less time for instance 10 seconds
and as more and more flows are registered, time is increasing significantly and at last it is around 30 minutes.

Note that, All these flows are registered in parallel

We are expecting this time to be same or vary slightly as new beans are being added to the context.

Here is link to stakoverflow question

https://stackoverflow.com/questions/69657739/dynamic-integration-flow-registration-causing-performance-degradation/69663343?noredirect=1#comment123140594_69663343

[artembilan]

Hi @pinkeshsagar-harptec !

Please, find a PR on the matter: #3654.

It would be great if you can try your solution with a 5.5.6-SNAPSHOT based on that PR.
You need to pull the project locally, switch that my GH-3653 branch and run ./gradlew publishToMavenLocal -x test.
Then use a spring-integration-core-5.5.6-SNAPSHOT from your mavenLocal() to test your project to see a performance difference.

On the other hand it would be great if you share more about your dynamic flows to let us to build some unit-test to measure performance.

Thank you!

[pinkeshsagar-harptec]

Hi, thank you for looking into the issue.

We tried with latest code changes from your specified branch and performance is improved a bit, i would say 5%.

Regarding dynamic flows, here is an example,

Rss Feed > Store(DB) > Select keyword(DB) > Text Match > Store Matched News(DB) > Email send > WebSocket publish

Description of the flow:-

1. fetch rss feeds
2. save it to mysql db
3. select keywords from mysql db (1000+ keywords)
4. match news from rss feeds with keywords selected
5. save matched news to mysql db
6. send matched news in email
7. publish matched news on websocket

Every output of each flow is connected to input of next flow and direct channel is used between all of them.

Test 300 - 500 flow of same type parallelly with different rss feeds as input to check the performance.

[artembilan]

I wonder if this your SO question is related somehow to your measure of the performance: https://stackoverflow.com/questions/69663753/dynamically-registered-beans-are-unresponsive-after-some-days.
Which is already not about a registration phase, but runtime execution.
So, it is indeed possible to have some degradation at runtime for that 300 - 500 number of flow since there is really not enough threads to process them in parallel. More over it won't be possible in the end to have a good performance for big number of thread since all of them lend on the CPU process anyway.

Another observation for your flow description:
It looks like that dynamic part over there is really just an RSS Inbound Channel Channel Adapter because of static RSS URL definition. The rest of the flow seems the same and can be declared as a single IntegrationFlow bean starting with some channel. That channel can be used as an output of that dynamic flow just via .channel("processRssChannel"). This way you would drastically increase dynamic flow registration since not too many beans have to be processed at runtime. See more info in docs how to declared channels in the IntegrationFlow: https://docs.spring.io/spring-integration/docs/current/reference/html/dsl.html#java-dsl-channels.

Anyway I would like see your dynamic flow definition to determine if there are some beans which could be filtered properly in the framework to avoid the mentioned in SO question this.beanFactory.getBeansOfType() overhead.

[pinkeshsagar-harptec]

Regrading dynamic flow registration, below link shows pictorial representation of a sample flow which i mentioned above

Regarding the reusability of this flow, that is not possible because each flow has different components
doing different tasks and a flow contains any combination of components so reusability is not possible.

Here, component is separate integration flow which has direct channel as input and output.

Besides that, example was just illustration of how dynamic integration flow registration can be
used to create a flow with different tasks independent of others.

As per the documentation, to register the flow, unique id should be passed so that it does not do extra
checks but still it is taking time so what is benefit of dynamic registration.

If you create such kind of dynamic flows and test them then you will get the idea.

In real world scenario, request to create flows will be parallelly that's why we are doing these to check if it can handle that efficiently or not.

[artembilan]

Thanks for this explanation.

But still not clear what is a business goal to write such a complex logic on the fly?

May I see a couple code snippets of those fully different dynamic flows and how you register them?

Maybe some part of those flows is really resusable, like that email sending and webscoket broadcasting?..

[artembilan]

You know I have just spotted another piece of logic where I don't skip NamedComponent from scanning when it does not have a beanName.
Please, consider to pull the latest changes from that my branch, rebuild it, install locally and try your solution again.

Thanks

[pinkeshsagar-harptec]

Thank you so much for resolving the query. After running code on latest changes, we have seen significant
improvement in time.

Description about the test:

Each sample flow contains multiple components(same or unique) executing some task.
Now, Length of the sample flow vary from minimum 5 components to max 120 components.
We have tested these variations of flows.

Result:

Initially, when there are no flows registered, It used to take 25-30 seconds to register a flow with 120 components but after you made changes, it took 15 seconds, which is Hugh improvement.

Now, as more and more flows are being registered, registration time increases linearly but keep fluctuating.

At last, when 220+ sample flow are registered, performance is drastically degraded like 13 minutes then 23 minutes and at last 48 minutes.

I don't know why it is happened? I am attaching system config on which we tested the flow.

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   43 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              1
Core(s) per socket:              1
Socket(s):                       8
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           63
Model name:                      Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Stepping:                        2
CPU MHz:                         2294.686
BogoMIPS:                        4589.37
Hypervisor vendor:               VMware
Virtualisation type:             full
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        200 MiB
NUMA node0 CPU(s):               0-7

[artembilan]

Thanks for confirmation!
I think for the current ticket the work done is enough.
Everything else you describe is out of scope not only Spring Integration and its dynamic flow registration, but the whole JVM.
I mean it is too heavy for only 8 CPUs to do so much work you request. I remember that you do register in parallel, plus you mention a scheduling for those RSS Inbound Channel Adapters.
Therefore 220+ integration flows may lead to 220*100 beans in the application context where 100 at minimum relies on a scheduling.
This way you have too much stress for CPUs just from a single application.
Also I guess there is a good GC impact as well.

I know this all sounds like a clear personal assumption since I don't bring any test results to confirm, but everything has a limit and going to stuck eventually with constant heavy load...

You probably need to think about changing an architecture of your solution and move it into (for example) Kubernetes with pods generations on the fly with those particular IntegrationFlows as individual Microservices.