Changed consumer/message processing to synchronized mode - former async (#4095)

[dao74]

resolves #4095

The asynchronous processing of incoming message can exhaust the database connection pool if there is a (big) lag in the kafka queue.
On other hand, the ACK of each message should perform after workflow is successfully created (or retried when the process fails) to not lose the kafka message.
This PR is fixing the synchronize processing and ACK issue.

Note:
Changing to synchronize will slowdown the whole process for larger amount of kafka messages. Therefore, the consumer should be increased - in the same count like the partition size of the topic.
Therefore the partition size should be configurable in the Kafka Activity.

[dao74]

@dotnet-policy-service agree

[dwoldo]

This also resolves #4192

[Snotax]

I'm not sure this fixes the problem correctly. With this implementation, the consumation speed is unusable for us(we're talking seconds per message). @dao74 What did you do to increase the speed or with what kind of data did you test this ?

We're thinking of reverting to the old version again.

FYI @sfmskywalker

[Snotax]

@dao74 Where did you even encounter your original error, we couldn't reproduce this?

[dao74]

Hello all, are we talking about side affects? We are good with this fix, I don't remember the details of our problem but would like to know what is wrong and why you need the revert.

[Snotax]

We are experiencing an unusable increase in time used to handle incoming messages. Like multiple seconds per message.

[dao74]

The reason for the slowness is not our fix; please check TriggerWorkflowsAsync method in the Worker class. You will find the following code which throttles the process flow: *await Task.Delay(_delay, cancellationToken);* We were not sure why it is necessary to have a sleep between the messages therefore we didn' touch this part. But in our internal tests and (load tests) we commented it out, with satisfied results. Please check whether we can waive the Task.Delay line.

[Snotax]

I know this line. But the increase in consumation time is significantly higher with your change.

[dao74]

I know this line. But the increase in consumation time is significantly higher with your change.

Of course, process time might be increased because of the sync mode (instead of async) but I am not sure how significant - we didn't compared or made a benchmark. The async processing brings other problems, therefore - in my opinion - it must be stay in sync mode.

We made the performance tests (bulk processing) without the delay line and we were satisfied with the result. You should do the same before making a decision.

[Snotax]

Ok, we will do some more performance testing next week. We'll keep you updated.