Refactor logic to drain queues after each task

[sergey-koryshev]

Description of issue:
Sometimes task successfully completes but it's marked as failed on ADO side. It was found that logic related to queues draining affects this. Possibly the issue occurred due to the fact that we call methods ProcessWebConsoleLinesQueueAsync, ProcessFilesUploadQueueAsync and ProcessTimelinesUpdateQueueAsync while the methods are called in the beginning of job and being executed in additional thread till job's completion. So additional calls of the methods can interfere with executing ones and probably something goes wrong with updating timeline records.

Description of fix:
I have refactored logic to drain queues. I removed extra call of the methods and implemented two additional fields-flags ForceDrainWebConsoleQueue and ForceDrainTimelineQueue. If the properties are set then web console and timeline queues will be drained as soon as possible by methods ProcessWebConsoleLinesQueueAsync and ProcessTimelinesUpdateQueueAsync respectively which are being already executed during a job. When draining is completed the properties will be set to false automatically. I also kept knob AGENT_DISABLE_DRAIN_QUEUES_AFTER_TASK to be able to disable this logic in case any problems.

How it was tested:
There is no way to reproduce the issue and test it locally. So only original issue with hang UI was tested. Since customers mitigated the issue by enabling FF AGENT_DISABLE_DRAIN_QUEUES_AFTER_TASK, we will disable it in production on a few scale units and see if issue be back or not.

There is a pipeline which can be used to test if this refactored logic still fixes original issue with hang UI: https://abtttestorg.visualstudio.com/Other/_build/results?buildId=6432&view=results

Behavior with disabled AGENT_DISABLE_DRAIN_QUEUES_AFTER_TASK:

Behavior with enabled AGENT_DISABLE_DRAIN_QUEUES_AFTER_TASK:

[mkonjikovac]

When draining queues, agent is hitting ADO timeline record update endpoint and stating that the task result is probably failed (since we are having this issue), even though it's successful.

This is where the agent makes update timeline record call to ADO fom the draining queue logic:

What we're doing here now is reordering these timeline record updates, so that first we'll have draining queue logic which will send us Failed result for some hanging task, and then we'll continue with the Complete part of the logic where we will send the Successful result for the same task. So we will be updating the result of the task twice, Failed and then Successful (in this case which has caused the issue for some of our customers).

I think that we need to add a condition before draining queus logic in order to check whether a task should really be marked as failed. So check the Complete status of the task result and then in the Draining queues update the logic so that we don't update the same timeline record with a totally different result (in a way skip the update).

[KonstantinTyukalov]

I'll take a closer look at the logic soon