Add tracking for active node and task execution counts in propeller

[sshardool]

Tracking issue

#4593

Why are the changes needed?

Fixes this from the issue description

Currently the only ActiveNodeExecutions and ActiveTaskExecutions metrics exported are from flyteadmin. 
This can result in inaccuracies if, for example, flyteadmin restarts or there is an uncommon inconsistency in data and control plane state (ex. FlyteWorkflow CR manually deleted).

What changes were proposed in this pull request?

The metrics for active node and task executions are added to flytepropeller with the expectation that the workflow re-evaluation duration is an acceptable delay for updating the metrics (eventually consistent)

How was this patch tested?

Tested on a dev flyte setup (single node).
Tested is in progress in larger scale internal deployments.

Sample of the new metrics:

flyte:propeller:flyte_dev:execstats:active_node_executions 9
flyte:propeller:flyte_dev:execstats:active_task_executions 7
flyte:propeller:flyte_dev:execstats:active_workflow_executions 3

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

Attention: Patch coverage is 44.80519% with 85 lines in your changes are missing coverage. Please review.

Project coverage is 59.06%. Comparing base (37255a1) to head (38efa8d).
Report is 1 commits behind head on master.

Files	Patch %	Lines
...er/pkg/controller/workflowstore/execution_stats.go	32.60%	60 Missing and 2 partials ⚠️
...ller/pkg/controller/executors/execution_context.go	37.50%	10 Missing ⚠️
flytepropeller/pkg/controller/controller.go	0.00%	8 Missing ⚠️
flytepropeller/pkg/controller/workflow/executor.go	83.33%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4986      +/-   ##
==========================================
- Coverage   59.10%   59.06%   -0.04%     
==========================================
  Files         645      646       +1     
  Lines       55581    55714     +133     
==========================================
+ Hits        32851    32910      +59     
- Misses      20136    20208      +72     
- Partials     2594     2596       +2     

Flag	Coverage Δ
unittests	59.06% <44.80%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pvditt]

@sshardool this PR is great.

Have you noticed any increases in latency for workflow executions with this running? How many workers/threads are running for your propeller deployments? I figure this should only add in the order of nanoseconds for propeller loops with lock contention with ExecutionStatsHolder updates so it shouldn't be a concern.

We might want to run this on an internal cluster for a little bit prior to merging, but this PR LGTM.

[sshardool]

@sshardool this PR is great.

Have you noticed any increases in latency for workflow executions with this running? How many workers/threads are running for your propeller deployments? I figure this should only add in the order of nanoseconds for propeller loops with lock contention with ExecutionStatsHolder updates so it shouldn't be a concern.

We might want to run this on an internal cluster for a little bit prior to merging, but this PR LGTM.

Thanks for the review @pvditt . We currently have this running with 4 worker threads and have not seen any increase in execution latencies. You're right, critical section for all operations which are inline with the work-queue tasks is very small (single map entry update). Let me know if an additional metric tracking the wait duration (i.e time taken for mutex acquisition) in ExecutionStatsHolder:AddOrUpdateEntry() will help.

[pvditt]

thank you for adding this

[welcome]

Congrats on merging your first pull request! 🎉