feat: rebalance cost-based autoscaler for best throughput

[Fly-Style]

This PR retunes the cost-based autoscaler scoring for better throughput-oriented decisions and exposes idealIdleRatio as a configurable knob for the U-shaped idle-cost function.
It also updates autoscaler logging to show the effective idle ratio and max observed processing rate used during optimal task-count calculation.

Details

Picture 1

• Small supervisor case: 48 partitions, starting from 6 tasks.
• Shows that the new cost function scales up gradually as lag becomes more important, while higher observed idle ratios keep the recommendation conservative.
• Useful sanity check for avoiding aggressive jumps on smaller supervisors.

Picture 2

• Medium supervisor case: 125 partitions, plot title shows startTaskCount=25.
• Shows that low-idle/high-pressure cases recommend materially more tasks, while moderate or high idle ratios stay well below the partition count.
• Useful for checking that the scoring does not blindly chase one-task-per-partition when idle capacity is already available.

Picture 3

• Large supervisor case: 500 partitions, starting at the maximum task count.
• Shows the strongest separation between low-idle and high-idle scenarios: low idle can keep the recommendation near max capacity, while higher idle ratios strongly favor scale-down.
• Useful for validating that the idle-cost side still prevents waste at large partition counts.

This PR has:

• been self-reviewed.
• a release note entry in the PR description.

[kfaraz]

In conclusion, does the auto-scaler now prefer under-provisioning by default?

[kfaraz]

Why change this to 2.5 instead of 2.0?

[Fly-Style]

I tried 2.0, but 2.5 looked better in simulation.

[kfaraz]

I feel 2x penalty on over-provisioning is already enough. Changing these constants very frequently will make it difficult to gather enough data to refine the constants empirically.

[kfaraz]

Let's use 2 for now and see how it does in real clusters. This PR is already interchanging the under provisioned penalty and the over provisioning penalty, so we will have enough stuff to validate anyway.

[Fly-Style]

Okay, let's use 2, but I can assure you 2.5 looks even better on simulation. We'll see 😺

[kfaraz]

Why this change?
Technically, 1000 was also an arbitrary number but 5000 definitely seems to be on the higher side, especially when dealing with bulkier records with large (say JSON) column values. I would rather the user choose whether they want to prefer lag recovery or throughput by tweaking the weights.

[Fly-Style]

I decided to tweak it too and realized that 1000 is too permissible to scaleup during minimal lag. 5000 implies more strict behaviour in that critical state where we have not received metrics yet.

[kfaraz]

If we have not received metrics yet, auto-scaling would be skipped since CostBasedAutoScaler.validateMetricsForScaling would return an error.

I have personally seen tasks in prod clusters maxing out at 5000 records/sec when dealing with large records.
So, using a large MIN_PROCESSING_RATE would cause us to always under-estimate lagRecoveryTime, irrespective of the actual avgProcessingRate.
As such, let's keep a low value of MIN_PROCESSING_RATE(maybe even as low as 100), since it is meant to be a safe-side measure that kicks in only when avgProcessingRate is very low.

The penalty for scale-up is already driven by the optimal task idleness, and can be controlled using the weights.

[Fly-Style]

P.S. reverting

[kfaraz]

Thanks!

[kfaraz]

In a future PR, I think we can remove the MIN_PROCESSING_RATE altogether and maybe use the window maxProcessingRate, but I haven't fully thought it through yet. It might have some unforeseen side effects.

[Fly-Style]

@kfaraz I forgot about Math.max(...) and then I reconsider my approach 😁

[FrankChen021]

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Found 1 issue.

Reviewed 6 of 6 changed files.

---

This is an automated review by Codex GPT-5.5

[kfaraz]

Left a non-blocking comment.