[rayci] Upload pipeline in batches when job count exceeds Buildkite limit

[andrew-anyscale]

Buildkite rejects pipeline uploads with more than 500 jobs

buildkite-agent: fatal: Failed to upload and process pipeline: Pipeline upload rejected: The number of jobs in this upload exceeds your organization limit of 500. Please break the upload into batches below this limit, or contact support to discuss an increase

When the pipeline exceeds this limit (counting parallelism-expanded jobs), split it into batches by group and upload each batch separately. If any single group exceeds 500 jobs, error with a message to split the group.

Topic: upload-batches
Signed-off-by: andrew andrew@anyscale.com

[andrew-anyscale]

Reviews in this chain:
└#483 [rayci] Upload pipeline in batches when job count exceeds Buildkite limit

[andrew-anyscale]

#	head	base	diff	date	summary
0	db2a0e7f	18233682	diff	Apr 7 10:55 AM	3 files changed, 233 insertions(+), 3 deletions(-)
1	b5817af4	18233682	diff	Apr 7 12:34 PM	1 file changed, 3 insertions(+), 9 deletions(-)
2	89784b75	18233682	diff	Apr 7 12:38 PM	1 file changed, 15 insertions(+), 25 deletions(-)

[gemini-code-assist]

Code Review

This pull request implements pipeline batching for Buildkite uploads to handle job limits. It adds functionality to calculate total jobs (including parallelism) and split pipelines into batches, ensuring the 'Notify' configuration is preserved in the first batch. Comprehensive tests were added to verify batching behavior. The reviewer suggested simplifying the implementation in main.go by always using the batching logic to avoid code duplication between the single-batch and multi-batch cases.