Skip to content

Increase max stage concurrency#779

Merged
Andyz26 merged 1 commit intomasterfrom
andyz/increaseStageConcurrencyMax
Jul 29, 2025
Merged

Increase max stage concurrency#779
Andyz26 merged 1 commit intomasterfrom
andyz/increaseStageConcurrencyMax

Conversation

@Andyz26
Copy link
Collaborator

@Andyz26 Andyz26 commented Jul 23, 2025

Context

Test a larger max stage concurrency limit.

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable

@github-actions
Copy link

Test Results

658 tests  ±0   647 ✅ ±0   9m 8s ⏱️ +4s
151 suites ±0    11 💤 ±0 
151 files   ±0     0 ❌ ±0 

Results for commit f517e4a. ± Comparison against base commit 01e9286.

@github-actions
Copy link

Uploaded Artifacts

To use these artifacts in your Gradle project, paste the following lines in your build.gradle.

resolutionStrategy {
    force "io.mantisrx:mantis-client:0.1.0-20250723.202740-585"
    force "io.mantisrx:mantis-common-serde:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-common:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-common-akka:0.1.0-20250723.202740-20"
    force "io.mantisrx:mantis-discovery-proto:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-network:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-jm-akka:0.1.0-20250723.202740-14"
    force "io.mantisrx:mantis-runtime-autoscaler-api:0.1.0-20250723.202740-14"
    force "io.mantisrx:mantis-runtime-executor:0.1.0-20250723.202740-120"
    force "io.mantisrx:mantis-runtime-loader:0.1.0-20250723.202740-585"
    force "io.mantisrx:mantis-rxcontrol:0.1.0-20250723.202740-58"
    force "io.mantisrx:mantis-runtime:0.1.0-20250723.202740-585"
    force "io.mantisrx:mantis-shaded:0.1.0-20250723.202740-583"
    force "io.mantisrx:mantis-remote-observable:0.1.0-20250723.202740-585"
    force "io.mantisrx:mantis-connector-job-source:0.1.0-20250723.202740-36"
    force "io.mantisrx:mantis-connector-kafka:0.1.0-20250723.202740-585"
    force "io.mantisrx:mantis-testcontainers:0.1.0-20250723.202740-254"
    force "io.mantisrx:mantis-connector-iceberg:0.1.0-20250723.202740-583"
    force "io.mantisrx:mantis-control-plane-core:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-control-plane-server:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-control-plane-client:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-connector-publish:0.1.0-20250723.202740-584"
    force "io.mantisrx:mantis-examples-core:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-control-plane-dynamodb:0.1.0-20250723.202740-45"
    force "io.mantisrx:mantis-examples-groupby-sample:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-examples-sine-function:0.1.0-20250723.202740-577"
    force "io.mantisrx:mantis-examples-jobconnector-sample:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-examples-twitter-sample:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-examples-mantis-publish-sample:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-publish-netty:0.1.0-20250723.202740-577"
    force "io.mantisrx:mantis-examples-wordcount:0.1.0-20250723.202740-577"
    force "io.mantisrx:mantis-examples-synthetic-sourcejob:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-publish-netty-guice:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-server-agent:0.1.0-20250723.202740-577"
    force "io.mantisrx:mantis-server-worker-client:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-publish-core:0.1.0-20250723.202740-577"
    force "io.mantisrx:mantis-source-job-publish:0.1.0-20250723.202740-578"
    force "io.mantisrx:mantis-source-job-kafka:0.1.0-20250723.202740-578"
}

@Andyz26 Andyz26 merged commit be1148f into master Jul 29, 2025
5 checks passed
@Andyz26 Andyz26 deleted the andyz/increaseStageConcurrencyMax branch July 29, 2025 18:34
andresgalindo-stripe pushed a commit to andresgalindo-stripe/mantis that referenced this pull request Oct 30, 2025
andresgalindo-stripe added a commit that referenced this pull request Nov 6, 2025
* Add variety of cleanups, fix warnings, improve code/performance (#771)

* More fixes

* Review feedback, add more

* Update nebula.netflixoss use sonatype central portal (#774)

* Use com.netflix.nebula.netflixoss 11.6.0 to move publishing to Sonatype Central Portal from Sonatype Legacy OSSRH

* Github action: checkout v4

* Introduce batching into worker discovery during scaling (#773)

* Fix worker state filtering and scheduling update gaps during scaling. This reduces scaling update storms from N individual updates to 1-3 batched updates.
  - Filter JobSchedulingInfo to only include Started
  workers, preventing downstream connection failures
  - Add smart refresh batching with pending worker
  detection to avoid premature flag resets
  - Implement WorkerState.isPendingState() helper for
   consistent state checking
  - Add comprehensive tests covering scaling
  scenarios and flag reset edge cases
  - Include detailed context and analysis documentation of
  connection mechanisms and scaling optimizations

* try stablize flaky ut

* add analysis context doc

* remove refresh discovery trigger on scaleup request

* Fix Worker Request flow to properly use batching (#775)

* Introduce batching into worker discovery during scaling (#773)

* Fix worker state filtering and scheduling update gaps during scaling. This reduces scaling update storms from N individual updates to 1-3 batched updates.
  - Filter JobSchedulingInfo to only include Started
  workers, preventing downstream connection failures
  - Add smart refresh batching with pending worker
  detection to avoid premature flag resets
  - Implement WorkerState.isPendingState() helper for
   consistent state checking
  - Add comprehensive tests covering scaling
  scenarios and flag reset edge cases
  - Include detailed context and analysis documentation of
  connection mechanisms and scaling optimizations

* try stablize flaky ut

* add analysis context doc

* remove refresh discovery trigger on scaleup request

* Fix Worker Request flow to properly use batching (#775)

* Support default tag config as fallback on artifact loading failure (#778)

* increase max stage concurrency (#779)

* Fix a typo in the Group By docs (#783)

* Fix a typo in the Group By docs

* Fix broken link to heartbeat documentation

* Handle out of sync restarted TE (#784)

* Handle out of sync restarted TE

* use terminte event on heartbeat

* clean up + tests

* Revert "Fix Worker Request flow to properly use batching (#775)" (#785)

This reverts commit 3b0c92f.

* Move common code to utils and cleanup (#789)

Co-authored-by: ggao <ggao@netflix.com>

* Add job id to log and add running worker failure metrics (#790)

Co-authored-by: ggao <ggao@netflix.com>

* add job clusters update metrics (#791)

* Update worker failure metric (#792)

Co-authored-by: ggao <ggao@netflix.com>

* Refactor RCActor props overload (#795)

Co-authored-by: ggao <ggao@netflix.com>

* Add log to check #TE archived was not in disabled state (#793)

Co-authored-by: ggao <ggao@netflix.com>

* Update CODEOWNERS (#796)

* Cleanup autoscaler metric subscriptions on shutdown (#798)

* fix leaked auto scaler instance (#801)

* Fix test race condition (#803)

When disabling a job cluster, the response would sometimes return before
the associated jobs were killed.  The Delete action would then fail
because the job was still active.  I was able to reliably reproduce by
adding a 200ms sleep in the JobActor.onJobKill.

To fix, we just check if the response is returning that error. If so, we
retry.  Otherwise, we perform the standard checks.

[CI Example](https://github.com/Netflix/mantis/pull/797/checks?check_run_id=52633941202)

* Fixed up test

* Debugging

* Validating breakage is from rate limiting

* Updating rate limit

---------

Co-authored-by: Michael Braun <n3ca88@gmail.com>
Co-authored-by: OdysseusLives <achipman@netflix.com>
Co-authored-by: Andy Zhang <87735571+Andyz26@users.noreply.github.com>
Co-authored-by: Daniel Trager <43889268+dtrager02@users.noreply.github.com>
Co-authored-by: eliot-stripe <58606410+eliot-stripe@users.noreply.github.com>
Co-authored-by: Gigi Gao <ggjbetty@gmail.com>
Co-authored-by: ggao <ggao@netflix.com>
Co-authored-by: timmartin-stripe <131782471+timmartin-stripe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants