Skip to content

Add variety of cleanups, fix warnings, improve code/performance#771

Merged
Andyz26 merged 2 commits intomasterfrom
michaelbraun/variety-of-cleanups-fix-warnings-improvements
May 9, 2025
Merged

Add variety of cleanups, fix warnings, improve code/performance#771
Andyz26 merged 2 commits intomasterfrom
michaelbraun/variety-of-cleanups-fix-warnings-improvements

Conversation

@michaelbraun
Copy link
Collaborator

Context

Found a number of opportunities in code to fix warnings, simplify code. Changed code to apply these fixes

Checklist

  • ./gradlew build compiles code correctly
  • Added new tests where applicable
  • ./gradlew test passes all tests
  • Extended README or added javadocs where applicable

});

streamEventList.stream()
.map(e -> process(stream, e))
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process(...) does side effects, so this shouldn't be done in map

* @return
*/
Set<Subscription> getStreamSubscriptions(final String streamName) {
return streamSubscriptionsMap.getOrDefault(streamName, new ConcurrentSkipListSet<>());
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was creating a new ConcurrentSkipListSet every time if not found.

@github-actions
Copy link

github-actions bot commented May 9, 2025

Test Results

150 files  ±0  150 suites  ±0   8m 56s ⏱️ +15s
650 tests ±0  639 ✅  - 1  10 💤 ±0  1 ❌ +1 
651 runs  +1  640 ✅ ±0  10 💤 ±0  1 ❌ +1 

For more details on these failures, see this check.

Results for commit 510bb85. ± Comparison against base commit b3b85bc.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented May 9, 2025

Uploaded Artifacts

To use these artifacts in your Gradle project, paste the following lines in your build.gradle.

resolutionStrategy {
    force "io.mantisrx:mantis-client:0.1.0-20250509.190127-577"
    force "io.mantisrx:mantis-common-akka:0.1.0-20250509.190127-12"
    force "io.mantisrx:mantis-common-serde:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-common:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-jm-akka:0.1.0-20250509.190127-6"
    force "io.mantisrx:mantis-network:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-runtime-autoscaler-api:0.1.0-20250509.190127-6"
    force "io.mantisrx:mantis-runtime:0.1.0-20250509.190127-577"
    force "io.mantisrx:mantis-runtime-executor:0.1.0-20250509.190127-112"
    force "io.mantisrx:mantis-discovery-proto:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-rxcontrol:0.1.0-20250509.190127-50"
    force "io.mantisrx:mantis-shaded:0.1.0-20250509.190127-575"
    force "io.mantisrx:mantis-testcontainers:0.1.0-20250509.190127-246"
    force "io.mantisrx:mantis-runtime-loader:0.1.0-20250509.190127-577"
    force "io.mantisrx:mantis-connector-kafka:0.1.0-20250509.190127-577"
    force "io.mantisrx:mantis-remote-observable:0.1.0-20250509.190127-577"
    force "io.mantisrx:mantis-connector-job-source:0.1.0-20250509.190127-28"
    force "io.mantisrx:mantis-control-plane-client:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-connector-iceberg:0.1.0-20250509.190127-575"
    force "io.mantisrx:mantis-control-plane-server:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-control-plane-core:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-connector-publish:0.1.0-20250509.190127-576"
    force "io.mantisrx:mantis-examples-groupby-sample:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-examples-jobconnector-sample:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-control-plane-dynamodb:0.1.0-20250509.190127-37"
    force "io.mantisrx:mantis-examples-synthetic-sourcejob:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-examples-sine-function:0.1.0-20250509.190127-569"
    force "io.mantisrx:mantis-examples-core:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-examples-mantis-publish-sample:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-publish-core:0.1.0-20250509.190127-569"
    force "io.mantisrx:mantis-examples-twitter-sample:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-publish-netty-guice:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-examples-wordcount:0.1.0-20250509.190127-569"
    force "io.mantisrx:mantis-publish-netty:0.1.0-20250509.190127-569"
    force "io.mantisrx:mantis-server-worker-client:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-source-job-kafka:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-source-job-publish:0.1.0-20250509.190127-570"
    force "io.mantisrx:mantis-server-agent:0.1.0-20250509.190127-569"
}

@michaelbraun michaelbraun force-pushed the michaelbraun/variety-of-cleanups-fix-warnings-improvements branch from bb6ec04 to cb80aad Compare May 9, 2025 17:27
@michaelbraun michaelbraun had a problem deploying to Integrate Pull Request May 9, 2025 17:27 — with GitHub Actions Failure
@michaelbraun michaelbraun force-pushed the michaelbraun/variety-of-cleanups-fix-warnings-improvements branch from cb80aad to e70708d Compare May 9, 2025 17:34
@michaelbraun michaelbraun had a problem deploying to Integrate Pull Request May 9, 2025 17:37 — with GitHub Actions Failure
Copy link
Contributor

@hmitnflx hmitnflx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking the time and making a contribution to mantis.
The changes look good to me in a broad sense. I left a minor comments.

if (status >= 200 && status < 300) {
AppJobClustersMap appJobClustersMap = serializer.fromJSON(response.entityAsString(), AppJobClustersMap.class);
logger.debug(appJobClustersMap.toString());
logger.debug("{}", appJobClustersMap);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: while you are here, could you please add a helpful message in the log?

@Override
public int hashCode() {
return Objects.hash(getSubscriptionList());
return Objects.hashCode(getSubscriptionList());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this changes the hashCode implementation slightly. I dont think mantis has any usages where the hashCode is serialized but it'd still be nice to keep it unchanged.

@michaelbraun michaelbraun temporarily deployed to Integrate Pull Request May 9, 2025 18:14 — with GitHub Actions Inactive
@michaelbraun michaelbraun temporarily deployed to Integrate Pull Request May 9, 2025 18:49 — with GitHub Actions Inactive
@Andyz26 Andyz26 merged commit 21683f2 into master May 9, 2025
6 of 7 checks passed
@Andyz26 Andyz26 deleted the michaelbraun/variety-of-cleanups-fix-warnings-improvements branch May 9, 2025 21:30
andresgalindo-stripe pushed a commit to andresgalindo-stripe/mantis that referenced this pull request Oct 30, 2025
andresgalindo-stripe added a commit that referenced this pull request Nov 6, 2025
* Add variety of cleanups, fix warnings, improve code/performance (#771)

* More fixes

* Review feedback, add more

* Update nebula.netflixoss use sonatype central portal (#774)

* Use com.netflix.nebula.netflixoss 11.6.0 to move publishing to Sonatype Central Portal from Sonatype Legacy OSSRH

* Github action: checkout v4

* Introduce batching into worker discovery during scaling (#773)

* Fix worker state filtering and scheduling update gaps during scaling. This reduces scaling update storms from N individual updates to 1-3 batched updates.
  - Filter JobSchedulingInfo to only include Started
  workers, preventing downstream connection failures
  - Add smart refresh batching with pending worker
  detection to avoid premature flag resets
  - Implement WorkerState.isPendingState() helper for
   consistent state checking
  - Add comprehensive tests covering scaling
  scenarios and flag reset edge cases
  - Include detailed context and analysis documentation of
  connection mechanisms and scaling optimizations

* try stablize flaky ut

* add analysis context doc

* remove refresh discovery trigger on scaleup request

* Fix Worker Request flow to properly use batching (#775)

* Introduce batching into worker discovery during scaling (#773)

* Fix worker state filtering and scheduling update gaps during scaling. This reduces scaling update storms from N individual updates to 1-3 batched updates.
  - Filter JobSchedulingInfo to only include Started
  workers, preventing downstream connection failures
  - Add smart refresh batching with pending worker
  detection to avoid premature flag resets
  - Implement WorkerState.isPendingState() helper for
   consistent state checking
  - Add comprehensive tests covering scaling
  scenarios and flag reset edge cases
  - Include detailed context and analysis documentation of
  connection mechanisms and scaling optimizations

* try stablize flaky ut

* add analysis context doc

* remove refresh discovery trigger on scaleup request

* Fix Worker Request flow to properly use batching (#775)

* Support default tag config as fallback on artifact loading failure (#778)

* increase max stage concurrency (#779)

* Fix a typo in the Group By docs (#783)

* Fix a typo in the Group By docs

* Fix broken link to heartbeat documentation

* Handle out of sync restarted TE (#784)

* Handle out of sync restarted TE

* use terminte event on heartbeat

* clean up + tests

* Revert "Fix Worker Request flow to properly use batching (#775)" (#785)

This reverts commit 3b0c92f.

* Move common code to utils and cleanup (#789)

Co-authored-by: ggao <ggao@netflix.com>

* Add job id to log and add running worker failure metrics (#790)

Co-authored-by: ggao <ggao@netflix.com>

* add job clusters update metrics (#791)

* Update worker failure metric (#792)

Co-authored-by: ggao <ggao@netflix.com>

* Refactor RCActor props overload (#795)

Co-authored-by: ggao <ggao@netflix.com>

* Add log to check #TE archived was not in disabled state (#793)

Co-authored-by: ggao <ggao@netflix.com>

* Update CODEOWNERS (#796)

* Cleanup autoscaler metric subscriptions on shutdown (#798)

* fix leaked auto scaler instance (#801)

* Fix test race condition (#803)

When disabling a job cluster, the response would sometimes return before
the associated jobs were killed.  The Delete action would then fail
because the job was still active.  I was able to reliably reproduce by
adding a 200ms sleep in the JobActor.onJobKill.

To fix, we just check if the response is returning that error. If so, we
retry.  Otherwise, we perform the standard checks.

[CI Example](https://github.com/Netflix/mantis/pull/797/checks?check_run_id=52633941202)

* Fixed up test

* Debugging

* Validating breakage is from rate limiting

* Updating rate limit

---------

Co-authored-by: Michael Braun <n3ca88@gmail.com>
Co-authored-by: OdysseusLives <achipman@netflix.com>
Co-authored-by: Andy Zhang <87735571+Andyz26@users.noreply.github.com>
Co-authored-by: Daniel Trager <43889268+dtrager02@users.noreply.github.com>
Co-authored-by: eliot-stripe <58606410+eliot-stripe@users.noreply.github.com>
Co-authored-by: Gigi Gao <ggjbetty@gmail.com>
Co-authored-by: ggao <ggao@netflix.com>
Co-authored-by: timmartin-stripe <131782471+timmartin-stripe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants