Development: Add safeguards against split Hazelcast clusters

[Hialus]

Checklist

General

• I tested all changes and their related features with all corresponding user types on a test server.
• This is a small issue that I tested locally and was confirmed by another developer on a test server.
• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Server

• Important: I implemented the changes with a very good performance and prevented too many (unnecessary) database calls.
• I strictly followed the server coding and design guidelines.
• I added multiple integration tests (Spring) related to the features (with a high test coverage). Not possible via JUnit
• I documented the Java code using JavaDoc style.

Motivation and Context

Currently when starting Artemis it can happen that the Hazelcast cluster is not created properly and some nodes form separate cluster of varying sizes (1+). There is currently no way to fix this, except to restart the affected nodes. The prevention for this so far was to just start the nodes one-by-one with a delay of 20 seconds. This made deployment tedious.

Description

I added 2 changes to address this:

1. I added a scheduled task to regularly check if the service registry contains hosts that are not in the Hazelcast cluster and then add those
2. I added a basic configuration for split brain handling. This can help to resolve if the cluster split during runtime into multiple clusters.

As a small side-quest I also added the possibility for nodes to have different Hazelcast ports between one another. This could theoretically replace the old local dev cluster handling, though I left it in place for now.

Steps for Testing

Scenario 1

Prerequisites:

• 1 Admin account on a multi node test server (ideally TS3)

1. Deploy to the test server
2. Go to the admin health page (via the "Server Administration" dropdown in the navbar)
3. Click on the eye on the right for the "Hazelcast" row
4. If there are 3 members: You are done
5. If not, please wait a minute, refresh the page and start at 3. again
6. It can take several minutes until the cluster heals itself, as Hazelcast only checks for a split brain every 2 minutes

Scenario 2

Prerequisites:

• Do this on Staging ICL ideally

1. Deploy to staging primary and directly after (possibly even while it is still running) secondary (it is important to start them in quick succession to break the cluster on startup)
2. Do some stuff that stores stuff the Hazelcast cluster (e.g. continously trigger a lot of builds on an ICL server)
3. In the beginning not all builds will be processed (You can check this in the build queue of the course by contionously reloading to get the states of the different nodes)
4. After the cluster merged (maybe check in the admin area) the builds will be processed and the build queue is the same across instances

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked

Review Progress

Performance Review

• I (as a reviewer) confirm that the server changes (in particular related to database calls) are implemented with a very good performance

Code Review

• Code Review 1
• Code Review 2

Manual Tests

• Test 1
• Test 2

Screenshots

How the health panel should look like:

Summary by CodeRabbit

• New Features

  • Enhanced cluster connectivity checks with scheduled tasks in the system configuration.
  • Implemented split brain protection for improved system resilience.

• Style

  • Reformatted code indentation in BuildJobContainerService to improve readability.

[krusche]

Code changes look good to me 👍 great work!

[pzdr7]

Tested on TS3 (cluster was split, but healed after a while) and LocalCI staging (I encountered two build queue states that resolved eventually)

[coderabbitai]

Actionable comments posted: 1

Out of diff range and nitpick comments (4)

src/main/java/de/tum/in/www1/artemis/config/CacheConfiguration.java (4)

Line range hint 158-252: The configuration of the Hazelcast instance is comprehensive, covering various scenarios like local and production environments. However, consider documenting the implications of disabling multicast and auto-detection, as well as the choice of TCP/IP config, to ensure clarity for future maintainers.

---

Line range hint 253-261: The configuration of the Hazelcast queue for local CI environments is appropriate. Consider adding more detailed comments explaining the choice of backup count and the role of the priority comparator class for better understanding.

---

Line range hint 262-267: The serializer configuration for Path objects is crucial for caching files in the Hazelcast cluster. Consider adding comments to explain its importance and the choice of serializer implementation for future maintainers.

---

Line range hint 268-270: The configuration of the key generator using Git and Build properties is a good practice for distinguishing cache entries. Consider documenting the rationale behind using prefixed keys to enhance clarity for future maintainers.

[bensofficial]

Reapprove

[krusche]

Code looks good to me 👍