Embedded Hazelcast in Kubernetes environment won't detect members anymore after latest release #24688
Labels
Comments
OmarHawk
changed the title
|
If you need more logs (i.e. on debug level), please let me know. I may have to strip some hopefully not so important information from them then though. ;-) |
|
@OmarHawk - we're running into a very similar issue with a later version. Looking at the KubernetesAPIProvider, it appears to be searching for a k8s port named "hazelcast-service-port", which is not a valid port name for our version of k8s. I'll keep an eye out on this issue, we were going to open up a new ticket. If I do, I'll link it here. |
|
Hello @OmarHawk, could you share your Hazelcast configuration (in particular networking configuration) and the K8s deployment/services spec? |
|
Hi, The strange thing about this is - on the very same cluster, we do have applications with hazelcast 5.1.6 with the very same kind of k8s service + same kind of hazelcast config in the service and one of them does work and the other one does not work. As if maybe the used k8s APIs return the ports sometimes in a different order than specified in the service descriptor (therefore luckily finding the working port first) - or the ports being handled somewhere in an unordered List / Set inbetween, therefore running only 'sometimes' into the same problem as @bwmeier has. I'll provide further information tomorrow, but I'd assume, it is the very same problem as @bwmeier has. |
|
@OmarHawk, this is definitely the same issue. But we would like to see how your config is different, to provide a good fix for it. |
|
Ok, here you are. This is the part of the log output which shows the current com.hazelcast.config.Config - mostly defaults actually. ;-) We do have following additional system properties set: Then, here is our deployment config (the containers ports part; left away the rest, hope, this is enough): Then here the Service configuration - we do have one specific Service descriptor just for the Hazelcast ports: And a second one for the application itself and its actual Ingress, but I guess, this won't play a role... |
SeriyBg
added a commit
that referenced
this issue
Jun 19, 2023
Change the port name from `hazelcast-service-port` that is longer than 15 characters to `hazelcast`. Fixes #24688 Fixes #24705 Breaking changes (list specific methods/types/messages): * API * client protocol format * serialized form * snapshot format Checklist: - [+] Labels (`Team:`, `Type:`, `Source:`, `Module:`) and Milestone set - [+] Label `Add to Release Notes` or `Not Release Notes content` set - [+] Request reviewers if possible - [+] Send backports/forwardports if fix needs to be applied to past/future releases - [+] New public APIs have `@Nonnull/@Nullable` annotations - [+] New public APIs have `@since` tags in Javadoc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Hi,
we do get Hazelcast in embedded mode as part of a Spring Boot transitive dependency, and after performing the last update (Hazelcast from 5.1.5 to 5.1.6), we noticed that the pods failed to detect each other. The only difference seems to be the version change. Since I didn't see any hint regarding this in the release notes (apart from maybe #24044 or maybe #23932), I wonder, if there needs to be something configured differently now.
Failing log output - see this blacklist entry at the end:
Working log output:
Both logs were produced on the very same kubernetes cluster... Maybe, it is something related to the ports, as the failing one mentions 5702 while the succeeding one (with the older version) mentions 5701 as the target...
Expected behavior
5.1.5 and 5.1.6 should behave the same, when configured the same
To Reproduce
Steps to reproduce the behavior:
Additional context
Spring Boot 3.0.6->3.0.7 update
JDK 17
The text was updated successfully, but these errors were encountered: