Multus interfaces cannot communicate with each other when pods are on multiple nodes

[MichalzDev]

Hi, I have scenario on 3 node cluster, where node1 is for UE+gNB, node2 is for 5GCore, node3 for UPF only.
In that scenario pods cannot see (ping) each other multus interfaces.
Eg. from UPF I cannot ping SMF N4 (10.100.50.244) nor gNB N3 unless there are on the same node.
When everything is on 1 node it works flawlessly. When I spited UERANSIM (node1) and free5gc (node2) I managed to pair AMF and gNB by enabling ngap and changing N2 in gNB yaml to newly created service IP but UE still cannot communicate via tunnel due to issue from above.
Is it possible to create similar services to other pods/interfaces ? Or maybe do you know other way to make to make it work ?
Thanks for help

[raoufkh]

Hello

Exposing other interfaces with K8s services is not possible due to the configuration template of Free5GC network functions. For your implementation, did you check that the promiscuous mode on MACVLAN master interfaces is enabled?

[MichalzDev]

I enabled promiscuous mode and it worked, thanks :)
However now I have situation when UPF is on its own node, SMF throws me this error

UPF can ping SMF (10.100.50.244) and SMF can also ping UPF (10.100.50.241).
When I place UPF on the same node as other 5G Core functions, everything works as intended.
Have you encountered this situation before ?

[raoufkh]

At least the problem is no longer in the network. Can you publish the entire SMF and UPF logs please? I'm not sure that they are well associated

[MichalzDev]

I managed to somehow fix this issue. What I did was:

• I set up promisc mode on interface in crontab at reboot
• rebooted all hosts
• Uninstalled and installed both Helm charts

[raoufkh]

Great I'll close this issue