Single binary Loki with memberlist config

[ahmedTouati]

Describe the bug
In Loki there are three modes monolithic, simple scalable and microservices modes, and you can run single binary loki as Horizontally scalable also according to architecture docs:
**Horizontally scale up a monolithic mode deployment to more instances by using a shared object store, and by configuring the memberlist_config section 1 to share state between all instances.

High availability can be configured by running two Loki instances using memberlist_config configuration and a shared object store.**

So i followed this example configuration for memberlist config (Examples | Grafana Labs) in a single binary loki installation and increased replicas to two (loki-0,loki-1).
For join_members: i used loki headless service (loki-headless.loki.svc.cluster.local:3100) since loki-gossip-ring service is not available in this installation, but loki always fail to resolve it:

ts=2022-03-15T09:14:24.010697792Z caller=memberlist_logger.go:74 level=warn msg="Failed to resolve loki-headless.loki.svc.cluster.local:3100: lookup loki-headless.loki.svc.cluster.local on 172.20.0.10:53: no such host"

Environment:

• Infrastructure: EKS
• Deployment tool: helm
• Loki: v2.3.0

[chaudum]

Hey @ahmedTouati

It appears that you are using the wrong port for the join_members setting: loki-headless.loki.svc.cluster.local:3100 <- you specified 3100, which is the HTTP port. However, you need to use the memberlist port, which is 7946 (see your linked docs example).

[ahmedTouati]

Hey @ahmedTouati

It appears that you are using the wrong port for the join_members setting: loki-headless.loki.svc.cluster.local:3100 <- you specified 3100, which is the HTTP port. However, you need to use the memberlist port, which is 7946 (see your linked docs example).

No i used bind_port: 7946 then loki-headless.loki.svc.cluster.local:7946 but the port still not exposed