store probably dont handle TLS connection to azure redis cache over 6380 port.

[dmilind]

Thanos, Prometheus and Golang version used:
Thanos - 0.26.0,0.27.0 and 0.28.1

Object Storage Provider:
azure container storage

What happened:
In my setup I am using azure cache for redis with 6380 (TLS 1.2). When store is trying to connect to azure cache over 6380, store errors out with connection reset by peer mostly. Whereas it works like a charm on 6379 port when tls is disabled. During investigation we found out that , messages are in plain text instead of being encrypted and because of that azure redis is sending reset signal to close down the connection.

What you expected to happen:
Store should connect to azure redis cache over 6380 with tls 1.2 enabled

How to reproduce it (as minimally and precisely as possible):
Get store configured with azure cache redis over non sls port with tls 1.2 on.

Full logs to relevant components:

thanos-store-out1m-0 thanos-store-out1m {"caller":"redis_client.go:236","err":"read: connection reset by peer","items":0,"level":"warn","msg":"failed to mget items from redis","ts":"2022-10-20T22:56:51.626987079Z"}

Anything else we need to know:
Store configuration:

- store 
            - --log.level=info
            - --log.format=json
            - --data-dir=./thanos/store/data
            - --objstore.config=$(OBJSTORE_CONFIG)
            - --chunk-pool-size=20GB
            - --grpc-address=0.0.0.0:10906
            - --http-address=0.0.0.0:10907
            - --block-meta-fetch-concurrency=32
            - --block-sync-concurrency=32
            - --sync-block-duration=1m
            - --ignore-deletion-marks-delay=12h
            - --index-cache.config-file=/thanos/store/redis/redis_cache_conf.config
            - --store.caching-bucket.config-file=/thanos/store/redis/redis_cache_conf.config
            - --max-time=-4w

redis config: 
type: REDIS
        config:
          addr: "xx.redis.cache.windows.net:6380"
          password: "xx"
          db: 1
          dial_timeout: 10s
          read_timeout: 10s
          write_timeout: 30s
          pool_size: 100
          min_idle_conns: 10
          idle_timeout: 5m0s
          max_conn_age: 0s
          max_get_multi_concurrency: 100
          get_multi_batch_size: 100
          max_set_multi_concurrency: 100
          set_multi_batch_size: 100
          tls_enabled: true
          tls_config:
            insecure_skip_verify: true

[dmilind]

Also is there any way to provide --sni in redis config ?

[dmilind]

Input:
I tried to get tcpdump from thanos store, I observed that message is not encrypted, it is a plain text sending to remote azure redis cache instance, remote instance is expecting to receive it in encrypted so redis is closing down connection. If you observe thanos store is sending plain text redis key to auth Please take a look at ss

[hanjm]

TLS support is add in this PR #5674, 0.28.1 does not handle TLS, Please use 0.29.0 https://github.com/thanos-io/thanos/releases/tag/v0.29.0-rc.0

[dmilind]

Awesome, tested out tls connection with v0.29.0-rc.0. Works pretty well. Thanks to contributor.