Connecting to Dragonfly via memcache client doesn't work

[rocktavious]

Describe the bug
We are trying to use the memcached compliant API to connect a service via a Ruby client and we are getting errors. This same client works with an AWS Elasticache in memcache mode.

The dragonfly configuration

kind: Dragonfly
metadata:
  labels:
    app.kubernetes.io/name: dragonfly-memcache
    app.kubernetes.io/instance: dragonfly-opslevel
    app.kubernetes.io/managed-by: kustomize
  name: memcache
  namespace: staging
spec:
  priorityClassName: critical-priority
  replicas: 1
  memcachedPort: 11211
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 600m
      memory: 3Gi

The Ruby app in the same kubernetes cluster showing that we can resolve the DNS

irb(main):001> require 'resolv'
=> false
irb(main):002> Resolv.getname "10.100.251.152"
=> "memcache.staging.svc.cluster.local"
irb(main):009> Resolv.getaddress "memcache.staging.svc.cluster.local"
=> "10.100.251.152"

Also from the shell inside the Ruby pod a sample showing we can connect

# telnet memcache.staging.svc.cluster.local 11211
Trying 10.100.251.152...
Connected to memcache.staging.svc.cluster.local.
Escape character is '^]'.
stats
STAT pid 1
STAT uptime 3465
STAT time 1716310286
STAT version v1.16.0
STAT libevent iouring
STAT pointer_size 8
STAT rusage_user 30.763
STAT rusage_system 19.0581
STAT max_connections -1
STAT curr_connections 1
STAT total_connections -1
STAT rejected_connections -1
STAT bytes_read 5284
STAT bytes_written 5046
STAT limit_maxbytes -1
END

But when we go to use the client library we get this

irb(main):013> require "dalli"
=> false
irb(main):014> dc = Dalli::Client.new("memcache.staging.svc.cluster.local:11211")
=> #<Dalli::Client:0x00007f1cb51ef108 @key_manager=#<Dalli::KeyManager:0x00007f1cc0335b88 @key_options={:digest_class=>Digest::MD5}, @namespace=nil>, @normalized_servers=["memcache.staging.svc.cluster.local:11211"], @options={}, @ring=nil>
irb(main):015> dc.set('abc', 123)
W, [2024-05-21T18:44:13.310039 #96]  WARN -- : memcache.staging.svc.cluster.local:11211 failed (count: 0) Timeout::Error: IO timeout: {:host=>"memcache.staging.svc.cluster.local", :port=>11211, :down_retry_delay=>30, :socket_timeout=>1, :socket_max_failures=>2, :socket_failure_delay=>0.1, :keepalive=>true}
(irb):15:in `<main>': No server available (Dalli::RingError)

I have filed a ticket with the client library but they point back to dragonfly not being fully compliant as the culprit since it works fine with actual memcached. petergoldstein/dalli#1003

To Reproduce
Steps to reproduce the behavior:

1. Standup a dragonfly cluster with memcache enabled
2. Standup a pod with Ruby and install the dalli client
3. Try connecting using the dalli client to see the same error as above

Expected behavior
Ability to make connections using memcached clients

Screenshots
See the above terminal sessions

Environment (please complete the following information):

• OS: Amazon Linux Kubernetes Node - Using the k8s operator
• Kernel: 5.10.173-154.642.amzn2.x86_64
• Containerized?: Kubernetes
• Dragonfly Version: 1.16.1

[romange]

@rocktavious could it be this client library used the binary protocol?

[eapache-opslevel]

Yes, dalli uses the binary protocol by default. But https://www.dragonflydb.io/faq/how-to-connect-to-memcached has explicit example code that uses the binary protocol, and I don't see any other documents indicating it only supports the text protocol?

[romange]

I am sorry, it is indeed confusing. @Niennienzz we should probably clarify this question does not refer to Dragonfly.

@eapache-opslevel it's about memcached and not about Dragonfly, the whole FAQ is about other in-memory systems, Dragonfly docs are located here: https://dragonflydb.io/docs

[Niennienzz]

I will update both the FAQ and the Docs pages shortly.

[eapache-opslevel]

OK, thanks for the clarification, so Dragonfly only supports the text-based memcached protocol? Do you know if that is the (very old) "ASCII" protocol, or the (much newer) "meta" protocol? Memcached's own documentation on this point is incredibly out of date, but https://github.com/petergoldstein/dalli/wiki/Requirements-and-Integrations makes it sound like there are two text-based protocols now.

[eapache-opslevel]

specifically:

In Dalli 3.2 support was added for memcached's meta protocol. This protocol, an evolution of the older ASCII protocol, is the planned go-forward protocol for memcached. It supported all required functions as of memcached 1.6.13, and Dalli is compatible with memcached's meta protocol for memcached 1.6.13 and later 1.6.x patch versions.

[romange]

I think we support what it is called an "ascii" protocol, i.e. set, get etc. It should not be too hard to implement the meta protocol, if you open an issue maybe someone from the community will add the support for it.