memcache with ext_path maxing out at 400gig

[mzupan]

I'm running memcached via the following

-m 48096 -I 2m -c 2024 -t 12 -o ext_path=/ssd/cache:1500G,ext_threads=10

Over the last 4 days it seems to max out to 400gig in memory used with ram/ssd cache. Evicts start to happen when it reaches 400gig.

[mzupan]

so my setup had a pod with 52gig of memory and I gave memcache 48gig. It was able to allocate 400gig to the file on disk. I was thinking if I gave memcache less memory it would use more of the free system memory for file locations. I dropped memcache memory to 38096 and was only able to use around 100g of local storage.

So still digging into how I can allocate more ssd cache without using a giant ec2 instance to get more memory for the pod

[dormando]

though immediately after I write that: I'm not sure offhand if loki's parallelism argument limits your consumer side CPU usage. If that's the case "Batch size 4 parallelism 8" might be something worth trying.

[mzupan]

I can hop on discord.. so after about 15min of no searching its up

STAT slab_global_page_pool 15104

[dormando]

https://discord.gg/4c4GDKk discord link (guess I can get a named invite at some point?)

good to see it does recover, since that rules out a lot of things. It should be fixed by cutting the batch/parallelism down, I hope.

[dormando]

Note: upping ext_wbuf_size=8 to 64 might help as well. I didn't realize just how aggressively data was being loaded here.

(sorry, meant to put that in a thread reply)

[dannykopping]

FWIW this is how we're running extstore in production:

      - args:
        - -m 3072
        - -I 2m
        - -c 64000
        - -v
        - -u
        - root
        - --extended
        - ext_page_size=256,ext_wbuf_size=32,ext_threads=16
        - --extended
        - slab_automove_freeratio=0.05,ext_max_sleep=100000
        - --extended
        - ext_path=/mnt/disks/ssd0/datafile:345G,ext_path=/mnt/disks/ssd1/datafile:345G,ext_path=/mnt/disks/ssd2/datafile:345G,ext_path=/mnt/disks/ssd3/datafile:345G

Each instance requests 6 CPUs & 5GiB of RAM.
We run 4 separate disks to spread out the load on each instance.

for right now I just have 1 memcache instance

I would suggest scaling out if possible. Using 1 memcached pod is going to put a whole lot of pressure on it; Loki is very heavy on cache and as @dormando mentions we're not pipelining correctly yet; it's on my infinity backlog™️, though.