Suspected cache pollution when adding multiple hosts

[mattvb91]

When adding multiple hostnames to an upstream (?) after a certain amount of time (or certain number of hostnames) there is some kind of cache pollution happening.

Once the cache has been flushed the files return to giving good responses for a certain amount of time before the same happens again.

This has been reproduced in this sample repository here with minimal config: https://github.com/mattvb91/caddy-cache-bug

Samples taken from discussions over from #26 (comment)

After cache flush:

Here is a video of it happening in action:

Screencast.2022-05-18.09.10.13.mp4

[darkweak]

After investigation this is a badger issue


because I request the right key but badger returns the bad key. I don't know why it doesn't clean properly the key when it reach the cache size limit.

[darkweak]

https://discuss.dgraph.io/t/badger-doesnt-return-the-good-value/17350

[mattvb91]

Aaah thanks for the heads up. I will try at some stage with the redis driver to see if that gets me further just to be able to continue development with the cache system

[darkweak]

I will probably switch from badger to rosedb/lotusdb.
I recommend to you to use the embedded olric provider instead using the olric directive in the caddyfile/json config until I implement the new default provider.

[mattvb91]

So I can confirm that on the first initial look my CI pipeline runs through using olric cache driver so from that side it seems to have improved something and locally it seems to look fine too on first glance.

The requests do come back with correct cache headers etc.. however now on the cache endpoint in my case /__cache/souin that lists active cache entries it is now always empty? (Even tho I can see cache correctly working)

[darkweak]

I didn't tag the new Souin version, but instead of using github.com/caddyserver/cache-handler as with option you can use github.com/darkweak/souin/plugins/caddy@latest (or the master commit hash). This will include the patch for the surrogate-keys.

[mattvb91]

Just had a quick look with the @latest build and there is still some issue happening although not sure if related or new again.

The timings seem incredibly slow compared to badger:

Olric:

Badger:

That nearly looks like olric cache isnt even working? Might explain why my /__cache/souin list is empty?

When I switch back to badger the /__cache/souin list works fine. So I think theres an issue with olric caching too.

My main config at the moment:

                "cache" => [
                    'api'         => [
                        'basepath' => '/__cache',
                        'souin'    => [
                            'basepath' => '/souin',
                            'enable'   => true,
                        ],
                    ],
                    'distributed' => true,
                    'olric'       => [
                        'url' => 'olric:3320',
                    ],
                    "log_level"   => "info",
                    'ttl'         => '3600s',
                ],

[darkweak]

Do you have a running olric instance in the same network?
You don't have to specify the url or a configuration if you want to run olric as embedded mode (in memory). I think the documentation is not really explicit about that.

[mattvb91]

ah my bad yes I was instantiating a new olric container in my docker-compose.yml I didnt realize it was already included in the caddy image! Thats looking way better now.

For anyone else that stumbles on here from google / issue search my current config now looks like this:

                "cache" => [
                    'api'         => [
                        'basepath' => '/__cache',
                        'souin'    => [
                            'basepath' => '/souin',
                            'enable'   => true,
                        ],
                    ],
                    'distributed' => true,
//                    'olric'       => [
//                        'url' => 'olric:3320',
//                    ],
                    "log_level"   => "info",
                    'ttl'         => '3600s',
                ],

and I can see the /__cache/souin list working. Also on initial glance I can do PURGE requests against for example http://localhost/__cache/souin/GET-test-1-yargvp50.localhost-/api/shop/get/first-product and I can see it getting removed from the list so its definitely working smoother!

I will play around with the surrogate-keys later again from the other issue to report back on it!

[eth-limo]

I will probably switch from badger to rosedb/lotusdb. I recommend to you to use the embedded olric provider instead using the olric directive in the caddyfile/json config until I implement the new default provider.

@darkweak thank you for this project. How can I use the embedded olric provider?

I'm building the Caddy binary with this command:

xcaddy build ${CADDY_VERSION} \
    --with github.com/caddyserver/cache-handler

Here is a snippet of my Caddyfile.json:

            "cache": {
                "headers": [
                    "Content-Type"
                ],
                "log_level": "info",
                "olric": {
                    "path": "/etc/olric/olric.yaml"
                  },
                  "ttl": "300s"
            },

/etc/olric/olric.yaml:

    olricd:
      bindAddr: 0.0.0.0
      bindPort: 3320
      serializer: "msgpack"
      keepAlivePeriod: "300s"
      requestTimeout: "5s"
      partitionCount:  271
      replicaCount: 1
      writeQuorum: 1
      readQuorum: 1
      readRepair: false
      replicationMode: 0 
      tableSize: 1048576
      memberCountQuorum: 1

    logging:
      verbosity: 6
      level: DEBUG
      output: stderr

    memberlist:
      environment: "local"
      bindPort: 3322
      enableCompression: false
      joinRetryInterval: "1s"
      maxJoinAttempts: 1

However I still see the following in my Caddy logs:

badger 2022/06/02 20:14:47 INFO: Set nextTxnTs to 0

This project is killer and is exactly what I need, however this cache pollution issue is causing us a heacache. Thanks!

[darkweak]

If you attempt to load the embedded olric provider you just have to set "olric": {} in the caddyfile configuration.

[eth-limo]

If you attempt to load the embedded olric provider you just have to set "olric": {} in the caddyfile configuration.

Thanks, I'll try this. Is it possible to use https://github.com/buraksezer/olric-cloud-plugin with the Caddyfile.json configuration I pasted above? I'd like to ideally cluster these caddy instances in k8s.