Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Cache] add integration with Messenger to allow computing cached values in a worker #30572

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
4 participants
@nicolas-grekas
Copy link
Member

nicolas-grekas commented Mar 14, 2019

Q A
Branch? master
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets -
License MIT
Doc PR -

This PR needs and for now embeds #30334. See 2nd commit.

Using the new CacheInterface enables probabilistic early expiration by default. This means that from time to time, items are elected as early-expired while they are still fresh (see Wikipedia for details).

This PR adds a new messenger_bus option when configuring cache pools. When this option is set, cache pools are configured to send those early-expired items on a Messenger bus, then serve the current value immediately, while the updated value is computed in a worker in the background.

CacheInterface::get($key, $callback) accepts any callable, but sending any callable on a bus is not possible (e.g. a Closure cannot be serialized). To bypass this constraint, this feature works only with callables in the form [$service, 'publicMethod'], where $service is any public or reversible service.

This limitation is a serious one: this $service must compute a value when knowing only its key. This means keys should embed enough information for this to happen. I think that's not that hard - and we may find ways to provide additional context in the future.

At least the goal is worth it: in theory, this strategy allows achieving a 100% hit ratio even when invalidation-by-expiration is used.

@Nyholm
Copy link
Member

Nyholm left a comment

Cool. I really like this feature.

Could you show a code example how I would use it.

I’m also trying to figure out how I can work around the limitation..

Show resolved Hide resolved src/Symfony/Component/Cache/Messenger/Message.php Outdated

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:cache-bus branch from 07f2817 to bac41f9 Mar 15, 2019

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 15, 2019

@Nyholm thanks for having a look :)

To test this, you need to configure a pool with the messenger_bus option set, and have the consumer run:

framework:
    cache:
        pools:
            test.cache:
                messenger_bus: message_bus
    messenger:
        transports:
            amqp: '%env(MESSENGER_TRANSPORT_DSN)%'
        routing:
            'Symfony\Component\Cache\Messenger\Message': amqp

BTW, this shows another area where Messenger needs to improve: how should a bundle provide configuration for this Message-to-transport mapping out of the box? ping @sroze @ogizanagi @weaverryan.

Then e.g. in a controller:

    /**
     * @Route("/test", name="test")
     */
    public function index(CacheInterface $testCache)
    {
        $value = $testCache->get('foo', [$this, 'compute']);

        return $this->render('test/index.html.twig', [
            'value' => $value,
        ]);
    }

    public function compute(ItemInterface $item)
    {
        $item->expiresAfter(5);
        sleep(1);

        return sprintf('#%06X', mt_rand(0, 0xFFFFFF));
    }

Runnig the dev weberver: symfony serve
Running the worker: bin/console messenger:consume -vvv
etc.

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:cache-bus branch 3 times, most recently from 1967fcb to 4020350 Mar 15, 2019

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:cache-bus branch from 4020350 to 4f69bb1 Mar 15, 2019

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

nicolas-grekas commented Mar 15, 2019

See https://github.com/nicolas-grekas/cache-bus for a running example.

@sroze

This comment has been minimized.

Copy link
Member

sroze commented Mar 19, 2019

Looks like a great idea 😍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.