Options for projection in command

[darrylhein]

What's the recommended way to pass options to a projection? I'm looking to change the projector options.

I'm currently using the event-store:projection:run command but as far as I can tell, there's no way to modify the options without creating a new command with a lot of the code that's in AbstractProjectionCommand.

A few ideas:

1. Add an options property to AbstractProjectionCommand and then add my own command that uses the abstract overriding the options property. But this seems like near-copying code for no good reason (though less than copying the entire AbstractProjectionCommand).

2. Add options/arguments to the command to support all/most values that projectors support. The only issue I see here is that they'll get long and ugly unless we come up with shorten versions.

3. Add config options. The problem I see here is that you may want different options depending on the situation and we'd need to load them "everywhere" (or maybe just in the command). Though could be worth adding this along with 1 or 2.

I'm okay to attempt a first version of the change, but don't want assume which direction is preferred.

[danizord]

@darrylhein how about adding support for projector options in the projections yaml config?

[darrylhein]

@danizord I'm guessing you'd need separate nodes in the yaml config for each projection. I like the idea of passing them to the command as it gives you the ability to change on the fly and not have to update the config. But, it would be nice in yaml or other config that it'd remember for you. But maybe it'd be better to have a separate command for each projection if there's a lot of customization needed.

Here's a hacked together version to support the sleep param: https://github.com/xmmedia/starter_symfony_4/blob/master/src/Command/ProjectionRunCommand.php

[danizord]

@darrylhein

@danizord I'm guessing you'd need separate nodes in the yaml config for each projection.

We already do, under projections we have a node for each projection.

I like the idea of passing them to the command as it gives you the ability to change on the fly and not have to update the config.

Can't we have both? :)

[unixslayer]

@prolic this issue can probably be closed

[webdevilopers]

The OPTION_GAP_DETECTION has been added. Not sure if it is missing in this bundle currently.

• Add gap detection to projections pdo-event-store#221
• projection options #69

It would be nice to have a configuration for single projections or enable die gap detection globally.

At least the option can be added manually directly to a projection manager:

$gapDetection = new GapDetection();

$projection = $projectionManager->createProjection('test_projection', [
            PdoEventStoreProjector::OPTION_GAP_DETECTION => $gapDetection,
        ]);

[unixslayer]

@webdevilopers gap detection can be defined using projection options as tagged service. However this way defined options must have a tag for each projection separately:

services:
    Prooph\ProophessorDo\Projection\Options\ToDoProjectionOptions:
        tags:
            - { name: prooph_event_store.projection_options, projection_name: projection_1 }
            - { name: prooph_event_store.projection_options, projection_name: projection_2 }

What you need is a nice addition 👍🏻 If in a hurry, you can introduce a PR or wait some time for me to figure it out.

[webdevilopers]

Thanks @unixslayer , we will try your suggested solution.

In addition @Mattin also suggested this projection command extension:

• MySQL Projections skipping events (SingleStreamStrategy) pdo-event-store#189 (comment)

Indeed it would be nice to configure cap datection via parameter too.

<?php

declare(strict_types=1);

namespace Prooph\ProophessorDo\Projection\Options;

use Prooph\Bundle\EventStore\Projection\ProjectionOptions;
use Prooph\EventStore\Pdo\Projection\GapDetection;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBag;

final class ToDoProjectionOptions implements ProjectionOptions
{
    private ParameterBag $parameterBag;

    public function __construct(ParameterBag $parameterBag)
    {
        $this->parameterBag = $parameterBag;
    }
    
    public function options(): array
    {
        return [
            'cache_size' => $this->parameterBag->get('projection.cache_size'),
            'sleep' => $this->parameterBag->get('projection.sleep'),
            'persist_block_size' => $this->parameterBag->get('projection.persist_block_size'),
            'lock_timeout_ms' => $this->parameterBag->get('projection.lock_timeout_ms'),
            'trigger_pcntl_dispatch' => $this->parameterBag->get('projection.trigger_pcntl_dispatch'),
            'update_lock_threshold' => $this->parameterBag->get('projection.update_lock_threshold'),
            'gap_detection' => new GapDetection([0, 5, 5, 10, 15, 25, 40, 65, 105]),
        ];
    }
}

[webdevilopers]

@unixslayer I just remembered that we talked about the "event sourcing" and "projection" dependencies in the current versions:#

on dev-master dependency for prooph/event-sourcing was dropped.

• Plugins and Metadata Enrichers are not recognized #73 (comment)

Currently we still use prooph/event-store-symfony-bundle ^0.8.0 because changing the config would require some work:
*#73 (comment)

Adding projection options in ^0.8.0 will cause:

Unrecognized option "options" under "prooph_event_store.projection_managers.default_projection_manager.projections.account_details".
Available options are "projection", "read_model".

Similar issue w/ the tagged example since the extended class Prooph\Bundle\EventStore\Projection\ProjectionOptions does not exist.

So I guess there is no other option than updating this bundle?

[unixslayer]

@webdevilopers I think you still should be able to use latest version of this bundle and require prooph/event-sourcing separately

[webdevilopers]

Hm, without having to change the config?

• Plugins and Metadata Enrichers are not recognized #73 (comment)

And keep the 0.8.0 for event-sourcing?

[webdevilopers]

Ok, got it!

"prooph/event-store-symfony-bundle": "^0.9.1",

Since we're on Symfony 5.1.

We "only" have to replace the repositories e.g. via:

class SomeAggregateRepository extends AggregateRepository
{
    public function __construct(EventStore $eventStore)
    {
        parent::__construct(
            $eventStore,
            AggregateType::fromAggregateRootClass(SomeAggregate::class),
            new AggregateTranslator(),
            null,
            new StreamName('some_aggregate_stream')
        );
    }
}

• https://github.com/prooph/event-store-symfony-bundle/blob/master/doc/migrating_from_0.8.0.md

Will give it a try now!

[webdevilopers]

Looks promising so far @unixslayer .

Where can I verify that projection options are adapted?

prooph_event_store:
  stores:
    default:
      event_store: Prooph\EventStore\EventStore

  projection_managers:
    default_projection_manager:
      event_store: 'prooph_event_store.default'
      connection: '@app.event_store.pdo_connection.pgsql'
      projections:
        visit_details:
          read_model: Acme\Visit\Infrastructure\Projection\MongoVisitDetailsReadModel
          projection: Acme\Visit\Infrastructure\Projection\MongoVisitDetailsProjection
          options:
            - lock_timeout_ms: 666
            - sleep: 100000000000

Next I tried a simple dump inside a projector:

final class MongoVisitDetailsProjection implements ReadModelProjection
{
    public function project(ReadModelProjector $projector): ReadModelProjector
    {dd($projector);

But the result does not use my custom values:

...
  -initCallback: null
  -handler: null
  -handlers: []
  -isStopped: false
  -currentStreamName: null
  -lockTimeoutMs: 1000
  -eventCounter: 0
  -sleep: 100000
  -triggerPcntlSignalDispatch: false
  -updateLockThreshold: 0
  -gapDetection: null
  -query: null
  -vendor: "pgsql"
  -lastLockUpdate: null
  -metadataMatcher: null
}

This happens with 0.9.1. Any ideas?

[webdevilopers]

BTW the tagged service works like a charm!

<?php declare(strict_types=1);

namespace Acme\Common\Infrastructure\Prooph\Projection;

use Prooph\Bundle\EventStore\Projection\ProjectionOptions;
use Prooph\EventStore\Pdo\Projection\GapDetection;
use Symfony\Component\DependencyInjection\ParameterBag\ParameterBagInterface;

final class GlobalProjectionOptions implements ProjectionOptions
{
    private ParameterBagInterface $parameterBag;

    public function __construct(ParameterBagInterface $parameterBag)
    {
        $this->parameterBag = $parameterBag;
    }

    public function options(): array
    {
        return [
            /*
            'cache_size' => $this->parameterBag->get('projection.cache_size'),
            'sleep' => $this->parameterBag->get('projection.sleep'),
            'persist_block_size' => $this->parameterBag->get('projection.persist_block_size'),
            'lock_timeout_ms' => $this->parameterBag->get('projection.lock_timeout_ms'),
            'trigger_pcntl_dispatch' => $this->parameterBag->get('projection.trigger_pcntl_dispatch'),
            'update_lock_threshold' => $this->parameterBag->get('projection.update_lock_threshold'),
            */
            'update_lock_threshold' => 666,
            'gap_detection' => new GapDetection([0, 5, 5, 10, 15, 25, 40, 65, 105]),
        ];
    }
}

...
  -updateLockThreshold: 666
  -gapDetection: Prooph\EventStore\Pdo\Projection\GapDetection {#119
    -retryConfig: array:9 [
      0 => 0
      1 => 5
      2 => 5
      3 => 10
      4 => 15
      5 => 25
      6 => 40
      7 => 65
      8 => 105
    ]
    -detectionWindow: null
    -retries: 0
  }

[unixslayer]

@webdevilopers sorry for late response. I just checked it and looks like options should be adapted properly when defined in central projection configuration as you can see bellow:

[unixslayer]

@webdevilopers I've just released v0.9.3 which allows to define projection options in central configuration (eg. yaml). Please refer to docs example. Take note that both retry_config and detection_window are optional for GapDetection.

[webdevilopers]

Thanks for the great work! Will try to check it out in the next days!