Too minimalistic approach for config documentation

[damianprzygodzki]

I am totally confused with documentation that is constructed really poor.

I understand that we have topics to connect messages to processors, but there is no information about connecting this to transport queue. There is also no info about ability of configuration more clients (Clients or topics?). Naming in config is also confusing, like 'routerQueue' vs. 'defaultQueue'.

Wasted lot of time to research it on my own...

[Steveb-p]

You're welcome to provide improvements to documentation.

As with many open-source projects, oftentimes documentation is satisfactory to the one writing the actual library, but not for end users - which is obviously due to the fact that author does have an understanding of what code does and "assumes" knowledge of it's internals.

At the very least please provide links to documentation pages that were confusing for you.

[damianprzygodzki]

You're welcome to provide improvements to documentation.

It is totally hopeless idea to let person who struggle with bundle create documentation.

As with many open-source projects, oftentimes documentation is satisfactory to the one writing the actual library, but not for end users

That's why i created issue - to announce that there is a need.

At the very least please provide links to documentation pages that were confusing for you.

It is also funny because each of documentation available (official github repo or readthedocs service) has different and not up to date docs, so if you need URL, each one available is confusing.

Issue is about config documentation, config is documented in one chapter, how can i be more precise?

https://github.com/php-enqueue/enqueue-dev/blob/master/docs/bundle/config_reference.md

[Steveb-p]

It is totally hopeless idea to let person who struggle with bundle create documentation.

Quite the contrary. You were looking for a particular option or configuration value and was unable to deduce what it stands for or where it can be found.
If you have it figured out - you said you spent time on it - then you're best suited to provide contribution since your knowledge is "fresh". If you did not, then ask a specific question.

As I said - even though I'm only "responsible" for Kafka part of library - I can imagine it might be difficult for maintainer to cater to everyone. Usually https://github.com/php-enqueue/enqueue-dev/blob/master/docs/bundle/quick_tour.md file is sufficient for most basic configurations, configuration reference that you have linked is just that - a config reference. Certainly it can be improved, but keep in mind we're spending our private time doing so - so pointing us in the right direction as specifically as possible is more than welcome.

If you want to discuss it further then drop me a message on gitter, so we don't pollute this issue with just our conversation. Just keep in mind I might not be online and I might not be able to answer your questions (since, again, I'm officially only looking after Kafka part of enqueue).

[damianprzygodzki]

If you did not, then ask a specific question.

Thanks, i do.

client:
    traceable_producer:   true
    # What is prefix? Where it applies? Why it is defined as another parameter? Why it has to be always string and there is no ability to disable it? 
    prefix:               enqueue
    separator:            .
    # What is this parameter for?
    app_name:             app
    # What is router topic?
    router_topic:         default
    # Is it queue name for my topic?
    router_queue:         default
    router_processor:     null
    redelivered_delay_time: 0
    # Is it fallback for router_queue? Why do we have fallback? What is the case we need to define it here for?
    default_queue:        default

[sustmi]

I also currently struggle with the docs. If you don't mind, I will write my specific problems here. Or I can create separate issue from them if you want to keep this discussion focused on different topic.

1. Transport vs Client

Only during writing this comment I realized what is the difference between Client and Transport.
Surely, both terms are described in the quick tour: https://github.com/php-enqueue/enqueue-dev/blob/0.9.9/docs/quick_tour.md#transport
but I think those two look very similar and it would be nice to put them in a direct contrast in order to make these key concepts clear.

I suggest to start the Quick Tour guide with section called Transport vs Client. This section would first pinpoint that Transport is low-level while the Client is a higher-level abstraction for both production and consumption of messages.
Then it could have subsections that will be basically the current Transport and Client sections.

2. The documentation of message production using the Symfony bundle is not clear about the situation when there are multiple enqueue configurations

The Send event example shows that one should just get the ProducerInterface from dependency-injection container but it is not clear which one I will get and how can I get a specific one.
I realized that each configuration under the enqueue key is responsible for creating several services in the container. To name few of them:

• enqueue.client.<configuration name>.context being instance of Interop\Queue\Context
• enqueue.client.<configuration name>.producer implementing the ProducerInterface (being instance of Enqueue\Client\TraceableProducer)
• enqueue.transport.<configuration name>.context again being instance of Interop\Queue\Context

There are also enqueue.client.default.producer and enqueue.transport.default.context which are aliases for the default services. (Based on the first in the YAML configuration?)

I suggest to use https://github.com/php-enqueue/enqueue-dev/blob/0.9.9/docs/bundle/config_reference.md as a place that explains what types of services are created by the individual enqueue configurations.
The https://github.com/php-enqueue/enqueue-dev/blob/0.9.9/docs/bundle/message_producer.md guide could then link to the config reference and mention that one can pick any of the producer sevices based on which configuration one wants to use.

I think that Doctrine Bundle's documentation does a decent job on explaining this: https://symfony.com/doc/master/bundles/DoctrineBundle/configuration.html#doctrine-dbal-configuration

The database_connection service always refers to the default connection, which is the first one defined or the one configured via the default_connection parameter.

Each connection is also accessible via the doctrine.dbal.[name]_connection service where [name] is the name of the connection.

3. Also, how are the individual configurations called? Are those "transports"?

The configuration reference calls the configurations by very general terms "key" and "prototype": https://github.com/php-enqueue/enqueue-dev/blame/0.9.9/docs/bundle/config_reference.md#L18
Should I call these "configuations of transports" or "clients" or "connections"?
Because when using simply "enqueue configurations", how do we know if we are talking about enqueue bundle configuration (as whole) or individual (connection|transport|client) configurations?

I suggest to pick a one distinct term for the single configuration and use it both in EnqueueBundle. Quick tour and Config reference sections.

4. It is unclear how to specify additional options for Kafka

The Kafka transport section says:

You can also you extended configuration to pass additional options, if you don't want to pass them via DSN string or need to pass specific options. Since rdkafka uses librdkafka (being basically a wrapper around it) most configuration options are identical to those found at https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.

OK, lets say that I want to set the request.timeout.ms option that libkafka accepts for topic configuration: https://github.com/edenhill/librdkafka/blob/v1.0.0/CONFIGURATION.md#topic-configuration-properties

Where should I put this configuration? The only place in the bundle's Config reference which looks like it accepts some additional options is driver_options in the client section.
But is it the right place? And why is it in client section and not transport section?
And what is driver anyway. Nowhere in the documentation is any explanation of term "driver", what does it mean in the context of this library and how it is related to transport and client.

Judging from the source code the only place that uses these driver_options is \Enqueue\Client\DriverFactory::createRabbitMqStompDriver() method.

I suggest to explain how exactly can one pass the librdkafka configuration options (and what type of options can be used: global or topic?) in the Kafka transport section. Also, it will be helpful to explain what "driver" means at least in the comment in Config reference that describes driver_options.

EDIT: I realized that the librdkafka configuration options go to global and topic section in the transport section.

What are your opinions on my suggestion? Do they make sense to you? Do you want to fix those issues by yourself @Steveb-p or should I create a pull request?

[Steveb-p]

@sustmi thanks, I'll take a detailed look in the evening :)

[sustmi]

This may seem out-of-topic but since @damianprzygodzki already mentioned the poorly documented client options and I think this is related.

5. Why no messages arrived to my Kafka topic and what is router?

I am trying to send a message to a specific Kafka topic. I get no errors or warnings but no messages arrived to the topic yet.

My config.yml contains this:

enqueue:
    my_enqueue_subconfig:
        transport:
            dsn: "rdkafka://"
            global:
                metadata.broker.list: "%kafka_brokers%"
        client: ~

and I am sending the message like this:

$message = new \Enqueue\Client\Message('messageBody');

$this->producer->sendEvent(
    'my_kafka_topic_name',
    $message
);

where $this->producer is injected with service @enqueue.client.my_enqueue_subconfig.producer.

I tried to do some debugging and I found that this code gets executed: https://github.com/php-enqueue/enqueue-dev/blob/0.9.9/pkg/enqueue/Client/Driver/GenericDriver.php#L62
and $topic is instance of \Enqueue\RdKafka\RdKafkaTopic with $name set to enqueue.app.default instead of my_kafka_topic_name.

It looks like the topic name is set to [client.prefix].[client.app_name].[client.router_queue]. I don't get it. What is router? Is it some kind of abstraction? Why does RdKafkaTopic contain this topic name instead of the one I specified when sending the message?

[Steveb-p]

Why does RdKafkaTopic contain this topic name instead of the one I specified when sending the message?

To clarify it on the spot: topic as enqueue understands it is NOT Kafka topic. Topic set in sendEvent method is only present in message properties. For all other purposes there is only one RdKafkaTopic object in use.

[sustmi]

Maybe I got the whole Enqueue library wrong.
I thought about Enqueue as a Doctrine database library but for message queues.

But now I have a feeling that Enqueue is actually one level of abstraction higher. Instead of comparing it to a Database access library it could be compared to Entity-Attribute-Value model that stores all data in a single database table. Am I right?
I mean that it does not use the native topic functionality that Kafka provides but it implements its own topic system on top of any messaging system.
Instead of sending message to some arbitrary Kafka topic it sends all messages to a single enqueue.app.default Kafka topic and the actual ("virtual") topic name is part of the message itself.

Is this true or am I still misunderstanding this library?

[Steveb-p]

@sustmi I can't say for sure, since I have only a rough idea how other transports work.

Technically speaking, there is not much preventing enqueue from using multiple Kafka topics for it's communication. It's only that at the moment when it was created it simply did not (I'd think about more like a proof of concept that Kafka can work as a transport).
EDIT: As a matter of fact, I'm pretty sure I've seen Symfony Messenger Adapter allowing control of which Kafka topic to write to, and it's loosely based on an adapter that uses enqueue.

There is definitely a lot of improvement that can be made. For example, current implementation of enqueue is unable of listening to multiple Kafka topics, even though the underlying library is capable of such consumption (see #832). Producers still need some love regarding poll method from KafkaProducer (see #749).
Functionaly, multiple Kafka topics will probably open the library (at least this particular transport) to truly use Commands from CQRS concept (possibly allowing replying to a commands, not only sending messages / events).

Working with multiple Kafka topics is constrained by unability of current Phprdkafka library to handle Kafka's Admin API (which was introduced in librdkafka in latest 1.0.0 release, which was sometime in february this year?). Otherwise enqueue will rely on topic defaults for those.
When ability to control Kafka topics arrive in phprdkafka it will allow controlling of topic retention and partitioning.

I'm pretty sure what you described (enqueue using already existing topic implementation in Kafka) is the way Makasim does intend to go with it eventually.

EDIT: I'm available most of the time on enqueue's gitter if you want to discuss it further and not pollute this issue thread.

[makasim]

The Send event example shows that one should just get the ProducerInterface from dependency-injection container but it is not clear which one I will get and how can I get a specific one.

Auto wire works for default transport\client (explicitly named default or the first configured).

[makasim]

It looks like the topic name is set to [client.prefix].[client.app_name].[client.router_queue]. I don't get it. What is router? Is it some kind of abstraction? Why does RdKafkaTopic contain this topic name instead of the one I specified when sending the message?

You mistake Enqueue Client Topic with Kafka ones. These are different concepts. The message does not get anywhere because there is no subscribers (processors) listening for that topic. Hence the message goes nowhere. In contrast, if there is no processor for the command an exception is thrown.

[makasim]

What is router?

Not all transports support pub\sub out of the box (file, redis, sqs to name a few). So we added a router concept to overcome that limitation.

[makasim]

I mean that it does not use the native topic functionality that Kafka provides but it implements its own topic system on top of any messaging system.

It depends on what you are using. If you use Enqueue Client then it is a high level abstraction and
of course it hides away some kafka parts. If you use rdkafka transport provide low level access to kafka features almost as is.