[QUESTION] Getting "Unsupported value "snappy" for configuration property "compression.codec" while creating a consumer

[grunk]

Hi @mateusjunges , i have the same error as #105 but when creating a consumer :

$builder = Kafka::createConsumer($this->topics, $this->group, $this->host.':'.$this->port);
$builder->withHandler(function(KafkaConsumerMessage $message) {
	// Handle your message here
	echo $message->getBody();
	echo $message->getTopicName();
})
->stopAfterLastMessage()
->withMaxMessages(100);

$consumer = $builder->build();

$consumer->consume();

I'm using the following versions (on windows 11) :

laravel-kafka : 1.8.1
rdKafka : 6.0.1
librdkafka version (runtime) : 1.6.2
librdkafka version (build) :| 1.6.2.255

I tried to change the KAFKA_COMPRESSION_TYPE option to "none" or "gzip" without any success.

Note that by following the high level consumer exemple from rdkafka i am receiving message just fine.

Any ideas what am i missing ?

Thanks

[mateusjunges]

Hey @grunk please check this confluentinc/librdkafka#2496 issue and let me know if it helps

[mateusjunges]

Don't know why but I'm not able to reproduce this nor #105

[grunk]

Thanks for your answer.
I'm not sure that the build of librdkafka is at fault here as i manage to consume my message with php-rdkafka.

To be honest, i'm in a kind of special dev environment.
Apache/PHP on windows and Kafka running on an other windows machine with docker. Not really representative of production , maybe this is why i have this kind of error.

For now i'm using php-rdkafka directly so you can close this issue if you think it's something too specific.

[mateusjunges]

As soon as I have time I'll try to setup a environment as close to what you mentioned here and then I can get back to you.

[murilofurquim]

In the function below is hard-coded"snappy" instead of reading it from the config KAFKA_COMPRESSION_TYPE. This affects Windows environment as there is no snappy installed.

public function getProducerOptions(): array
    {
        $config = [
            'compression.codec' => 'snappy',
            'bootstrap.servers' => $this->broker,
            'metadata.broker.list' => $this->broker,
        ];

        return collect(array_merge($config, $this->customOptions, $this->getSaslOptions()))
            ->reject(fn ($option) => in_array($option, self::CONSUMER_ONLY_CONFIG_OPTIONS))
            ->toArray();
    }

[6gds]

In the function below is hard-coded"snappy" instead of reading it from the config KAFKA_COMPRESSION_TYPE. This affects Windows environment as there is no snappy installed.

public function getProducerOptions(): array
    {
        $config = [
            'compression.codec' => 'snappy',
            'bootstrap.servers' => $this->broker,
            'metadata.broker.list' => $this->broker,
        ];

        return collect(array_merge($config, $this->customOptions, $this->getSaslOptions()))
            ->reject(fn ($option) => in_array($option, self::CONSUMER_ONLY_CONFIG_OPTIONS))
            ->toArray();
    }

Can you tell me file and path where i must insert this code?

[murilofurquim]

In the function below is hard-coded"snappy" instead of reading it from the config KAFKA_COMPRESSION_TYPE. This affects Windows environment as there is no snappy installed.

public function getProducerOptions(): array
    {
        $config = [
            'compression.codec' => 'snappy',
            'bootstrap.servers' => $this->broker,
            'metadata.broker.list' => $this->broker,
        ];

        return collect(array_merge($config, $this->customOptions, $this->getSaslOptions()))
            ->reject(fn ($option) => in_array($option, self::CONSUMER_ONLY_CONFIG_OPTIONS))
            ->toArray();
    }

Can you tell me file and path where i must insert this code?

The path is vendor/mateusjunges/laravel-kafka/src/Config/Config.php. But you dont need to insert this code, what I meant is that this code is wrong because of the line
'compression.codec' => 'snappy',
This line should be something like 'compression.codec' => env(KAFKA_COMPRESSION_TYPE, 'snappy'),.
So on Windows we changed to 'compression.codec' => 'none', temporarily so that our developers can use the library.

[mateusjunges]

Hey folks, I've just merged a PR to fix this. Now, instead of the hardcoded snappy compression type, this library uses the one defined in the compression key of the kafka.php configuration file. Please let me know if it really solves this issue. Released on v1.8.4