Skip to content
Experimental pure PHP Kafka Client library based on PHP 7.4, ffi and librdkafka
C PHP Dockerfile
Branch: master
Clone or download
dirx add FFIMe based librdkafka binding generation
* still header file parsing issues here
* just playing around ... for now
Latest commit 884f8a1 Oct 4, 2019

README.md

Pure PHP Kafka Client library based on php7.4-dev, ffi and librdkafka

EXTREMLY EXPERIMENTAL WIP

Build Status Test Coverage Maintainability

This is a pure PHP Kafka Client library as replacement for php rdkafka extension.

Playing around with:

Prepare

Directory overview

  • /benchmarks - phpbench based benchmark tests
  • /docs - docs dir (md prefered)
  • /examples - example scripts
  • /resources
    • /docker
      • /php74-librdkafka-ffi - dockerfile for PHP 7.4 image with librdkafka and ffi & rdkafka ext (based on php:7.4-rc-cli )
    • /test-extension - base dir for rdkafka ext compatibility tests
  • /src - source dir
  • /tests - tests dir

Build images

Build all images

docker-compose build --no-cache --pull

Alternative: build the image individually

docker-compose build --no-cache --pull php74

Test - should show latest 7.4 rc version

docker-compose run php74 php -v

Test - should show FFI in modules list

docker-compose run php74 php -m

Test ffi librdkafka binding - should show 1.0.0 version of librdkafka:

docker-compose run -v `pwd`:/app -w /app php74 php examples/version.php

Test - should show rdkafka in modules list

docker-compose run php74 php -dextension=rdkafka.so -m

Startup

Startup php & kafka

docker-compose up -d

Having fun with examples

Examples use topic playground.

Updating Dependencies

docker-compose run --rm --no-deps php74 composer update

Creating new topic playground with 3 partitions ...

docker-compose run --rm php74 php examples/create-topic.php -tplayground -p3 -rf1

Producing ...

docker-compose run --rm php74 php examples/producer.php

Consuming ...

docker-compose run --rm php74 php examples/consumer.php

Broker metadata ...

docker-compose run --rm php74 php examples/metadata.php

Delete topic playground ...

docker-compose run --rm php74 php examples/delete-topic.php -tplayground

Run tests

Tests use topics test*.

Updating Dependencies

docker-compose run --rm --no-deps php74 composer update

Run tests

docker-compose run --rm php74 vendor/bin/phpunit

Run tests with coverage

docker-compose run --rm php74 vendor/bin/phpunit --coverage-html build/coverage

Run tests against rdkafka extension / PHP 7.4

Updating Dependencies

docker-compose run --rm --no-deps php74 composer update -d /app/resources/test-extension --ignore-platform-reqs

Run tests

 docker-compose run --rm php74 php -dextension=rdkafka.so resources/test-extension/vendor/bin/phpunit -c resources/test-extension/phpunit.xml

Run benchmarks

Benchmarks use topic benchmarks.

Run & store benchmarks for ffi based rdkafka binding

docker-compose run --rm php74 phpbench run benchmarks --config=/app/benchmarks/ffi.json --report=default --store --tag=ffi

Run & store benchmarks for extension based rdkafka binding

docker-compose run --rm php74 phpbench run benchmarks --config=/app/benchmarks/ext.json --report=default --store --tag=ext    

Show comparison

docker-compose run --rm php74 phpbench report --uuid=tag:ffi --uuid=tag:ext --report='{extends: compare, compare: tag}'

Benchmarks

Just some first benchmarks based on PHP74-RC1, librdkafka v1.0.0, ext latest master, ffi without preload enabled.

benchmark subject set revs tag:ffi:mean tag:ext:mean
ConsumerBench benchConsume1Message 0 100 28,418.128μs 26,091.144μs
ConsumerBench benchConsume100Messages 0 100 40,182.836μs 38,656.394μs
ConsumerBench benchConsumeBatch100Messages 0 100 35,136.240μs 27,183.174μs
ProducerBench benchProduce1Message 0 100 4,076.694μs 3,563.914μs
ProducerBench benchProduce100Messages 0 100 3,663.710μs 3,731.920μs

See concrete benchmarks details for ffi & extension bindings.

Shutdown & cleanup

Shutdown and remove volumes:

docker-compose down -v

Todos

  • Callbacks
  • High Level KafkaConsumer
  • Tests, tests, tests, ... and travis
  • Support admin features
  • Compatible to librdkafka ^1.0.0
  • Benchmarking against rdkafka extension
  • Documentation
  • Compatible to rdkafka extension ^3.1.0
  • Generate binding class with https://github.com/ircmaxell/FFIMe / use default header file
  • Sig Handling & destruct (expect seg faults & lost msgs & shutdown hangs)
  • Provide ffi preload
You can’t perform that action at this time.