Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pulsar: init at 2.6.0 #91348

Closed

Conversation

@samdroid-apps
Copy link
Contributor

samdroid-apps commented Jun 23, 2020

Motivation for this change

This PR adds a package for Apache Pulsar, based on the Apache Zookeeper package. In a followup PR, I'll add a service to run Pulsar.

Testing pulsar
Pulsar requires configuration to start, here's an example script to start all the pieces:

After building pulsar to ./result (nix-build -A pulsar ~/nixpkgs/default.nix):

export PULSAR_LOG_DIR=/tmp/pulsar/logs
mkdir -p /tmp/pulsar/conf
export PULSAR_ZK_CONF=/tmp/pulsar/conf/zookeeper.conf
cat <<-EOF > /tmp/pulsar/conf/zookeeper.conf
  # remember to "echo 1 > /tmp/pulsar/zookeeper/myid"
  clientPort=2181
  dataDir=/tmp/pulsar/zookeeper
  autopurge.purgeInterval=1
  initLimit=5
  syncLimit=2
  tickTime=2000
EOF
export PULSAR_BOOKKEEPER_CONF=/tmp/pulsar/conf/bookkeeper.conf
cat <<-EOF > /tmp/pulsar/conf/bookkeeper.conf
  zkServers=localhost:2181
  advertisedAddress=127.0.0.1

  journalDirectories=/tmp/pulsar/bk-journal
  ledgerDirectories=/tmp/pulsar/bk-ledger
  storage.range.store.dirs=/tmp/pulsar/bk-range-store-dirs
EOF
export PULSAR_BROKER_CONF=/tmp/pulsar/conf/broker.conf
cat <<-EOF > /tmp/pulsar/conf/broker.conf
  zookeeperServers=127.0.0.1:2181
  configurationStoreServers=127.0.0.1:2181
  clusterName=pulsar-cluster-1

  # to not conflict with the bookkeeper admin interface
  webServicePort=8081

  # running it locally, we only have 1 bookkeeper node
  managedLedgerDefaultEnsembleSize=1
  managedLedgerDefaultWriteQuorum=1
  managedLedgerDefaultAckQuorum=1
EOF
export PULSAR_CLIENT_CONF=/tmp/pulsar/conf/client.conf
cat <<-EOF > /tmp/pulsar/conf/client.conf
  webServiceUrl=http://127.0.0.1:8081
  brokerServiceurl=pulsar://127.0.0.1:6650
EOF

# Start zookeeper, using the bundled pulsar version
result/bin/pulsar zookeeper &
# One time setup, required to start bookkeeper
result/bin/pulsar initialize-cluster-metadata \
    --cluster pulsar-cluster-1 \
    --zookeeper 127.0.0.1:2181 \
    --configuration-store 127.0.0.1:2181 \
    --web-service-url http://127.0.0.1:8080 \
    --broker-service-url pulsar://127.0.0.1:6650
# Start bookkeeper using the bundled pulsar version
result/bin/pulsar bookie &

# Start pulsar message broker
result/bin/pulsar broker

You can create a test message consumer in another terminal:

result/bin/pulsar-client consume "test-topic" --subscription-name "test-subscription"

And send a message in a separate terminal:

result/bin/pulsar-client produce "test-topic" --messages "Yay a test message!"
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@danieldk
Copy link
Member

danieldk commented Jun 24, 2020

This change adds pkgs/servers/default.nix rather than pkgs/servers/pulsar/default.nix.

@samdroid-apps samdroid-apps force-pushed the samdroid-apps:sparky/add-pulsar-package branch from 7866b78 to 817d319 Jun 25, 2020
@samdroid-apps samdroid-apps force-pushed the samdroid-apps:sparky/add-pulsar-package branch from 817d319 to 46cae93 Jun 25, 2020
@samdroid-apps
Copy link
Contributor Author

samdroid-apps commented Jun 25, 2020

🤦 Good point @danieldk, I've updated with correct paths this time.

@samdroid-apps
Copy link
Contributor Author

samdroid-apps commented Jul 1, 2020

Closing in favor of #91837, which also includes a nixos module and tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.