If you are new to DataStax Luna Streaming and its Apache Pulsar enhancements, these FAQs are for you.
DataStax Luna Streaming is a new Kubernetes-based distribution of Apache Pulsar, based on the technology that Kesque built to run its Pulsar-as-a-service.
In addition to Apache Pulsar itself, DataStax Luna Streaming provides:
-
An installer that can stand up a dev or production cluster on bare metal or VMs without a pre-existing Kubernetes environment
-
A helm chart that can deploy and manage Pulsar on your current Kubernetes infrastructure
-
Cassandra, Elastic, Kinesis, Kafka, and JDBC connectors
-
A management dashboard
-
A monitoring and alerting system
DataStax Luna Streaming {luna-version} is based on its distribution of Apache Pulsar {pulsar-version}, plus features and additional enhancements from DataStax contributors.
DataStax Luna Streaming is a hardened version of Apache Pulsar that been run through additional testing to ensure it is ready for production use. It also includes additional tooling to help monitor your system, including an enhanced Admin Console and a Heartbeat service to monitor the system health.
Yes, DataStax Luna Streaming is open source. See the repos FAQ.
They include Minikube, K8d, Kind, Google Kubernetes Engine (GKE), Microsoft Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS), and other commonly used platforms.
There are several public repos, each with a different purpose. See:
-
https://github.com/datastax/pulsar : This is the distro repo (a fork of apache/pulsar).
-
https://github.com/datastax/pulsar-admin-console : This is the repo for the Pulsar admin console, which allows for the configuration and monitoring of Pulsar.
-
https://github.com/datastax/pulsar-heartbeat : This is a monitoring/observability tool for Pulsar that tracks the health of the cluster and can generate alerts in Slack and OpsGenie.
-
https://github.com/datastax/pulsar-helm-chart : This is the Helm chart for deploying the DataStax Pulsar Distro in an existing Kubernetes cluster.
-
https://github.com/datastax/pulsar-sink : This is the DataStax Apache Pulsar Connector (
pulsar-sink
for Cassandra) repo. -
https://github.com/datastax/burnell : This is a utility for Pulsar that provides various functions, such as key initialization for authentication, and JWT token creation API.
The DataStax Luna Streaming distribution is designed for Java 17. However, because the product releases Docker images, you do not need to install Java 17 in advance. Java 17 is bundled in the Docker image.
-
Use the Helm chart provided at https://github.com/datastax/pulsar-helm-chart to install DataStax Luna Streaming in an existing Kubernetes cluster on your laptop or hosted by a cloud provider.
-
Use the tarball provided at https://github.com/datastax/pulsar/releases to install DataStax Luna Streaming on a server or VM.
-
Use the DataStax Ansible scripts provided at https://github.com/datastax/pulsar-ansible to install DataStax Luna Streaming on a server or VM with our provided playbooks.
Follow the full instructions in Quick Start for Helm Chart installs.
From the Admin Console, you can:
-
Add and run Pulsar clients
-
Establish credentials for secure connections
-
Define topics that can be published for streaming apps
-
Set up Pulsar sinks that publish topics and make them available to subscribers, such as for a Cassandra database table
-
Control namespaces used by Pulsar
-
Use the Admin API
Pulsar Heartbeat monitors the availability, tracks the performance, and reports failures of the Pulsar cluster. It produces synthetic workloads to measure end-to-end message pubsub latency. Pulsar Heartbeat is a cloud-native application that can be installed by Helm within the Pulsar Kubernetes cluster.
Prometheus is an open-source tool to collect metrics on a running app, providing real-time monitoring and alerts.
Grafana is a visualization tool that helps you make sense of metrics and related data coming from your apps via Prometheus, for example.
What are the features provided by DataStax Apache Pulsar Connector (pulsar-sink
) that are not supported in kafka-sink
?
The Pulsar IO framework provides many features that are not possible in Kafka, and has different compression formats and auth/security features. The features are handled by Pulsar. For more, see Luna Streaming IO Connectors.
The DataStax Apache Pulsar Connector allows single-record acknowledgement and negative acknowledgements.
What features are missing in DataStax Apache Pulsar Connector (pulsar-sink
) compared with kafka-sink
?
-
No support for
tinyint
(int8bit
) andsmallint
(int16bit
). -
The key is always a String, but you can write JSON inside it; the support is implemented in pulsar-sink, but not in Pulsar IO.
-
The “value” of a “message property” is always a String; for example, you cannot map the message property to
ttl
or totimestamp
. -
Field names inside structures must be valid for Avro, even in case of JSON structures. For example, field names like
Int.field
(with dot) orint field
(with space) are not valid.
There are two packages:
-
The
pulsar-sink
functionality of DataStax Apache Pulsar Connector is included with DataStax Luna Streaming. It’s built in! -
You can optionally download the DataStax Apache Pulsar Connector tarball from the DataStax Downloads site, and then use it as its own product with your open-source Apache Pulsar install.
If you’re using open-source software (OSS) Apache Pulsar, you can use DataStax Apache Pulsar Connector with the OSS to take advantage of this pulsar-sink
for Cassandra. See the DataStax Apache Pulsar Connector documentation.
The same as for Apache Pulsar. See https://pulsar.apache.org/docs/en/client-libraries/.