External Stream supports read/write Apache Pulsar #338
Description
Use case
Essentially we are building the best streaming SQL engine for live data. Apache Kafka and API-compatiable streaming platforms are well supported in Proton and Timeplus Cloud.
Currently Apache Pulsar is supported in Timeplus Cloud via other components. It will be great to have Proton support Apache Pulsar out of box. One workaround is to using KoP(Kafka on Pulsar) but native integration with Apache Pulsar is preferred.
We welcome the community to contribute more integrations for Proton with other systems. This ticket is specifically about the integration with Apache Pulsar via Proton External Stream.
Describe the solution you'd like
Similar to https://docs.timeplus.com/proton-kafka, we expect the users can create an external stream to connect Proton with the Apache Pulsar deployment, such as
CREATE EXTERNAL STREAM [IF NOT EXISTS] stream_name (<col_name1> <col_type>)
SETTINGS type='pulsar', brokers='ip:port',topic='..',security_protocol='..',username='..',password='..',sasl_mechanism='..'If this is applicable to Apache Pulsar, we expect various authentication and security protocols are supported, just like our Kafka integration
The supported values for security_protocol are:
- PLAINTEXT: when this option is omitted, this is also the default value.
- SASL_SSL: when this value is set, username and password should be specified.
The supported values for sasl_mechanism are:
- PLAIN: when you set security_protocol to SASL_SSL, this is the default value for sasl_mechanism.
- SCRAM-SHA-256
- SCRAM-SHA-512
It should supports both local deployment of Apache Pulsar or StreamNative Cloud.
Once the external stream for Apache Pulsar is created, the user can read from the stream to get existing or new messages in Pulsar, or write data to Pulsar.
Plaintext, Avro, Protobuf, JSON Schema format should be supported.
We should be able to read the entire message body as a string, or break them to multiple columns. Similar behavior/capabilities as the Kafka External Stream.
Describe alternatives you've considered
Additional context
Refer to https://docs.timeplus.com/proton-kafka to learn the expected feature set for the new external stream.
Refer to the Pulsar source/sink UI in Timeplus Cloud, e.g.
