Skip to content

Support protobuf native schema#486

Merged
BewareMyPower merged 4 commits into
apache:masterfrom
shibd:protobuf_native
May 26, 2026
Merged

Support protobuf native schema#486
BewareMyPower merged 4 commits into
apache:masterfrom
shibd:protobuf_native

Conversation

@shibd
Copy link
Copy Markdown
Member

@shibd shibd commented May 25, 2026

Fixes #

Master Issue: #

Motivation

Support protobuf native schema

Modifications

  1. Support protobuf native schema
  2. Provide tools to help the user create schema info for the protobuf native schema.

Verifying this change

  • Added protobuf schema end-to-end test

Documentation

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@shibd shibd self-assigned this May 25, 2026
@shibd shibd added this to the 1.18.0 milestone May 25, 2026
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds first-class support for Pulsar Protobuf Native schema in the Node client by exposing helper utilities to generate the required schema info from protobufjs descriptor output, updating the native binding schema plumbing, and adding an end-to-end test validating produce/consume behavior.

Changes:

  • Add ProtobufNativeSchema helper module (root creation from reflection JSON + schema info generation using FileDescriptorSet).
  • Extend schema type handling in the native binding to support ProtobufNative.
  • Add an end-to-end Jest test (with generated protobuf artifacts) and include protobufjs as a dependency.

Reviewed changes

Copilot reviewed 7 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/SchemaInfo.h Switch schema representation to Pulsar C++ SchemaType + std::map properties.
src/SchemaInfo.cc Add ProtobufNative mapping and set schema on producer/consumer configs via C++ API.
src/ProtobufNativeSchema.js New helper utilities to build Protobuf Native schema info from protobufjs.
index.js Export ProtobufNativeSchema from the main module.
index.d.ts Add ProtobufNativeSchema typings and extend SchemaType union.
package.json Add protobufjs dependency.
yarn.lock / package-lock.json Lockfile updates for protobufjs (and transitive deps).
.eslintignore Ignore generated protobuf artifacts under tests.
tests/protobuf_native_schema.test.js New end-to-end test for Protobuf Native schema produce/consume.
tests/protobuf_schema/generated/* Generated protobuf JS/TS artifacts + reflection JSON used by the test.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/SchemaInfo.cc
Comment thread src/SchemaInfo.cc
Comment thread src/ProtobufNativeSchema.js Outdated
Copy link
Copy Markdown
Contributor

@BewareMyPower BewareMyPower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you verify if it can work with Java client? You can fork https://github.com/BewareMyPower/pulsar-protobuf-native-demo to test the latest implementation.

@shibd
Copy link
Copy Markdown
Member Author

shibd commented May 26, 2026

Could you verify if it can work with Java client? You can fork https://github.com/BewareMyPower/pulsar-protobuf-native-demo to test the latest implementation.

Sure, I verified the following case:

  1. A Node.js client sending messages and a Java client consuming them.
  2. A Java client sending messages and a Node.js client consuming them.

After this PR is released, I’ll prepare a repository like the one you mentioned so everyone can validate it more easily.

@shibd shibd requested a review from BewareMyPower May 26, 2026 01:19
@BewareMyPower BewareMyPower merged commit 5f5d0cb into apache:master May 26, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants