Skip to content
This repository has been archived by the owner on Jan 24, 2024. It is now read-only.

handle batch messages when record is write to and read from pulsar #16

Closed
jiazhai opened this issue Jul 23, 2019 · 0 comments · Fixed by #42
Closed

handle batch messages when record is write to and read from pulsar #16

jiazhai opened this issue Jul 23, 2019 · 0 comments · Fixed by #42
Labels
type/feature Indicates new functionality

Comments

@jiazhai
Copy link
Contributor

jiazhai commented Jul 23, 2019

there is several things need pay attention:

  • try to make pulsar client also could read the added batch messages.
  • the original offset of each Record map to Pulsar BatchMessageId.
@jiazhai jiazhai added the type/feature Indicates new functionality label Jul 23, 2019
@jiazhai jiazhai mentioned this issue Jul 23, 2019
Merged
@jiazhai jiazhai mentioned this issue Sep 18, 2019
Merged
@sijie sijie closed this as completed in #42 Oct 4, 2019
sijie pushed a commit that referenced this issue Oct 4, 2019
@jiazhai @sijie
Add Batch support (#42)
4a88999 
Fix #16 
In KoP we would like to keep batch message. In Kafka message passed into Broker in format `Records`, each `Records` contains 1 or more `Record`. This is similar to the batched Message in Pulsar.
But because we have to turn Kafka `Records` into Pulsar BatchedMessage to make message could both be recognized by both Pulsar and Kafka client , we have to read each Record out from Records, and turn into Pulsar Message, This may involve some overhead of un-batch/re-batch. 

changes:
- move message produce/consume logic from KafkaRequestHandler.java into separate files.
- add support for batch produce/consume.
- add support for message headers.
- change offset format in MessageRecordUtils.java to support batch index.
- add test for added code.
michaeljmarshall pushed a commit to michaeljmarshall/kop that referenced this issue Dec 13, 2022
@eolivelli
Proxy: honour tlsEnabledWithBroker while using brokerWebServiceURLTLS…
fe404b4 
… and handle missing advertisedAddress (streamnative#16)
Demogorgon314 pushed a commit to Demogorgon314/kop that referenced this issue Aug 14, 2023
@BewareMyPower
Support GSSAPI mechanism for Kerberos authentication (streamnative#16)
7749dad 
Fixes streamnative/ksn#7

### Modifications

- Migrate `KerberosLogin` from Kafka for the Kerberos related
operations, it's simplified that many configurations are not
configurable.
- Implement a trivial `KerberosCallbackHandler`
- When the client sends authentication requests with GSSAPI mechanism,
create the `SaslServer` via `SaslServer` with the params that are parsed
from `KerberosLogin#subject()`.
- Use the same logic with Kafka to convert the original Kerberos
principal to the short principal. e.g.
  - `client@EXAMPLE.com` to `client`
  - `client/localhost@EXAMPLE.com` to `client`
- Add the `GssapiAuthenticationTest` to verify granting permission on a
Kerberos principal and the short name conversion.
- Add the `GsspiCustomAuthorizerTest` to verify it works with a custom
authorization provider.

New configs that are similar to Pulsar's configs for Kerberos:
| KoP config | Pulsar config |
| :- | :- |
| `kopKerberosServerSectionName` | `saslJaasServerSectionName` |
| `kopKerberosClientAllowedIds` | `saslJaasClientAllowedIds` |
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
type/feature Indicates new functionality
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Batch support
1 participant
@jiazhai