Skip to content

v2.8.0
Compare
Choose a tag to compare
@codelipenghui codelipenghui released this 15 Jun 10:39
· 5797 commits to master since this release
v2.8.0
89ac98e

PIPs

Transactions

  • Support pending ack state persistence #8881
  • Stable position and low watermark for the transaction buffer #9195
  • Transaction timeout support #9229
  • Transaction coordinator retry to complete the transaction #9236
  • Fix race condition when appending transaction log #9238
  • Transaction buffer snapshot #9490
  • Add metrics for transaction coordinator #9706
  • Clean the useless transaction individual acknowledgements based on low watermark #9722
  • Fix memory leak when deleting transaction marker #9751
  • Check the transaction state at the client side #9776
  • Clean aborted transactions for the transaction buffer #9974
  • Fix transaction coordinator retry to end transaction #10131
  • Fix NPE when opening a new transaction #10139
  • Fix transaction log failed to recover #10146
  • Fix transaction coordinator recover timeout #10162
  • Handling committing and aborting state when recovering transaction coordinator #10179
  • Fix NPE in case of enableTransactionCoordinator=false #10182
  • Fix transaction buffer client timeout #10206
  • Fix recover max local id issue for the transaction coordinator #10224
  • Support using transactions on standalone #10238
  • Fix transaction buffer lookup issue #10257
  • Fix transaction timeout issue at the client side #10323
  • Fix transaction client reconnect issue after transaction coordinator unloaded #10327
  • Fix transaction timeout not canceled after expired #10366
  • Transaction log low watermark optimization #10422
  • Fix the issue of transaction buffer client channel is inactive #10407
  • Add Admin API for getting transaction coordinator stats #10639
  • Add Admin API for getting transaction in transaction buffer stats #10642
  • Add Admin API for getting transaction in pending ack stats #10648
  • Add Admin API for getting transaction buffer stats and pending ack stats #10650
  • Add Admin API for getting transaction coordinator internal stats #10653
  • Setup transaction metadata with metadata store #10677
  • Fix issue with acknowledge messages from multiple subscriptions of a topic #10689
  • Admin API for getting transaction metadata #10690
  • Admin API for getting slow transactions #10701
  • Fix transaction log handle managed ledger WriteFail state #10711
  • Admin API for getting pending ack internal stats #10725
  • Fix transaction ack delete marker position when don't have transaction ack #10741
  • Fix transaction pending ack generate managedLedgerStats fail exception #10797
  • Use Zookeeper Prometheus metric provider to export Zookeeper metrics #10803
  • Always allow system topic for transaction buffer snapshot auto create #10876

Security

  • Optional auth method name header in HTTP authentication #6799
  • Topics level authorization support #7523
  • Authorization for setting topic/subscription auto-creation and subscription expire time on namespace #7900
  • Allow serializable stream-provider field into AuthenticationTls #10020
  • Add configuration for running OWASP Dependency Check for all modules #10288
  • Update default TLS protocols to TLSv1.3 and TLSv1.2 for the broker and proxy #10598

Broker

  • Share EventLoopGroup between broker and BookKeeper client #2603
  • Dispatch batch messages according to consumer permits 7266
  • Improve the max pending publish buffer mechanism 7406
  • Allow disabling HTTP TRACE/TRACK verbs #7907
  • Fix DispatchRateLimiter does not take effect #8611
  • Validate namespace isolation policy regex before updating #8804
  • Perform automatically cert refresh for pulsar-admin #8831
  • Fix updating rack info dynamically #8844
  • Fix deadlock when checking message expiration #8877
  • Support limit max topics per namespace #8942
  • Make ledger rollover check task internally #8946
  • Clean up topic that failed to unload from the cache #8968
  • Support get broker response for the message acknowledgement #8996
  • Support message index for a topic #9039
  • Supporting limit max topics per namespace by namespace policy #9042
  • Streaming dipsatcher support #9056
  • Make Netty acceptor threadPool size configurable #9061
  • Fix deadlock when unloading namespace bundles #9116
  • Fixed checking for maxTopicsPerNamespace #9121
  • Change the service URL to a not required param when creating a cluster #9127
  • Support setting replicator dispatch rate policy at the topic level #9175
  • Fix max topic for namespace does not work #9193
  • Fix memory leak of the managed ledger interceptor #9194
  • Fix max consumers per topic cannot be disabled at the namespace level #9214
  • Support schema REST API for V1 topics #9218
  • Fix peek message metadata while enabling broker entry metadata #9255
  • Support fetching metadata from entry data in publish callback #9257
  • Wait for the async broker port listener close operations to complete at shutdown #9308
  • Support script based mapping network topology #9363
  • Make managed ledger storage configurable #9397
  • Support setting enabled subscription types #9401
  • Fixed NPE and cache invalidation in leader election #9460
  • Fix exception when get an optional field for Protobuf message #9468
  • Ignore replicated subscription configurations from the client when disabled by broker #9523
  • Support expose producer metrics through Prometheus endpoint #9541
  • Fix NPE that occurs in PersistentStickyKeyDispatcherMultipleConsumers when debug log enabled #9587
  • Make LocalPolicies immutable to avoid concurrent modify inconsistent #9598
  • Fix writing/encoding issue of GenericJsonRecord #9608
  • Expose the native record for struct schema #9614
  • Add metrics for producer throttling #9649
  • Fix MaxUnackedMessagesPerConsumer cannot be changed dynamically and cannot be disabled #9657
  • Added support for force deleting tenant #9677
  • Fix managed ledger not found exception when force delete namespace #9691
  • Reduce CPU consumption of metrics creation #9735
  • Ensure read-lock is not continuously held on a section while iterating over concurrent maps #9787
  • Add pending read subscription metrics to stats-internal #9788
  • Allow broker to discover and unblock stuck subscription #9789
  • Disabled the tenants/namespaces force deletion by default #9819
  • Add metrics for the connections of the broker #9876
  • Make readModifyUpdate in MetadataCacheImpl thread-safe #9900
  • Optimize NamespaceBundle.toString() which is on the hot path #9976
  • Support set compaction threshold in broker.conf #9989
  • Support set properties for a namespace #10015
  • Fix cannot cleanup expired data after managed-ledger restart #10087
  • [Default configuration] Enable sticky read by default #10090
  • Add time based backlog quota #10093
  • Fix IllegalStateException in PersistentReplicator #10098
  • Support set max consumers per subscription for non-persistent dispatcher #10121
  • Limit the number of producers/consumers that can connect per topic for each IP address #10188
  • Close namespace clients when PulsarService is closed #10196
  • Shutdown Broker gracefully, but forcefully after brokerShutdownTimeoutMs #10199
  • Update the authentication data when an authentication refresh happens #10203
  • Add preciseTopicPublishRateLimiterEnable to broker.conf #10216
  • Fix the typo in the chunkedMessageRate stats and metrics #10223
  • Fix entry cache size to be a negative value #10225
  • Fix replicated subscriptions related LightProto issues #10247
  • Pause the replicated subscriptions snapshots when there is no real traffic #10292
  • Fix the inconsistency of advertised address #10312
  • Support listenerName for HttpLookupService #10319
  • Support truncate topic #10326
  • Fix authorization error if partition number of partitioned topic is updated 10333
  • Fix managed ledger name that transaction log used #10334
  • Catch topic policy not hit exception in handleSubscribe #10341
  • Fix ConcurrentModificationException when attempting to update local broker data #10347
  • Support seek to separate messageId/time for multiple topic reader #10348
  • Resource locks should automatically revalidate after a metadata session is re-established #10351
  • Add authentication data for the remote cluster #10357
  • Support array type claims in JWT #10375
  • Optimize locks in AuthenticationAthenz #10381
  • Prevent carrying state of PositionImplRecyclable when recycled #10404
  • Dispatch messages to consumer with permits #10417
  • Fix NPE in unblock stuck subscription task when dispatcher has not created #10430
  • Fix topic loading fails without any error when replicator init fails #10432
  • Set timeout to unblock web-threads on update partition API #10447
  • Fix CPU 100% when deleting namespace #10454
  • Remove residual info after forcibly deleting the namespace #10465
  • Fix potential memory leak of TopicPolicies #10466
  • Fix publish callback's entry data is null during ledger rollover #10467
  • Fix readModifyUpdate should return the new value in the future for metadata cache #10474
  • Fix partitioned system topic check issue #10529
  • Removed AspectJ based metrics for ZooKeeper #10533
  • Allow creating MetadataCache with custom serde #10543
  • Fix ack receipt version check issue #10551
  • Expose average message size metrics for a topic #10553
  • Fixed missed ZK caching when fetching list of namespaces for a tenant #10594
  • Setup pulsar cluster with MetadataStore #10600
  • Setup initial namespaces with MetadataStore #10612
  • Convert bundle split into an HTTP async operation #10619
  • Add metrics for non-contiguous deleted messages range #10638
  • Fix consumer related topic stats only available while consumer or reader are connected #10644
  • Allow configuring the number of BK client worker threads #10649
  • Fix ConcurrentOpenLongPairRangeSet remove all ranges #10656
  • Ensure all the ReadHandle gets properly closed on cache invalidation #10659
  • Avoid the context switch when managedLedgerNewEntriesCheckDelayInMillis=0 #10660
  • Allow configuring busy-wait in broker and client #10661
  • Use a single JWT parser instance #10664
  • Fix issues in advanceNonDurableCursors #10667
  • LockManager use a concurrent hash map to handle locks notifications #10680
  • Avoid making copies of internal maps when iterating #10691
  • Fix issue where StackOverflowError occurs when trying to redeliver a large number of already acked messages #10696
  • Fix NPE when filtering read entries #10704
  • Fix deadlock when enabling managed ledger interceptor #10706
  • Fix inconsistent behavior in LongPairRangeSet #10713
  • Fix repeated iterator generation of LongPairRangeSet #10722
  • Cancel scheduled tasks as the first step in closing for the managed ledger #10739
  • Prevent race conditions between timeout and completion of managed ledger #10740
  • Add advertised listener name for geo-replicator #10751
  • Add a read lock when traversing batchDeletedIndexes #10763
  • Replace use of Prometheus client CollectorRegistry.getSampleValue #10764
  • Release OpAddEntry.data when entry is copied and discarded for the managed ledger #10773
  • Avoid warning logs on topic policies not present #10785
  • Remove pulsar-zookeeper module and call zookeeper quorumPeerMain to start zookeeper #10823
  • Fix consumer stuck issue due to reuse entry wrapper #10824
  • Fix possible data race in getFirstAvailableConsumerPermits #10831

Functions

  • Update default function log location in function_worker.yaml #8470
  • Optimize batch source discovery and task ack #8498
  • Provide an interface for functions worker service #8560
  • Improve naming of threads used in batch source #8608
  • Fix the reader used to read the metadata topic during worker service initialization not been closed #8637
  • Add timeout to hasMessageAvailable to leader election process #8687
  • Support key based batch builder for Go Functions #8761
  • Fix panic when discarding message for Go Functions #8776
  • Move initialize dlog namespace metadata to bin/pulsar #8781
  • Support URL fetching for Go/Python Functions #8808
  • Add cmd flag for retaining key ordering #8886
  • Supports to use package command #8973
  • Support create and update sink with package name #8987
  • Support create and update source with package name #8988
  • Allow stats operations not to be blocked in functions #9005
  • Adding timeout to open table call for function state #9006
  • Fix get function info error of REST API #9115
  • Improve the error message when creating sinks #9131
  • Expose Pulsar Admin through Function context #9246
  • Enable Function Workers to use exclusive producer to write to internal topics #9275
  • Add additional logging when setting up state table for function instance #9304
  • Allow memory limit to be set for the pulsar client used in the ThreadRuntime in Pulsar Function #9320
  • Make admin operations on Statestore non blocking #9348
  • Fix maxPendingAsyncRequests not affect Kubernetes Runtime #9349
  • Add download directory support to function Kubernetes Runtime #9377
  • Optimize built-in source/sink startup by eliminating redundant NAR unpacking and checksum calculation #9413 #9500
  • Enhance Kubernetes manifest customizer with default options #9445
  • Fix possible deadlock on broker-function service startup #9499
  • Close InputStreams properly #9568
  • Add maximum allowed amount of resources setting for functions #9584
  • Support writing general records to Pulsar sink #9590
  • Move metrics port configuration to InstanceConfig #9610
  • Add resource granularity settings for functions #9736
  • Prevent invalid broker or proxy configuration for authorization #9746
  • Log stacktraces of threads that failed to terminate on shutdown within timeout in ExecutorProvider #9840
  • Support get partition index for a Record #9947
  • Deleting a Pulsar Function with a name that includes a colon character crashes the pulsar broker #9946
  • Exposing Prometheus metrics for Pulsar function local run mode #10156
  • Expose local run config metrics-port-start to CLI #10185
  • Fix Pulsar Function localrun with multiple instances and metrics server is enabled #10208
  • Improve localrun performance by using JVM options #10273
  • Fix function API can not use authdata to check superuser #10364
  • Fix potential bug getting stats and remove unnecessary error log #10500
  • Fix deadlock on Monitoring thread blocked by LeaderService.isLeader() #10502
  • Allow user to set custom configs to plugin worker service #10504
  • K8s Function Name Length Check Allows Invalid StatefulSet #10531
  • Get function cluster from broker config when start function worker with broker #10552
  • Process async results in the same Java runnable thread #10618
  • Support using AutoProduceBytesSchema as the function output schema #10716
  • ReflectionUtils use Class.forName in order to properly discover classes in Functions Runtime while using DefaultImplementation #10827
  • Fix the out of index issue when dispatch messages based on the avgBatchSizePerMsg #10828

IO Connectors

  • [Kafka Source Connector] Fix invalid topic name generation #9035
  • [Kafka Source Connector] Allow managing Avro encoded messages #9448
  • Configure Pulsar IO connector YAML file generator for all connectors #9629
  • [Kafka Source Connector] KeyValue schema support for KafkaBytesSource #10002
  • Allow using GenericObject for developing a Sink connector #10034
  • Sink unwrap internal AutoConsumeSchema and allow to handle topics with KeyValue schema #10211
  • Using ObjectMapper instead of Gson to parse Source/SInk configuration #10441
  • Expose subscription type in the SinkContext #10446
  • Add the ability to seek/pause/resume for a topic to the SinkContext #10498
  • Use Message.getReaderSchema() in Pulsar IO Sinks when possible #10557
  • [Kinesis Sink Connector] Fix backoff class not found #10744
  • [Kinesis Sink Connector] Fix does not acknowledge messages #10769
  • [Kafka Sink Adaptor] Support non-primitive schemas #10410

Pulsar SQL

  • Fix the misleading setting in presto configuration #8549
  • Fix injection factory cast error #9472
  • Add max split entry queue size bytes limitation #9628
  • Fix Pulsar SQL query bytes schema data error #9631
  • Fix Pulsar SQL query compression data #9663
  • Support native Protobuf decoder #9841
  • Support query uppercase topic name #9980
  • Only return persistent topic when list tables #10368
  • Fix Presto startup on JDK11 #10559
  • Fix pulsar sql issues when run select count(*) for the table with primary schema #10840

Tiered Storage

  • Fix AWS credentials usages #8950
  • Support ALI OSS tiered storage #8985

Proxy

  • Fix the proxy does not support partitioned topic auto-creation type #7903
  • Enable Conscrypt for Jetty in the Broker and in the Proxy #10541

Java Client

  • Fix warn log on the producer side when duplicated messages have been dropped #8729
  • Allow setting subscription name for Reader #8801
  • Fix authParams showing in log with secret string(*****) #8910
  • Avoid enabling DLQ on Key_Shared subscription #9163
  • Add support for the JSON format token #9313
  • Expose reached end of topic API for Reader/Consumer #9381
  • Fix returned a completed future when acknowledging a batch message before complete the individual acknowledgments #9383
  • Fix NPE when strip checksum for the producer #9400
  • Fix inconsistent equals and hashCode for MessageIds #9440
  • Allow disabling producer max queue size #9650
  • Include pulsar-client-admin-api in the shaded version of pulsar-client-admin #9689
  • Fix NPE in the ClientCnx #9761
  • Fix DLQ can't work with AUTO_CONSUME schema #9935
  • Fix NPE caused by null value of SchemaInfo's properties #9985
  • Support multi-topic reader #9995
  • Process messages from different partitions on different listener threads #10017
  • Ensure close resource to avoid memory leak #10028
  • Support set start message ID for each topic/partition on Reader #10033
  • Add a JSON RecordBuilder to the GenericJsonSchema #10052
  • Allow GenericRecord to wrap any Java Object #10057
  • Fix NPE while configuring consumer builder #10063
  • Support get native schema #10076
  • Support KeyValue on Message.getValue() when using AutoConsumeSchema #10107
  • Change the default retry topic name and dead letter topic name #10129
  • Fix KeyValue with SEPARATED encoding for the GenericObject #10186
  • Ensure download schema before decoding the payload for the AUTO_CONSUME schema #10248
  • Fix typo of the maxPendingChunkedMessage method #10223
  • Trait NONE schema as BYTE schema for AUTO_CONSUME schema #10277
  • Fix pause consume issue with MultiTopicsConsumerImpl #10305
  • Make message consumption thread safe and lock-free #10352
  • Reset state before recycling OpSendMsg instance #10405
  • Fix hasMessageAvailable return true but can't read message #10414
  • Fix NPE in GenericJsonRecord #10482
  • Fix behaviour of Schema.AUTO_CONSUME() with KeyValueSchema and multi versions #10492
  • Avoid sending flow requests with zero permits #10507
  • Make failPendingMessages called from within the ProducerImpl object mutex #10528
  • Add schemaType field in SchemaHash #10573
  • Fix NPE when ACK grouping tracker checks duplicated message id #10586
  • Support consume multiple schema types messages by AutoConsumeSchema #10604
  • Fixed issues in pulsar-client shading configuration #10614
  • MessageCrypto interface should not expose Netty ByteBuf class in the API #10616
  • Added org.apache.bookkeeper:cpu-affinity to shaded profile #10681
  • Skip the periodic re-check of the partitions count on non-partitioned topic #10708
  • Unlock the write lock of the UnAckedMessageTracker before call redeliverUnacknowledgedMessages #10768
  • Fix AutoConsumeSchema decode data without schema version #10811

C++ Client

  • Fix dangling reference bug in getRandomName #8596
  • Optimize batch message buffer allocation #8749
  • Make pool connections configurable in perf tools #8913
  • Support setting listener name #9119
  • Fix batch message handling of the UnAckedMessageTracker #9170
  • Fix ServerError is not converted to string in log #9277
  • Remove Boost::System runtime dependency #9498
  • Removed usages of boost::regex #9533
  • Account for different variables names on different CMake versions #9559
  • Allow to disable static or dynamic lib at build time #9570
  • Avoid multiple compilations of same source files #9675
  • Support configure debug level logs simply #10031
  • Add /opt/homebrew/ as a possible path for OpenSSL on Mac #10141
  • Fix race condition in MemoryLimitController #10142
  • Fix releasing semaphore and memory quota after send timeout #10144
  • Allow configuring memory limit from C API #10145
  • Fix use-after-free undefined behavior due to object lifetime problem #10220
  • Support enable replicate subscription #10243
  • Fix C++ client cannot be built with Boost <=1.53 #10307
  • Support check connect state #10349
  • Avoid sending flow requests with zero permits #10506
  • Add single file logger factory #10712
  • Reduce redeliverMessages when message listener is enabled #10726

Python Client

  • Replace Exceptions with PulsarExceptions #7600
  • Added logger wrapper support to make logging configurable #7713
  • Initial python 3.9 client wheel build support #9389
  • Support enable replicate subscription #10243

WebSocket

  • Negative acknowledge support #8249
  • Support deliverAt and deliverAfter attribute #8945
  • Fix returned status code does not depend on the exception type #9031
  • Allow to consume and pass message to client without decryption #10026
  • Support pong command #10035

Pulsar Admin

  • Support get applied message TTL policy for a topic #9225
  • Support get applied inactive topic policy for a topic #9230
  • Support get applied delayed delivery policy for a topic #9245
  • Support get applied max unacked message per subscription policy for a topic #9290
  • Support get applied max producer for a topic #9293
  • Support get applied max consumer for a topic #9296
  • Support get applied deduplication status policy for a topic #9339
  • Support get applied retention policy for a topic #9362
  • Support get applied offloader policy for a topic #9505
  • Support get applied max unacked messages per consumer for a topic #9694
  • Support get applied dispatch rate policy for a topic #9824
  • Support get applied subscription dispatch rate policy for a topic #9827
  • Support get applied backlog quota policy for a topic #9828
  • Support get applied persistence policy for a topic #9831
  • Support get applied cluster subscribe rate for a topic #9832
  • Support get applied replicator dispatch rate for a topic #9833
  • Support get applied compaction threshold #10038
  • Lazily init PulsarAdmin in PulsarAdminTool #9312
  • Fix create partition of existing topic does not throw RestException #9342
  • Support get service URL of the leader broker #9799
  • Support get persistent topics or non-persistent topics for pulsar admin client #9877
  • Fix can not disable and remove max consumer per subscription #10070
  • Add support for setting time based limit on backlog quota #10401
  • Make client-admin-api to use interfaces with builders instead of POJOs #10818

Tools

  • [Pulsar Perf] Support load WebSocket service URL from config file #9000)
  • [BookKeeper Shell] Make bookkeeper shell more user friendly #9281
  • [Client Tools] Supports end-to-end encryption #9615
  • [Pulsar Perf] Support specify topics and subscriptions #9716
  • [Client Tools] Allow printing GenericRecord contents #9785
  • [Pulsar Perf] Fix compatibility issues with previous version #9838
  • [Pulsar Perf] Add ability to create partitioned topics #9859
  • [Client Tools] handle data with different schemas when consuming messages #10301
  • [Client Tools] Fix NoClassDefFoundError when running pulsar cli tools in dev directory #10807

Dependencies

Assets 2