Skip to content

Commit

Permalink
Grpc upgrade; BK upgrade for LS 2.7 (#14)
Browse files Browse the repository at this point in the history
* [Issue 9728] Upgrade GRPC and bookkeeper (apache#9729)

* Upgraded bookkeeper to 4.13.0; grpc to 1.33

Fixes apache#10937

*Motivation*

The previours bk version was introduce an incompatible issue
with BouncyCastle FIPS. BookKeeper already fix this and we
should upgrade the version to resolve the issue.

For more information: apache#10937

*Modifications*

* Upgrade to BookKeeper 4.14.3 (apache#12760)

* Fixed additional post-cherry-pick conflicts

Co-authored-by: hangc0276 <chenhang@apache.org>
Co-authored-by: chenhang <chenhang@bigo.sg>
Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com>
Co-authored-by: Matteo Merli <mmerli@apache.org>
  • Loading branch information
5 people committed Dec 1, 2021
1 parent 2a893c9 commit 116f90e
Show file tree
Hide file tree
Showing 25 changed files with 208 additions and 72 deletions.
6 changes: 6 additions & 0 deletions distribution/server/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>compile</scope>
</dependency>

<!-- bookkeeper http server -->
<dependency>
<groupId>org.apache.bookkeeper.http</groupId>
Expand Down
104 changes: 59 additions & 45 deletions distribution/server/src/assemble/LICENSE.bin.txt
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,11 @@ The Apache Software License, Version 2.0
- com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jsonSchema-2.11.1.jar
- org.codehaus.jackson-jackson-core-asl-1.9.11.jar
- org.codehaus.jackson-jackson-mapper-asl-1.9.11.jar
* Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
* Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.1.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.17.0.jar
* Joda -- org.joda-joda-convert-2.2.1.jar
* Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.2.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.12.0.jar
Expand Down Expand Up @@ -396,32 +400,31 @@ The Apache Software License, Version 2.0
- org.apache.logging.log4j-log4j-web-2.10.0.jar
* Java Native Access JNA -- net.java.dev.jna-jna-4.2.0.jar
* BookKeeper
- org.apache.bookkeeper-bookkeeper-common-4.12.0.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.12.0.jar
- org.apache.bookkeeper-bookkeeper-proto-4.12.0.jar
- org.apache.bookkeeper-bookkeeper-server-4.12.0.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.12.0.jar
- org.apache.bookkeeper-circe-checksum-4.12.0.jar
- org.apache.bookkeeper-cpu-affinity-4.12.0.jar
- org.apache.bookkeeper-statelib-4.12.0.jar
- org.apache.bookkeeper-stream-storage-api-4.12.0.jar
- org.apache.bookkeeper-stream-storage-common-4.12.0.jar
- org.apache.bookkeeper-stream-storage-java-client-4.12.0.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.12.0.jar
- org.apache.bookkeeper-stream-storage-proto-4.12.0.jar
- org.apache.bookkeeper-stream-storage-server-4.12.0.jar
- org.apache.bookkeeper-stream-storage-service-api-4.12.0.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.12.0.jar
- org.apache.bookkeeper.http-http-server-4.12.0.jar
- org.apache.bookkeeper.http-vertx-http-server-4.12.0.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.12.0.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.12.0.jar
- org.apache.bookkeeper.tests-stream-storage-tests-common-4.12.0.jar
- org.apache.distributedlog-distributedlog-common-4.12.0.jar
- org.apache.distributedlog-distributedlog-core-4.12.0-tests.jar
- org.apache.distributedlog-distributedlog-core-4.12.0.jar
- org.apache.distributedlog-distributedlog-protocol-4.12.0.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.12.0.jar
- org.apache.bookkeeper-bookkeeper-common-4.14.3.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.14.3.jar
- org.apache.bookkeeper-bookkeeper-proto-4.14.3.jar
- org.apache.bookkeeper-bookkeeper-server-4.14.3.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.14.3.jar
- org.apache.bookkeeper-circe-checksum-4.14.3.jar
- org.apache.bookkeeper-cpu-affinity-4.14.3.jar
- org.apache.bookkeeper-statelib-4.14.3.jar
- org.apache.bookkeeper-stream-storage-api-4.14.3.jar
- org.apache.bookkeeper-stream-storage-common-4.14.3.jar
- org.apache.bookkeeper-stream-storage-java-client-4.14.3.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.14.3.jar
- org.apache.bookkeeper-stream-storage-proto-4.14.3.jar
- org.apache.bookkeeper-stream-storage-server-4.14.3.jar
- org.apache.bookkeeper-stream-storage-service-api-4.14.3.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.14.3.jar
- org.apache.bookkeeper.http-http-server-4.14.3.jar
- org.apache.bookkeeper.http-vertx-http-server-4.14.3.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.14.3.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.14.3.jar
- org.apache.distributedlog-distributedlog-common-4.14.3.jar
- org.apache.distributedlog-distributedlog-core-4.14.3-tests.jar
- org.apache.distributedlog-distributedlog-core-4.14.3.jar
- org.apache.distributedlog-distributedlog-protocol-4.14.3.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.14.3.jar
* Apache HTTP Client
- org.apache.httpcomponents-httpclient-4.5.5.jar
- org.apache.httpcomponents-httpcore-4.4.9.jar
Expand Down Expand Up @@ -453,25 +456,34 @@ The Apache Software License, Version 2.0
* SnakeYaml -- org.yaml-snakeyaml-1.26.jar
* RocksDB - org.rocksdb-rocksdbjni-6.10.2.jar
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.3.4.jar
* Apache Thrifth - org.apache.thrift-libthrift-0.12.0.jar
* Apache Thrifth - org.apache.thrift-libthrift-0.14.2.jar
* OkHttp
- com.squareup.okhttp-okhttp-2.5.0.jar
- com.squareup.okhttp-okhttp-2.7.4.jar
* Okio - com.squareup.okio-okio-1.13.0.jar
* Javassist -- org.javassist-javassist-3.25.0-GA.jar
* gRPC
- io.grpc-grpc-all-1.18.0.jar
- io.grpc-grpc-auth-1.18.0.jar
- io.grpc-grpc-context-1.18.0.jar
- io.grpc-grpc-core-1.18.0.jar
- io.grpc-grpc-netty-1.18.0.jar
- io.grpc-grpc-okhttp-1.18.0.jar
- io.grpc-grpc-protobuf-1.18.0.jar
- io.grpc-grpc-protobuf-lite-1.18.0.jar
- io.grpc-grpc-stub-1.18.0.jar
- io.grpc-grpc-protobuf-nano-1.18.0.jar
* gRPC
- io.grpc-grpc-all-1.33.0.jar
- io.grpc-grpc-auth-1.33.0.jar
- io.grpc-grpc-context-1.33.0.jar
- io.grpc-grpc-core-1.33.0.jar
- io.grpc-grpc-netty-1.33.0.jar
- io.grpc-grpc-okhttp-1.33.0.jar
- io.grpc-grpc-protobuf-1.33.0.jar
- io.grpc-grpc-protobuf-lite-1.33.0.jar
- io.grpc-grpc-stub-1.33.0.jar
- io.grpc-grpc-testing-1.33.0.jar
- io.grpc-grpc-alts-1.33.0.jar
- io.grpc-grpc-api-1.33.0.jar
- io.grpc-grpc-grpclb-1.33.0.jar
- io.grpc-grpc-netty-shaded-1.33.0.jar
- io.grpc-grpc-services-1.33.0.jar
- io.grpc-grpc-xds-1.33.0.jar
* Perfmark
- io.perfmark-perfmark-api-0.19.0.jar
* OpenCensus
- io.opencensus-opencensus-api-0.18.0.jar
- io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar
- io.opencensus-opencensus-contrib-http-util-0.24.0.jar
- io.opencensus-opencensus-proto-0.2.0.jar
* Jodah
- net.jodah-typetools-0.5.0.jar
* Apache Avro
Expand Down Expand Up @@ -515,10 +527,16 @@ The Apache Software License, Version 2.0
- io.vertx-vertx-web-common-3.9.7.jar
* Apache ZooKeeper
- org.apache.zookeeper-zookeeper-jute-3.5.9.jar
* Google HTTP Client
- com.google.http-client-google-http-client-jackson2-1.34.0.jar
- com.google.http-client-google-http-client-1.34.0.jar
- com.google.auto.value-auto-value-annotations-1.7.jar
- com.google.re2j-re2j-1.2.jar

BSD 3-clause "New" or "Revised" License
* Google auth library
- com.google.auth-google-auth-library-credentials-0.9.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-credentials-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-oauth2-http-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
* LevelDB -- (included in org.rocksdb.*.jar) -- licenses/LICENSE-LevelDB.txt
* JSR305 -- com.google.code.findbugs-jsr305-3.0.2.jar -- licenses/LICENSE-JSR305.txt

Expand All @@ -531,17 +549,13 @@ MIT License
- org.slf4j-jul-to-slf4j-1.7.25.jar
- org.slf4j-slf4j-api-1.7.25.jar
- org.slf4j-jcl-over-slf4j-1.7.25.jar
* Animal Sniffer Annotations
- org.codehaus.mojo-animal-sniffer-annotations-1.17.jar
* The Checker Framework
- org.checkerframework-checker-compat-qual-2.5.2.jar
- org.checkerframework-checker-qual-3.5.0.jar

Protocol Buffers License
* Protocol Buffers
- com.google.protobuf-protobuf-java-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf-protobuf-java-util-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar -- licenses/LICENSE-protobuf.txt

CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Java Annotations API
Expand Down
1 change: 0 additions & 1 deletion distribution/server/src/assemble/NOTICE.bin.txt
Original file line number Diff line number Diff line change
Expand Up @@ -201,7 +201,6 @@ http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html
* Performance and memory usage improvement

------------------------------------------------------------------------------------
- org.checkerframework-checker-compat-qual-2.5.2.jar

A few parts of the Checker Framework have more permissive licenses.

Expand Down
5 changes: 5 additions & 0 deletions docker/pulsar-all/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@
<type>pom</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-offloader-distribution</artifactId>
Expand Down
19 changes: 16 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ flexible messaging model and an intuitive client API.</description>
<!-- apache commons -->
<commons-compress.version>1.19</commons-compress.version>

<bookkeeper.version>4.12.0</bookkeeper.version>
<bookkeeper.version>4.14.3</bookkeeper.version>
<zookeeper.version>3.5.9</zookeeper.version>
<netty.version>4.1.60.Final</netty.version>
<netty-tc-native.version>2.0.36.Final</netty-tc-native.version>
Expand Down Expand Up @@ -131,8 +131,8 @@ flexible messaging model and an intuitive client API.</description>
<protobuf2.version>2.4.1</protobuf2.version>
<protobuf3.version>3.11.4</protobuf3.version>
<protoc3.version>${protobuf3.version}</protoc3.version>
<grpc.version>1.18.0</grpc.version>

<grpc.version>1.33.0</grpc.version>
<perfmark.version>0.19.0</perfmark.version>
<protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
<gson.version>2.8.6</gson.version>
<sketches.version>0.8.3</sketches.version>
Expand Down Expand Up @@ -879,6 +879,19 @@ flexible messaging model and an intuitive client API.</description>
<version>${grpc.version}</version>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<version>${perfmark.version}</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<artifactId>error_prone_annotations</artifactId>
<groupId>com.google.errorprone</groupId>
</exclusion>
</exclusions>
</dependency>

<dependency>

<groupId>io.grpc</groupId>
Expand Down
1 change: 1 addition & 0 deletions pulsar-broker-shaded/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
<include>net.java.dev.jna:*</include>
<include>com.carrotsearch:*</include>
<include>io.prometheus:*</include>
<include>io.perfmark:*</include>
<include>com.github.ben-manes.caffeine:*</include>
<include>org.glassfish.jersey.*:*</include>
<include>org.rocksdb:*</include>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ public class DataSketchesOpStatsLogger implements OpStatsLogger {
private final LongAdder successSumAdder = new LongAdder();
private final LongAdder failSumAdder = new LongAdder();

DataSketchesOpStatsLogger() {
public DataSketchesOpStatsLogger() {
this.current = new ThreadLocalAccessor();
this.replacement = new ThreadLocalAccessor();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public class DataSketchesSummaryLogger {
private final LongAdder countAdder = new LongAdder();
private final LongAdder sumAdder = new LongAdder();

DataSketchesSummaryLogger() {
public DataSketchesSummaryLogger() {
this.current = new ThreadLocalAccessor();
this.replacement = new ThreadLocalAccessor();
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.pulsar.broker.stats.prometheus.metrics;

import java.util.concurrent.atomic.LongAdder;
import org.apache.bookkeeper.stats.Counter;

/**
* {@link Counter} implementation based on {@link LongAdder}.
*
* <p>LongAdder keeps a counter per-thread and then aggregates to get the result, in order to avoid contention between
* multiple threads.
*/
public class LongAdderCounter implements Counter {
private final LongAdder counter = new LongAdder();

public LongAdderCounter() {

}

@Override
public void clear() {
counter.reset();
}

@Override
public void inc() {
counter.increment();
}

@Override
public void dec() {
counter.decrement();
}

@Override
public void add(long delta) {
counter.add(delta);
}

@Override
public Long get() {
return counter.sum();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ public class ObserverGauge extends SimpleCollector<ObserverGauge.Child> implemen
public static class Builder extends SimpleCollector.Builder<Builder, ObserverGauge> {
private Supplier<Number> supplier;

public Builder() {}

public Builder supplier(Supplier<Number> supplier) {
this.supplier = supplier;
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,9 @@ public class PrometheusMetricsProvider implements StatsProvider {
/**
* These acts a registry of the metrics defined in this provider
*/
final ConcurrentMap<String, LongAdderCounter> counters = new ConcurrentSkipListMap<>();
final ConcurrentMap<String, SimpleGauge<? extends Number>> gauges = new ConcurrentSkipListMap<>();
final ConcurrentMap<String, DataSketchesOpStatsLogger> opStats = new ConcurrentSkipListMap<>();
public final ConcurrentMap<String, LongAdderCounter> counters = new ConcurrentSkipListMap<>();
public final ConcurrentMap<String, SimpleGauge<? extends Number>> gauges = new ConcurrentSkipListMap<>();
public final ConcurrentMap<String, DataSketchesOpStatsLogger> opStats = new ConcurrentSkipListMap<>();

public PrometheusMetricsProvider() {
this.cachingStatsProvider = new CachingStatsProvider(new StatsProvider() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@
import org.apache.bookkeeper.stats.StatsLogger;
import org.apache.bookkeeper.stats.prometheus.LongAdderCounter;

import java.util.HashMap;

/**
* A {@code Prometheus} based {@link StatsLogger} implementation.
*/
Expand All @@ -34,7 +36,7 @@ public class PrometheusStatsLogger implements StatsLogger {
private final PrometheusMetricsProvider provider;
private final String scope;

PrometheusStatsLogger(PrometheusMetricsProvider provider, String scope) {
public PrometheusStatsLogger(PrometheusMetricsProvider provider, String scope) {
this.provider = provider;
this.scope = scope;
}
Expand All @@ -46,7 +48,7 @@ public OpStatsLogger getOpStatsLogger(String name) {

@Override
public Counter getCounter(String name) {
return provider.counters.computeIfAbsent(completeName(name), x -> new LongAdderCounter());
return provider.counters.computeIfAbsent(completeName(name), x -> new LongAdderCounter(new HashMap<>()));
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}

static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry) throws IOException {
public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
throws IOException {
Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
while (metricFamilySamples.hasMoreElements()) {
MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ public SimpleGauge(final Gauge<T> gauge) {
this.gauge = gauge;
}

Number getSample() {
public Number getSample() {
return gauge.getSample();
}
}
1 change: 1 addition & 0 deletions pulsar-client-admin-shaded/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>com.squareup.*:*</include>
<include>com.google.*:*</include>
Expand Down
1 change: 1 addition & 0 deletions pulsar-client-all/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>io.netty:*</include>
<include>com.squareup.*:*</include>
Expand Down
Loading

0 comments on commit 116f90e

Please sign in to comment.