Skip to content

Commit

Permalink
Merge remote-tracking branch 'apache/master' into ZOOKEEPER-3188
Browse files Browse the repository at this point in the history
  • Loading branch information
Mate Szalay-Beko committed Oct 4, 2019
2 parents e823af4 + 874aaf1 commit 05eae83
Show file tree
Hide file tree
Showing 105 changed files with 4,775 additions and 2,061 deletions.
2 changes: 1 addition & 1 deletion README.md
@@ -1,4 +1,4 @@
# Apache ZooKeeper
# Apache ZooKeeper [![Build Status](https://travis-ci.org/apache/zookeeper.svg?branch=master)](https://travis-ci.org/apache/zookeeper) [![Maven Central](https://img.shields.io/maven-central/v/org.apache.zookeeper/zookeeper)](https://zookeeper.apache.org/releases.html) [![License](https://img.shields.io/github/license/apache/zookeeper)](https://github.com/apache/zookeeper/blob/master/LICENSE.txt)
![alt text](https://zookeeper.apache.org/images/zookeeper_small.gif "ZooKeeper")

For the latest information about Apache ZooKeeper, please visit our website at:
Expand Down
4 changes: 2 additions & 2 deletions build.xml
Expand Up @@ -37,7 +37,7 @@ xmlns:cs="antlib:com.puppycrawl.tools.checkstyle.ant">

<property name="audience-annotations.version" value="0.5.0" />

<property name="netty.version" value="4.1.36.Final"/>
<property name="netty.version" value="4.1.42.Final"/>

<property name="junit.version" value="4.12"/>
<property name="mockito.version" value="2.27.0"/>
Expand All @@ -55,7 +55,7 @@ xmlns:cs="antlib:com.puppycrawl.tools.checkstyle.ant">
<property name="javacc.version" value="5.0"/>

<property name="jetty.version" value="9.4.18.v20190429"/>
<property name="jackson.version" value="2.9.9.3"/>
<property name="jackson.version" value="2.9.10"/>
<property name="dependency-check-ant.version" value="4.0.2"/>

<property name="commons-io.version" value="2.6"/>
Expand Down
File renamed without changes.
23 changes: 23 additions & 0 deletions dev/docker/Dockerfile
@@ -0,0 +1,23 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#

FROM maven:3.6.1-jdk-11

RUN apt-get update
RUN apt-get install -y g++ cmake autoconf libcppunit-dev libtool
72 changes: 72 additions & 0 deletions dev/docker/run.sh
@@ -0,0 +1,72 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -e -x -u

SCRIPT_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

export IMAGE_NAME="zookeeper/dev"

pushd ${SCRIPT_DIR}

docker build --rm=true -t ${IMAGE_NAME} .

popd

if [ "$(uname -s)" == "Linux" ]; then
USER_NAME=${SUDO_USER:=$USER}
USER_ID=$(id -u "${USER_NAME}")
GROUP_ID=$(id -g "${USER_NAME}")
LOCAL_HOME=$(realpath ~)
else # boot2docker uid and gid
USER_NAME=$USER
USER_ID=1000
GROUP_ID=50
LOCAL_HOME="/Users/${USER_NAME}"
fi

docker build -t "${IMAGE_NAME}-${USER_NAME}" - <<UserSpecificDocker
FROM ${IMAGE_NAME}
RUN groupadd --non-unique -g ${GROUP_ID} ${USER_NAME} && \
useradd -g ${GROUP_ID} -u ${USER_ID} -k /root -m ${USER_NAME}
ENV HOME /home/${USER_NAME}
UserSpecificDocker

ZOOKEEPER_ROOT=${SCRIPT_DIR}/../..

CMD="
echo
echo 'Welcome to Apache ZooKeeper Development Env'
echo 'To build, execute'
echo ' mvn clean install'
echo
bash
"

pushd ${ZOOKEEPER_ROOT}

docker run -i -t \
--rm=true \
-w ${ZOOKEEPER_ROOT} \
-u "${USER}" \
-v "$(realpath $ZOOKEEPER_ROOT):${ZOOKEEPER_ROOT}" \
-v "${LOCAL_HOME}:/home/${USER_NAME}" \
${IMAGE_NAME}-${USER_NAME} \
bash -c "${CMD}"

popd

9 changes: 5 additions & 4 deletions pom.xml
Expand Up @@ -278,9 +278,9 @@
<mockito.version>2.27.0</mockito.version>
<hamcrest.version>1.3</hamcrest.version>
<commons-cli.version>1.2</commons-cli.version>
<netty.version>4.1.36.Final</netty.version>
<netty.version>4.1.42.Final</netty.version>
<jetty.version>9.4.18.v20190429</jetty.version>
<jackson.version>2.9.9.3</jackson.version>
<jackson.version>2.9.10</jackson.version>
<json.version>1.1.1</json.version>
<jline.version>2.11</jline.version>
<snappy.version>1.1.7</snappy.version>
Expand Down Expand Up @@ -574,7 +574,7 @@
<plugin>
<groupId>org.owasp</groupId>
<artifactId>dependency-check-maven</artifactId>
<version>5.1.0</version>
<version>5.2.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand All @@ -588,12 +588,13 @@
</dependency>
</dependencies>
<configuration>
<configLocation>checkstyle.xml</configLocation>
<configLocation>checkstyle-strict.xml</configLocation>
<suppressionsLocation>checkstyleSuppressions.xml</suppressionsLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failOnViolation>true</failOnViolation>
<includeResources>false</includeResources>
<includeTestResources>false</includeTestResources>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
</configuration>
<executions>
Expand Down
10 changes: 8 additions & 2 deletions zookeeper-client/zookeeper-client-c/src/zookeeper.c
Expand Up @@ -2716,9 +2716,12 @@ static void deserialize_response(zhandle_t *zh, int type, int xid, int failed, i
cptr->c.string_result(rc, 0, cptr->data);
} else {
struct CreateResponse res;
const char *client_path;
memset(&res, 0, sizeof(res));
deserialize_CreateResponse(ia, "reply", &res);
cptr->c.string_result(rc, res.path, cptr->data);
client_path = sub_string(zh, res.path);
cptr->c.string_result(rc, client_path, cptr->data);
free_duplicate_path(client_path, res.path);
deallocate_CreateResponse(&res);
}
break;
Expand All @@ -2729,8 +2732,11 @@ static void deserialize_response(zhandle_t *zh, int type, int xid, int failed, i
cptr->c.string_stat_result(rc, 0, 0, cptr->data);
} else {
struct Create2Response res;
const char *client_path;
deserialize_Create2Response(ia, "reply", &res);
cptr->c.string_stat_result(rc, res.path, &res.stat, cptr->data);
client_path = sub_string(zh, res.path);
cptr->c.string_stat_result(rc, client_path, &res.stat, cptr->data);
free_duplicate_path(client_path, res.path);
deallocate_Create2Response(&res);
}
break;
Expand Down
10 changes: 10 additions & 0 deletions zookeeper-client/zookeeper-client-c/tests/TestClient.cc
Expand Up @@ -450,6 +450,10 @@ class Zookeeper_simpleSystem : public CPPUNIT_NS::TestFixture

static void create_completion_fn(int rc, const char* value, const void *data) {
CPPUNIT_ASSERT_EQUAL((int) ZOK, rc);
if (data) {
const char *expected_value = (const char *)data;
CPPUNIT_ASSERT_EQUAL(string(expected_value), string(value));
}
count++;
}

Expand Down Expand Up @@ -991,6 +995,12 @@ class Zookeeper_simpleSystem : public CPPUNIT_NS::TestFixture
rc = zoo_create(zk_ch, path, "", 0, &ZOO_OPEN_ACL_UNSAFE, 0, path_buffer, path_buffer_len);
CPPUNIT_ASSERT_EQUAL((int) ZOK, rc);
CPPUNIT_ASSERT_EQUAL(string(path), string(path_buffer));

const char* path2282 = "/zookeeper2282";
rc = zoo_acreate(zk_ch, path2282, "", 0, &ZOO_OPEN_ACL_UNSAFE, 0,
create_completion_fn, path2282);
waitForCreateCompletion(3);
CPPUNIT_ASSERT(count == 0);
}

// Test creating normal handle via zookeeper_init then explicitly setting callback
Expand Down
34 changes: 33 additions & 1 deletion zookeeper-contrib/pom.xml
Expand Up @@ -63,8 +63,40 @@
</compilerArgs>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>3.1.0</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>${checkstyle.version}</version>
</dependency>
</dependencies>
<configuration>
<configLocation>checkstyle-simple.xml</configLocation>
<suppressionsLocation>checkstyleSuppressions.xml</suppressionsLocation>
<encoding>UTF-8</encoding>
<consoleOutput>true</consoleOutput>
<failOnViolation>true</failOnViolation>
<includeResources>false</includeResources>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
<includeTestResources>false</includeTestResources>
</configuration>
<executions>
<execution>
<id>checkstyle</id>
<phase>validate</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>

</project>
</project>
1 change: 1 addition & 0 deletions zookeeper-docs/src/main/resources/markdown/index.md
Expand Up @@ -38,6 +38,7 @@ archives.
Documents for Developers using the ZooKeeper Client API
+ [API Docs](api/index.html) - the technical reference to ZooKeeper Client APIs
+ [Programmer's Guide](zookeeperProgrammers.html) - a client application developer's guide to ZooKeeper
+ [ZooKeeper Use Cases](zookeeperUseCases.html) - a series of use cases using the ZooKeeper.
+ [ZooKeeper Java Example](javaExample.html) - a simple Zookeeper client application, written in Java
+ [Barrier and Queue Tutorial](zookeeperTutorial.html) - sample implementations of barriers and queues
+ [ZooKeeper Recipes](recipes.html) - higher level solutions to common problems in distributed applications
Expand Down
75 changes: 69 additions & 6 deletions zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
Expand Up @@ -124,8 +124,8 @@ is no full support.

#### Required Software

ZooKeeper runs in Java, release 1.8 or greater
(JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported).
ZooKeeper runs in Java, release 1.8 or greater
(JDK 8 LTS, JDK 11 LTS, JDK 12 - Java 9 and 10 are not supported).
It runs as an _ensemble_ of ZooKeeper servers. Three
ZooKeeper servers is the minimum recommended size for an
ensemble, and we also recommend that they run on separate
Expand Down Expand Up @@ -626,6 +626,13 @@ property, when available, is noted below.
reaches a runtime generated random value in the \[snapCount/2+1, snapCount]
range.The default snapCount is 100,000.

* *commitLogCount* * :
(Java system property: **zookeeper.commitLogCount**)
Zookeeper maintains an in-memory list of last committed requests for fast synchronization with
followers when the followers are not too behind. This improves sync performance in case when your
snapshots are large (>100,000).
The default commitLogCount value is 500.

* *snapSizeLimitInKb* :
(Java system property: **zookeeper.snapSizeLimitInKb**)
ZooKeeper records its transactions using snapshots and
Expand Down Expand Up @@ -822,6 +829,27 @@ property, when available, is noted below.
dropping. This parameter defines the threshold to decrease the dropping
probability. The default is 0.

* *zookeeper.connection_throttle_weight_enabled* :
(Java system property only)
**New in 3.6.0:**
Whether to consider connection weights when throttling. Only useful when connection throttle is enabled, that is, connectionMaxTokens is larger than 0. The default is false.

* *zookeeper.connection_throttle_global_session_weight* :
(Java system property only)
**New in 3.6.0:**
The weight of a global session. It is the number of tokens required for a global session request to get through the connection throttler. It has to be a positive integer no smaller than the weight of a local session. The default is 3.

* *zookeeper.connection_throttle_local_session_weight* :
(Java system property only)
**New in 3.6.0:**
The weight of a local session. It is the number of tokens required for a local session request to get through the connection throttler. It has to be a positive integer no larger than the weight of a global session or a renew session. The default is 1.

* *zookeeper.connection_throttle_renew_session_weight* :
(Java system property only)
**New in 3.6.0:**
The weight of renewing a session. It is also the number of tokens required for a reconnect request to get through the throttler. It has to be a positive integer no smaller than the weight of a local session. The default is 2.


* *clientPortListenBacklog* :
**New in 3.4.14, 3.5.5, 3.6.0:**
The socket backlog length for the ZooKeeper server socket. This controls
Expand Down Expand Up @@ -889,7 +917,7 @@ property, when available, is noted below.

* *advancedFlowControlEnabled* :
(Java system property: **zookeeper.netty.advancedFlowControl.enabled**)
Using accurate flow control in netty based on the status of ZooKeeper
Using accurate flow control in netty based on the status of ZooKeeper
pipeline to avoid direct buffer OOM. It will disable the AUTO_READ in
Netty.

Expand All @@ -915,6 +943,30 @@ property, when available, is noted below.
ZooKeeper when loading database from disk, and syncing with leader.
By default, this feautre is disabled, set "true" to enable it.

* *snapshot.trust.empty* :
(Java system property only: **zookeeper.snapshot.trust.empty**)
**New in 3.5.6:**
This property controls whether or not ZooKeeper should treat missing
snapshot files as a fatal state that can't be recovered from.
Set to true to allow ZooKeeper servers recover without snapshot
files. This should only be set during upgrading from old versions of
ZooKeeper (3.4.x, pre 3.5.3) where ZooKeeper might only have transaction
log files but without presence of snapshot files. If the value is set
during upgrade, we recommend to set the value back to false after upgrading
and restart ZooKeeper process so ZooKeeper can continue normal data
consistency check during recovery process.
Default value is false.

* *largeRequestMaxBytes* :
(Java system property: **zookeeper.largeRequestMaxBytes**)
**New in 3.6.0:**
The maximum number of bytes of all inflight large request. The connection will be closed if a coming large request causes the limit exceeded. The default is 100 * 1024 * 1024.

* *largeRequestThreshold* :
(Java system property: **zookeeper.largeRequestThreshold**)
**New in 3.6.0:**
The size threshold after which a request is considered a large request. If it is -1, then all requests are considered small, effectively turning off large request throttling. The default is -1.

<a name="sc_clusterOptions"></a>

#### Cluster Options
Expand Down Expand Up @@ -944,9 +996,9 @@ of servers -- that is, when deploying clusters of servers.
* *connectToLearnerMasterLimit* :
(Java system property: zookeeper.**connectToLearnerMasterLimit**)
Amount of time, in ticks (see [tickTime](#id_tickTime)), to allow followers to
connect to the leader after leader election. Defaults to the value of initLimit.
connect to the leader after leader election. Defaults to the value of initLimit.
Use when initLimit is high so connecting to learner master doesn't result in higher timeout.

* *leaderServes* :
(Java system property: zookeeper.**leaderServes**)
Leader accepts client connections. Default value is "yes".
Expand Down Expand Up @@ -1335,6 +1387,17 @@ the variable does.
problems will arise. This is really a sanity check. ZooKeeper is
designed to store data on the order of kilobytes in size.

* *jute.maxbuffer.extrasize*:
(Java system property: **zookeeper.jute.maxbuffer.extrasize**)
**New in 3.5.7:**
While processing client requests ZooKeeper server adds some additional information into
the requests before persisting it as a transaction. Earlier this additional information size
was fixed to 1024 bytes. For many scenarios, specially scenarios where jute.maxbuffer value
is more than 1 MB and request type is multi, this fixed size was insufficient.
To handle all the scenarios additional information size is increased from 1024 byte
to same as jute.maxbuffer size and also it is made configurable through jute.maxbuffer.extrasize.
Generally this property is not required to be configured as default value is the most optimal value.

* *skipACL* :
(Java system property: **zookeeper.skipACL**)
Skips ACL checks. This results in a boost in throughput,
Expand Down Expand Up @@ -1554,7 +1617,7 @@ options are used to configure the [AdminServer](#sc_adminserver).

### Metrics Providers

**New in 3.6.0:** The following options are used to configure metrics.
**New in 3.6.0:** The following options are used to configure metrics.

By default ZooKeeper server exposes useful metrics using the [AdminServer](#sc_adminserver).
and [Four Letter Words](#sc_4lw) interface.
Expand Down

0 comments on commit 05eae83

Please sign in to comment.