ISSUE-2640: BP-43: Run integration tests with current source code.

[pkumar-singh]

Motivation

Migrate bookkeeper to gradle.

• Build BKCTL and generate artifacts
• Build docker image from the jar created from existing source and run integration tests.

Changes

• A Dockerfile which will build docker image from existing source.
• A Shell file which will generate all the artifacts for building Docker image
• All build.gradle files to build and run integration tests.

Master Issue: #2640

How to build

./gradlew :tests:integration:cluster:test

` Task :tests:integration:cluster:buildImage
#1 [internal] load build definition from Dockerfile
#1 sha256:13c696b34811c50ef1582dd1c27e59e8cf726660222ada83feb0b294a4197b14
#1 transferring dockerfile: 37B 0.0s done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:028ece09d81436333eece58b183b9dc0a52cbc28bf2d07cfad5f3080e03d38b7
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/centos:7
#3 sha256:30875b35a89c8e8a29cd7cf120689bb68cdab8d769419707e07138dfe977d237
#3 DONE 9.7s

#4 [ 1/16] FROM docker.io/library/centos:7@sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e
#4 sha256:a82657dc9009ab4ecab0a966d899b2972800cfe1ee9d42fc736a0cac1581ac52
#4 DONE 0.0s

#11 [internal] load build context
#11 sha256:2ba06714f2ca9169cd544ca120d73724654078b8159628baf9e4af328b2d617a
#11 transferring context: 23B
#11 transferring context: 122.54MB 4.5s done
#11 DONE 4.5s

#13 [ 9/16] RUN mv /opt/server/lib/*.jar /opt/bookkeeper/lib/
#13 sha256:0edb1ad87b6c9035382a0500b77b4577388841c8eef32516507007f857f058a8
#13 CACHED

#18 [14/16] COPY ./conf/* /opt/bookkeeper/conf/
#18 sha256:53a4f93597253dff0934c46de43bac4f857ced79886d97523b3f5c190624aeb6
#18 CACHED

#19 [15/16] RUN chmod +x -R /opt/bookkeeper/scripts
#19 sha256:6be237cc3422cc33cb527d6ac9dac6132f82d474e1f0a28a4f50784454319dee
#19 CACHED

#7 [ 4/16] RUN mkdir /opt/bookkeeper/lib
#7 sha256:3fe6bccdc94f36263533c94142448099fa427415412b40e6c2d7003f0f9172f9
#7 CACHED

#17 [13/16] COPY ./temp_conf/* /opt/bookkeeper/conf/
#17 sha256:da439db0c1a32b3a7a60d1ffabc8fe4c4d23f1c4a7bd6095fc5d4a4530691f9d
#17 CACHED

#6 [ 3/16] WORKDIR /opt/bookkeeper
#6 sha256:75cb16fec34eda0a19b049218df4fd332274fb1af59e20dea96ce8bda3cac37f
#6 CACHED

#16 [12/16] COPY ./bin/* /opt/bookkeeper/bin/
#16 sha256:fe3bceec3581b5e8183cf2804a8e8c0ef06c5eb11b498260bd6fa5a3df7ec139
#16 CACHED

#8 [ 5/16] RUN mkdir /opt/bookkeeper/bin
#8 sha256:1efcf781a6c50daf837d1ed53d859145b6d07b9050cfa5ed2cab6a370d64c0fb
#8 CACHED

#15 [11/16] COPY ./temp_bin/* /opt/bookkeeper/bin/
#15 sha256:7e7fb09841111dd8b7444c6007be8c945bf0b84d698d81cf2e7cfda7df01bff0
#15 CACHED

#5 [ 2/16] RUN set -x && adduser "bookkeeper" && yum install -y java-1.8.0-openjdk-headless wget bash sudo && wget -q https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py && pip install zk-shell && rm -rf get-pip.py && mkdir -pv /opt && cd /opt && yum clean all
#5 sha256:055562ef0c2a679a10597b7a9c15d731dc1235a63e74c44e0ec246d059b2cdbd
#5 CACHED

#10 [ 7/16] RUN mkdir /opt/bookkeeper/scripts
#10 sha256:8ed4308f6f3b2710fc497036cc4d75df7e0c8d335b9f4ca3962f2fbac5ba55f5
#10 CACHED

#14 [10/16] COPY ./scripts/* /opt/bookkeeper/scripts/
#14 sha256:f78d90c1c59a8d4bf347fc9109ea28ad5a811ab605fc688a1e60336daf6e1564
#14 CACHED

#12 [ 8/16] ADD ./dist/server.tar /opt/
#12 sha256:1f8a62db7f0bcabf8bbc7b9293c362a4c5b91a90801fab595e264a106529ac45
#12 CACHED

#9 [ 6/16] RUN mkdir /opt/bookkeeper/conf
#9 sha256:271a4ee43b22ab37ab9ba5e511fc9a477950119c63da39d5d399bf72f4c7ab08
#9 CACHED

#20 [16/16] RUN chmod +x -R /opt/bookkeeper/bin
#20 sha256:65e54de91545e3a96a707a741727068de55978138d7ead28000b789a7dc4818f
#20 CACHED

#21 exporting to image
#21 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#21 exporting layers done
#21 writing image sha256:3c816ab89d94c687e47c91df2ccd85a20f481679bb76540848d137e60adbcbb0 done
#21 naming to docker.io/apachebookkeeper/bookkeeper-current:latest done
#21 DONE 0.0s

Task :tests:integration:cluster:test

org.apache.bookkeeper.tests.integration.cluster.SimpleClusterTest

Test test000_ClusterIsEmpty PASSED
Test test001_StartBookie PASSED (5.9s)
Test test002_StopBookie PASSED (10.5s)

org.apache.bookkeeper.tests.integration.stream.LocationClientTest

Test testLocateStorageContainers PASSED

org.apache.bookkeeper.tests.integration.stream.BkCtlTest

Test simpleTest PASSED (2.9s)
Test putGetKey PASSED (7.9s)
Test createTable PASSED (2.5s)
Test createNamespace PASSED (2.4s)
Test listBookies PASSED (1.3s)
Test incGetKey PASSED (6.9s)
Test showLastMark PASSED (1s)

org.apache.bookkeeper.tests.integration.stream.StorageAdminClientTest

Test testStreamAPIClientSideRouting PASSED
Test testNamespaceAPIClientSideRouting PASSED
Test testStreamAPIServerSideRouting PASSED
Test testNamespaceAPIServerSideRouting PASSED

org.apache.bookkeeper.tests.integration.stream.TableClientTest

Test testTableAPIClientSideRouting PASSED (3.9s)
Test testTableAPIServerSideRouting PASSED (4s)

SUCCESS: Executed 17 tests in 2m 47s

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 3m 6s
`

[sursingh]

Can we keep this list sorted. That will make it easier to resolve conflicts

[pkumar-singh]

Yeah.

[merlimat]

the old name was "codahale" with "A", but it was anyway renamed to dropwizard

[pkumar-singh]

Its just a variable I will rename it to dropwizard.

[merlimat]

Are these different from the scripts that are already in top level ./bin directory?

[pkumar-singh]

Yeah. It's stripped down version of that.
IN the bin it builds the classpath like the following.
`CLI_MODULE_NAME="(org.apache.bookkeeper-)?bookkeeper-tools"
CLI_MODULE_HOME=${BK_HOME}/${CLI_MODULE_PATH}

find the module jar
CLI_JAR=$(find_module_jar ${CLI_MODULE_PATH} ${CLI_MODULE_NAME})

set up the classpath
CLI_CLASSPATH=$(set_module_classpath ${CLI_MODULE_PATH})`

While in the docker image we know where the jars are put so we do not have to search for module jar or any jar for that matter.

Therefore we can directly use the classpath.
CLI_CLASSPATH=${BK_HOME}/lib/*

[pkumar-singh]

Same things for conf files.

[merlimat]

Conf files I think are ok, but the bin scripts it would be nice to have tests to use the same one (so that we can actually test them). Do you see a way to support the 2 different cases?

[pkumar-singh]

I understand that. It can be done If we name the jars accordingly.