KYLIN-4775 Use docker-compose to deploy Hadoop and Kylin

[liuyonghengheng]

Proposed changes

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue.

Types of changes

What types of changes does your code introduce to Kylin?
Put an x in the boxes that apply

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

• I have create an issue on Kylin's jira, and have described the bug/feature there in detail
• Commit messages in my PR start with the related jira ID, like "KYLIN-0000 Make Kylin project open-source"
• Compiling and unit tests pass locally with my changes
• I have added tests that prove my fix is effective or that my feature works
• If this change need a document change, I will prepare another pr against the document branch
• Any dependent changes have been merged

Further comments

If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc...

[coveralls]

Pull Request Test Coverage Report for Build 6441

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 455 unchanged lines in 19 files lost coverage.
• Overall coverage decreased (-0.005%) to 28.051%

---

Files with Coverage Reduction	New Missed Lines	%
core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java	1	60.58%
server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java	1	0%
tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java	1	81.58%
core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java	2	68.46%
core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java	3	0%
core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java	3	78.42%
server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java	7	0%
server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java	9	0%
server-base/src/main/java/org/apache/kylin/rest/controller/ModelController.java	10	0%
core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java	11	81.4%

Totals
Change from base Build 6388:	-0.005%
Covered Lines:	26255
Relevant Lines:	93598

---

💛 - Coveralls

[liuyonghengheng]

如何启动一个集群

进入docker 目录：

cd apachekylin/kylin/docker

执行如下命令可以 构建docker镜像，并启动docker-compose

sh setup_cluster.sh --cluster_mode write-read --hadoop_version 2.8.5 --hive_version 1.2.2 --enable_hbase yes --hbase_version 1.1.2  --enable_ldap no  

1. --cluster_mode 参数值可选项为[write,write-read]默认参数是 write，也就是会启动一个写集群，不做读写分离
   在write 模式下会启动一个 hadoop集群包含hive，hbase，等。在write-read模式下会启动两个hadoop集群，一个作为计算集群，一个作为查询集群，并且hbase装在查询集群中
2. --hadoop_version 可以指定hadoop的版本，默认是 2.8.5
   hadoop 集群包含 1个 namenode节点和3个datanode节点，以及1个resourcemanager和2个nodemanager，以及一个jobhistoryserver。
3. --hive_version 可以指定hive版本，默认是 1.2.2
   hive 包含一个 hivemetastore 和一个 hiveserver2 节点
4. --hbase_version 可以指定hbase版本，默认是 1.1.2
5. --enable_hbase 是否启动habase服务，可选参数为[yes,no], 默认参数是yes
   hbase 包含一个 hmaster 和两个 regionserver
6. --enable_ldap 是否启动ldap服务，可选参数为[yes,no], 默认参数是no
7. --enable_kafka 是否启动kafka broker，可选参数为[yes,no], 默认参数是no
8. --enable_kerberos 是否启动 kerberos 服务，可选参数为[yes,no], 默认参数是no。(目前尚为开发完成)

另外会启动一个kylin集群，包含一个all节点一个job节点和一个query节点。三个节点的kylin目录与 docker-compose/others/kylin/ 一一对应
其中。一般将打包好的kylin jar包解压到 对应的路径下，并修改相应的配置文件，即可实现部署和修改kylin配置的目的。

如果是第一次执行以上命令，那么会首先去构建和拉取相应的docker镜像，所以执行时间会很长，一般会在半个小时左右才能完成。

脚本执行完成后可以执行如下命令查询集群状态：

docker ps -a

等待所有的container都达到health状态即可执行kylin的测试工作。

关闭集群：

sh stop_cluster.sh --cluster_mode write-read --hadoop_version 2.8.5 --hive_version 1.2.2 --enable_hbase yes --hbase_version 1.1.2

注意事项：

如果想要单独启动或者关闭 docker-compose 而不是使用setup/stop脚本
那么在启动之前请先构建想需要的image，并设置 image相关的环境变量

export HBASE_VERSION=1.1.2
export HADOOP_VERSION=2.8.4
export HIVE_VERSION=1.2.2

export HADOOP_NAMENODE_IMAGETAG=apachekylin/kylin-hadoop-base:hadoop_${HADOOP_VERSION}
export HADOOP_DATANODE_IMAGETAG=apachekylin/kylin-hadoop-datanode:hadoop_${HADOOP_VERSION}
export HADOOP_NAMENODE_IMAGETAG=apachekylin/kylin-hadoop-namenode:hadoop_${HADOOP_VERSION}
export HADOOP_RESOURCEMANAGER_IMAGETAG=apachekylin/kylin-hadoop-resourcemanager:hadoop_${HADOOP_VERSION}
export HADOOP_NODEMANAGER_IMAGETAG=apachekylin/kylin-hadoop-nodemanager:hadoop_${HADOOP_VERSION}
export HADOOP_HISTORYSERVER_IMAGETAG=apachekylin/kylin-hadoop-historyserver:hadoop_${HADOOP_VERSION}
export HIVE_IMAGETAG=apachekylin/kylin-hive:hive_${HIVE_VERSION}_hadoop_${HADOOP_VERSION}

export HBASE_MASTER_IMAGETAG=apachekylin/kylin-hbase-base:hbase_${HBASE_VERSION}
export HBASE_MASTER_IMAGETAG=apachekylin/kylin-hbase-master:hbase_${HBASE_VERSION}
export HBASE_REGIONSERVER_IMAGETAG=apachekylin/kylin-hbase-regionserver:hbase_${HBASE_VERSION}

export KAFKA_IMAGE=bitnami/kafka:2.0.0
export LDAP_IMAGE=osixia/openldap:1.3.0
export CLIENT_IMAGETAG=apachekylin/kylin-client:hadoop_${HADOOP_VERSION}_hive_${HIVE_VERSION}_hbase_${HBASE_VERSION}

因为 docker-compose 文件中的 image 都是使用环境变量的方式，所以如果想要设置不同版本 image
可以通过设置环境变量的方式来启动不同版本的 container。

[hit-lacus]

Component Support Matrix

Hadoop Version	Hive Version	HBase Version	Verified
2.8.5	1.2.2	1.1.2	Yes
2.8	2.2	2.2	Checking
3.1.4	2.3.7	2.2.6	Checking
3.1.4	3.1.x	2.2.6	Checking

hadoop_3.1.4_hive_2.3.7_hbase_2.2.6

[hit-lacus]

Maybe "ENABLE_KERBEROS" instead of "KERBEROS" ?

[hit-lacus]

Do you mean hadoop_2.8.5 instead of hadoop_2.8. ?

[hit-lacus]

@liuyonghengheng Really appreacite your contribution! @zhangayqian will add your patchs in her PR( #1465) with some modification, so I will close this one .