Skip to content

Commit

Permalink
[INLONG-1934] update the image of the hive example after the bid chan…
Browse files Browse the repository at this point in the history
…ged (apache#220)

Co-authored-by: dockerzhang <dockerzhang@tencent.com>
  • Loading branch information
dockerzhang and dockerzhang committed Dec 8, 2021
1 parent 410e152 commit 8ec2130
Show file tree
Hide file tree
Showing 25 changed files with 145 additions and 145 deletions.
17 changes: 15 additions & 2 deletions docs/deployment/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,35 @@ sidebar_position: 2
Deploy all InLong module by Docker Compose, it's only available for development.

Requirements:

- [Docker](https://docs.docker.com/engine/install/) 19.03.1+
- Docker Compose 1.29.2+

## Deploy

Manually copy SQL files from `inlong-manager/manager-web/sql` to the `docker/compose/sql` directory.

```shell
cp inlong-manager/manager-web/sql/apache_inlong_manager.sql docker/docker-compose/sql
```

Then, start all components.

```shell
docker-compose up -d
```

## Use InLong

After all containers run successfully, you can access `http://localhost` with default account:
```

```shell
User: admin
Password: inlong
```

## Destroy
```

```shell
docker-compose down
```
2 changes: 1 addition & 1 deletion docs/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Apache InLong uses TDBank internally used by Tencent as the prototype, and reli
## Modules
Apache InLong serves the entire life cycle from data collection to landing, and provides different processing modules according to different stages of data, including the next modules:

- **inlong-agent**, data collection agent, supports reading regular logs from specified directories or files and reporting data one by one. In the future, DB collection and HTTP reporting capabilities will also be expanded.
- **inlong-agent**, data collection agent, supports reading regular logs from specified directories or files and reporting data one by one. In the future, DB collection capabilities will also be expanded.
- **inlong-dataproxy**, a Proxy component based on Flume-ng, supports data transmission blocking, placing retransmission, and has the ability to forward received data to different MQ (message queues).
- **inlong-tubemq**, Tencent's self-developed message queuing service, focuses on high-performance storage and transmission of massive data in big data scenarios and has a relatively good core advantage in mass practice and low cost.
- **inlong-sort**, after consuming data from different MQ services, perform ETL processing, and then aggregate and write the data into Apache Hive, ClickHouse, Hbase, IceBerg, etc.
Expand Down
48 changes: 13 additions & 35 deletions docs/modules/manager/quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,38 +2,22 @@
title: Deployment
---

## 1 Environmental preparation
- Install and start MySQL 5.7+, copy the `doc/sql/apache_inlong_manager.sql` file in the inlong-manager module to the
server where the MySQL database is located (for example, copy to `/data/` directory), load this file through the
following command to complete the initialization of the table structure and basic data:
## Environmental preparation
- Install and start MySQL 5.7+
- initialize database
there is `sql/apache_inlong_manager.sql` in `inlong-manager-web` directory, load this file through the
following command to complete the initialization of the table structure and basic data

```shell
# Log in to the MySQL server by username and password:
mysql -u xxx -p xxx
...
# Create database
CREATE DATABASE IF NOT EXISTS apache_inlong_manager;
USE apache_inlong_manager;
# Load the above SQL file through the source command:
mysql> source /data/apache_inlong_manager.sql;
# initialize database:
mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_manager.sql
```

- Refer to [Compile and deploy TubeMQ](https://inlong.apache.org/zh-cn/docs/modules/tubemq/quick_start.html) to install
and start the Tube cluster;

- Refer
to [Compile and deploy TubeMQ Manager](https://inlong.apache.org/zh-cn/docs/modules/tubemq/tubemq-manager/quick_start.html)
, install and start TubeManager.

## 2 Deploy and start manager-web
## Deploy and start manager-web

**manager-web is a background service that interacts with the front-end page.**

### 2.1 Prepare installation files

All installation files at `inlong-manager-web` directory.

### 2.2 Modify configuration
### Modify configuration

Go to the decompressed `inlong-manager-web` directory and modify the `conf/application.properties` file:

Expand All @@ -51,8 +35,8 @@ The dev configuration is specified above, then modify the `conf/application-dev.

```properties
spring.datasource.jdbc-url=jdbc:mysql://127.0.0.1:3306/apache_inlong_manager?useSSL=false&allowPublicKeyRetrieval=true&characterEncoding=UTF-8&nullCatalogMeansCurrent=true&serverTimezone=GMT%2b8
spring.datasource.username=xxxxxx
spring.datasource.password=xxxxxx
spring.datasource.username=DB_USER
spring.datasource.password=DB_PASSWD
```

2) Modify the connection information of the Tube and ZooKeeper clusters, among which `cluster.zk.root` suggests using
Expand All @@ -74,17 +58,11 @@ The dev configuration is specified above, then modify the `conf/application-dev.
sort.appName=inlong_app
```

### 2.3 Start the service
### Start the service

Enter the decompressed directory, execute `sh bin/startup.sh` to start the service, and check the
log `tailf log/manager-web.log`. If a log similar to the following appears, the service has started successfully:

```shell
Started InLongWebApplication in 6.795 seconds (JVM running for 7.565)
```

## 3 Service access verification

Verify the manager-web service:

Visit address: <http://[manager_web_ip]:[manager_web_port]/api/inlong/manager/doc.html#/home>
```
46 changes: 46 additions & 0 deletions docs/modules/website/overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
title: Overview
sidebar_position: 1
---

This is a website console for us to use the [Apache InLong](https://github.com/apache/incubator-inlong).

## Guide For Developer
You should check that `nodejs >= 12.0` is installed.

In the project, you can run some built-in commands:

If `node_modules` is not installed, you should first run `npm install` or `yarn install`.

Use `npm run dev` or `yarn dev` to run the application in development mode.

If the server runs successfully, the browser will open [http://localhost:8080](http://localhost:8080) to view in the browser.

If you edit, the page will reload.
You will also see any lint errors in the console.

The start of the web server depends on the back-end server `manger api` interface.

You should start the backend server first, and then set the variable `target` in `/inlong-website/src/setupProxy.js` to the address of the api service.

### Test

Run `npm test` or `yarn test`

Start the test runner in interactive observation mode.
For more information, see the section on [Running Tests](https://create-react-app.dev/docs/running-tests/).

### Build

First, make sure that the project has run `npm install` or `yarn install` to install `node_modules`.

Run `npm run build` or `yarn build`.

Build the application for production into the build folder.
Better page performance can be obtained in the constructed production mode.

After the build, the code is compressed, and the file name includes the hash value.
Your application is ready to be deployed!

For details, see the section on [deployment](https://create-react-app.dev/docs/deployment/).

55 changes: 10 additions & 45 deletions docs/modules/website/quick_start.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,20 @@
---
title: Deployment
sidebar_position: 2
---

## 1 About WebSite
This is a website console for us to use the [Apache InLong incubator](https://github.com/apache/incubator-inlong).

## 2 Build
## Prepare Docker Image
pull image from central hub:
```
docker pull inlong/website:latest
```
or build image from source:
```
mvn package -DskipTests -Pdocker -pl inlong-website
```

## 3 Run
## Run
```
# MANAGER_API_ADDRESS must be replaced by inlong-manager-web address
docker run -d --name website -e MANAGER_API_ADDRESS=127.0.0.1:8083 -p 80:80 inlong/website
```

## 4 Guide For Developer
You should check that `nodejs >= 12.0` is installed.

In the project, you can run some built-in commands:

If `node_modules` is not installed, you should first run `npm install` or `yarn install`.

Use `npm run dev` or `yarn dev` to run the application in development mode.

If the server runs successfully, the browser will open [http://localhost:8080](http://localhost:8080) to view in the browser.

If you edit, the page will reload.
You will also see any lint errors in the console.

The start of the web server depends on the back-end server `manger api` interface.

You should start the backend server first, and then set the variable `target` in `/inlong-website/src/setupProxy.js` to the address of the api service.

### 4.1 Test

Run `npm test` or `yarn test`

Start the test runner in interactive observation mode.
For more information, see the section on [Running Tests](https://create-react-app.dev/docs/running-tests/).

### 4.2 Build

First, make sure that the project has run `npm install` or `yarn install` to install `node_modules`.

Run `npm run build` or `yarn build`.

Build the application for production into the build folder.
Better page performance can be obtained in the constructed production mode.

After the build, the code is compressed, and the file name includes the hash value.
Your application is ready to be deployed!

For details, see the section on [deployment](https://create-react-app.dev/docs/deployment/).
```
10 changes: 5 additions & 5 deletions docs/quick_start/hive_example.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,23 +16,23 @@ Before we begin, we need to install InLong. Here we provide two ways:
2. Install InLong binary according to the [instructions here](deployment/bare_metal.md).

## 3 Create a data access
After deployment, we first enter the "Data Access" interface, click "Create an Access" in the upper right corner to create a new date access, and fill in the business information as shown in the figure below.
After deployment, we first enter the "Data Access" interface, click "Create an Access" in the upper right corner to create a new date access, and fill in the data streams group information as shown in the figure below.

<img src="/img/create-business.png" align="center" alt="Create Business"/>
<img src="img/create-group.png" align="center" alt="Create Group"/>

Then we click the next button, and fill in the stream information as shown in the figure below.

<img src="/img/create-stream.png" align="center" alt="Create Stream"/>
<img src="img/create-stream.png" align="center" alt="Create Stream"/>

Note that the message source is "File", and we don't need to create a message source manually.

Then we fill in the following information in the "data information" column below.

<img src="/img/data-information.png" align="center" alt="Data Information"/>
<img src="img/data-information.png" align="center" alt="Data Information"/>

Then we select Hive in the data flow and click "Add" to add Hive configuration

<img src="/img/hive-config.png" align="center" alt="Hive Config"/>
<img src="img/hive-config.png" align="center" alt="Hive Config"/>

Note that the target table does not need to be created in advance, as InLong Manager will automatically create the table for us after the access is approved. Also, please use connection test to ensure that InLong Manager can connect to your Hive.

Expand Down
Binary file added docs/quick_start/img/create-group.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/quick_start/img/create-stream.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/quick_start/img/data-information.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/quick_start/img/hive-config.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,13 @@ sidebar_position: 2
- Docker Compose 1.29.2+

## 部署

手动拷贝`inlong-manager/manager-web/sql`目录到`docker/compose/sql`目录.
```shell
cp inlong-manager/manager-web/sql/apache_inlong_manager.sql docker/docker-compose/sql
```

然后启动所有组件:
```
docker-compose up -d
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Apache InLong 以腾讯内部使用的 TDBank 为原型,依托万亿级别的
## 模块
Apache InLong 服务于数据采集到落地的整个生命周期,按数据的不同阶段提供不同的处理模块,主要包括:

- **inlong-agent**,数据采集 Agent,支持从指定目录或文件读取常规日志、逐条上报。后续也将扩展 DB 采集、HTTP 上报等能力
- **inlong-agent**,数据采集 Agent,支持从指定目录或文件读取常规日志、逐条上报。后续也将扩展 DB 采集等能力
- **inlong-dataproxy**,一个基于 Flume-ng 的 Proxy 组件,支持数据发送阻塞和落盘重发,拥有将接收到的数据转发到不同 MQ(消息队列)的能力。
- **inlong-tubemq**,腾讯自研的消息队列服务,专注于大数据场景下海量数据的高性能存储和传输,在海量实践和低成本方面有着良好的核心优势。
- **inlong-sort**,对从不同的 MQ 消费到的数据进行 ETL 处理,然后汇聚并写入 Hive、ClickHouse、Hbase、Iceberg 等存储系统。
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ Agent指标分为以下几项, 各项的属性分别为:
### PluginMetric
| 属性名称 | 说明 |
| ---- | ---- |
| readNum | 当前正在运行的job总数 |
| sendNum | 当前失败的job总数 |
| readNum | 读取的条数 |
| sendNum | 发送的条数 |
| sendFailedNum | 发送失败条数 |
| readFailedNum | 读取失败条数 |
| readSuccessNum | 读取成功条数 |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,35 +2,21 @@
title: 安装部署
---

## 1 环境准备
- 安装并启动 MySQL 5.7+,把 inlong-manager 模块中的 `doc/sql/apache_inlong_manager.sql` 文件拷贝到 MySQL 数据库所在的服务器
(比如拷贝到 `/data/` 目录下),通过下述命令加载此文件,完成表结构及基础数据的初始化:
## 环境准备
- 安装并启动 MySQL 5.7+
- 初始化数据库
`inlong-manager-web` 目录下有 `sql/apache_inlong_manager.sql`文件,通过下述命令加载此文件,完成表结构及基础数据的初始化:

```shell
# 通过用户名和密码,登录 MySQL 服务器:
mysql -u xxx -p xxx
...
# 创建数据库
CREATE DATABASE IF NOT EXISTS apache_inlong_manager;
USE apache_inlong_manager;
# 通过 source 命令加载上述 SQL 文件:
mysql> source /data/apache_inlong_manager.sql;
# 通过用户名和密码,创建DB和表:
mysql -uDB_USER -pDB_PASSWD < sql/apache_inlong_manager.sql
```

- 参照 [安装部署TubeMQ](https://inlong.apache.org/zh-cn/docs/modules/tubemq/quick_start.html),安装并启动 Tube 集群;

- 参照 [安装部署TubeMQ Manager](https://inlong.apache.org/zh-cn/docs/modules/tubemq/tubemq-manager/quick_start.html),安装并启动
TubeManager。

## 2 部署、启动 manager-web
## 部署、启动 manager-web

**manager-web 是与前端页面交互的后台服务。**

### 2.1 准备安装文件

安装文件在 `inlong-manager-web` 目录下。

### 2.2 修改配置
### 修改配置

前往 `inlong-manager-web` 目录,修改 `conf/application.properties` 文件:

Expand All @@ -48,8 +34,8 @@ spring.profiles.active=dev

```properties
spring.datasource.jdbc-url=jdbc:mysql://127.0.0.1:3306/apache_inlong_manager?useSSL=false&allowPublicKeyRetrieval=true&characterEncoding=UTF-8&nullCatalogMeansCurrent=true&serverTimezone=GMT%2b8
spring.datasource.username=xxxxxx
spring.datasource.password=xxxxxx
spring.datasource.username=DB_USER
spring.datasource.password=DB_PASSWD
```

2) 修改 Tube 和 ZooKeeper 集群的连接信息,其中 `cluster.zk.root` 建议使用默认值:
Expand All @@ -70,16 +56,10 @@ spring.profiles.active=dev
sort.appName=inlong_app
```

### 2.3 启动服务
### 启动服务

进入解压后的目录,执行 `sh bin/startup.sh` 启动服务,查看日志 `tailf log/manager-web.log`,若出现类似下面的日志,说明服务启动成功:

```shell
Started InLongWebApplication in 6.795 seconds (JVM running for 7.565)
```

## 3 服务访问验证

在浏览器中访问如下地址,验证 manager-web 服务:

地址:<http://[manager_web_ip]:[manager_web_port]/api/inlong/manager/doc.html#/home>
```
Loading

0 comments on commit 8ec2130

Please sign in to comment.