Skip to content

Latest commit

 

History

History
267 lines (196 loc) · 9.61 KB

20240201_03.md

File metadata and controls

267 lines (196 loc) · 9.61 KB
Raw

在宇宙最强数据库镜像中部署databend

作者

digoal

日期

2024-02-01

标签

PostgreSQL , PolarDB , DuckDB , 大数据 , 存储 , 计算 , 存储格式 , 优化器 , 协议 , 分离 , databend , rust , datafusion , snowflake

背景

在宇宙最强数据库镜像中部署databend

本地化部署, 快速测试体验databend

使用如下任意镜像的容器

进入容器后操作

1、创建databend用户和组

groupadd -g 3001 databend    
useradd -u 3001 -g databend -m -d /home/databend -s /bin/bash databend

2、创建databend目录, 修改权限

mkdir -p /var/log/databend  
mkdir -p /var/lib/databend/raft  
  
chown -R databend:databend /var/log/databend  
chown -R databend:databend /var/lib/databend

3、在databend用户下安装rust

(可选)如果你需要使用cargo编译bendsql客户端, 则需要安装rust

su - databend -c "curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 --proto '=https' --tlsv1.2 -sSf -L https://sh.rustup.rs | sh -s -- -y"    
su - databend -c "source /home/databend/.cargo/env"   
  
echo "[source.crates-io]" >> /home/databend/.cargo/config  
echo "replace-with = 'ustc'" >> /home/databend/.cargo/config            
echo "    "  >> /home/databend/.cargo/config  
echo "[source.ustc]"  >> /home/databend/.cargo/config            
echo "registry = \"sparse+https://mirrors.ustc.edu.cn/crates.io-index/\"" >> /home/databend/.cargo/config

4、配置databend用户环境变量

echo "#  add by digoal" >> /home/databend/.bashrc      
echo "alias rm='rm -i'" >> /home/databend/.bashrc      
echo "alias cp='cp -i'" >> /home/databend/.bashrc      
echo "alias ll='ls -larth'" >> /home/databend/.bashrc      
echo "alias mv='mv -i'" >> /home/databend/.bashrc

(可选)如果你需要使用cargo编译bendsql客户端, 则需要配置cargo环境变量

echo ". /home/databend/.cargo/env" >> /home/databend/.bashrc

5、下载并解压databend软件 , https://docs.databend.com/download

cd /tmp   
# tar -zxvf databend-v1.2.307-aarch64-unknown-linux-gnu.tar.gz -C /home/databend/    
mkdir -p /home/databend/databend_software    
tar -zxvf databend-aarch64.tar.gz -C /home/databend/databend_software

6、配置 databend

cd /home/databend/databend_software   
echo "  " >> ./configs/databend-query.toml  
echo "[[query.users]]" >> ./configs/databend-query.toml  
echo "name = \"root\"" >> ./configs/databend-query.toml  
echo "auth_type = \"no_password\"" >> ./configs/databend-query.toml  
  
chown -R databend:databend /home/databend/databend_software  
  
cd /tmp

7、(可选)使用cargo安装bendsql客户端 , https://docs.databend.com/guides/sql-clients/bendsql/

su - databend -c "cargo install bendsql"

7.1、下载编译好的bendsql客户端 , https://docs.databend.com/download

tar -zxvf bendsql-x86_64-unknown-linux-gnu.tar.gz -C /usr/local/bin/

or

tar -zxvf bendsql-aarch64-unknown-linux-gnu.tar.gz -C /usr/local/bin/

8、启动 databend

su - databend  
cd /home/databend/databend_software   
./scripts/start.sh

输出举例

Wait for 180s for active leader...  
Leader Id: 1  
    Metrics: id=1, Leader, term=1, last_log=Some(195), last_applied=Some(1-1-195), membership={log_id:Some(1-1-3), voters:[{1:{EmptyNode}}], learners:[]}  
  
Register this node: {id=1 raft=localhost:28103 grpc=127.0.0.1:9191}  
  
    Register-node: Ok  
  
Databend Metasrv started  
  
==> query.log <==  
    listened at 0.0.0.0:3307  
    connect via: mysql -u${USER} -p${PASSWORD} -h0.0.0.0 -P3307  
Clickhouse(http)  
    listened at 0.0.0.0:8124  
    usage:  echo 'create table test(foo string)' | curl -u${USER} -p${PASSWORD}: '0.0.0.0:8124' --data-binary  @-  
echo '{"foo": "bar"}' | curl -u${USER} -p${PASSWORD}: '0.0.0.0:8124/?query=INSERT%20INTO%20test%20FORMAT%20JSONEachRow' --data-binary @-  
Databend HTTP  
    listened at 0.0.0.0:8000  
    usage:  curl -u${USER} -p${PASSWORD}: --request POST '0.0.0.0:8000/v1/query/' --header 'Content-Type: application/json' --data-raw '{"sql": "SELECT avg(number) FROM numbers(100000000)"}'

9、使用bendsql连接databend测试 , https://docs.databend.com/guides/deploy/deploying-local

su - databend  
bendsql   

databend@localhost:8000/default>  
  
databend@localhost:8000/default> CREATE DATABASE test;  
0 row written in 0.042 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)  
  
databend@localhost:8000/default> use test;  
0 row read in 0.028 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)  
  
databend@localhost:8000/test> CREATE TABLE mytable(a int);  
0 row written in 0.053 sec. Processed 0 row, 0 B (0 row/s, 0 B/s)  
  
databend@localhost:8000/test> INSERT INTO mytable VALUES(1);  
1 row written in 0.108 sec. Processed 1 row, 5 B (9.27 row/s, 46 B/s)  
  
databend@localhost:8000/test> INSERT INTO mytable VALUES(2);  
1 row written in 0.102 sec. Processed 1 row, 5 B (9.81 row/s, 49 B/s)  
  
databend@localhost:8000/test> INSERT INTO mytable VALUES(3);  
1 row written in 0.120 sec. Processed 1 row, 5 B (8.33 row/s, 41 B/s)  
  
databend@localhost:8000/test> select * from mytable;   
┌─────────────────┐  
│        a        │  
│ Nullable(Int32) │  
├─────────────────┤  
│               1 │  
│               2 │  
│               3 │  
└─────────────────┘  
3 rows read in 0.066 sec. Processed 3 rows, 15 B (45.2 rows/s, 225 B/s)  
  
  
insert into mytable select * from mytable;  
... 若干次后  
  
  
root@localhost:8000/test> select count(*) from mytable;  
  
SELECT  
  count(*)  
FROM  
  mytable  
  
┌───────────┐  
│  count(*) │  
│   UInt64  │  
├───────────┤  
│ 402653184 │  
└───────────┘  
1 row read in 0.034 sec. Processed 1 row, 1 B (29.58 row/s, 29 B/s)

10、观察数据文件

databend@ba30599c4edc:~/databend_software$ cd /var/lib/databend/  
databend@ba30599c4edc:/var/lib/databend$ ll  
total 20K  
drwxr-xr-x 1 root     root     4.0K Feb  1 06:23 ..  
drwxr-xr-x 4 databend databend 4.0K Feb  1 06:23 .  
drwxr-xr-x 3 databend databend 4.0K Feb  1 06:32 data  
drwxr-xr-x 4 databend databend 4.0K Feb  1 07:20 raft  
databend@ba30599c4edc:/var/lib/databend$ du -sh *  
168M	data  
1.1M	raft

11、观察databend日志

databend@ba30599c4edc:/var/lib/databend$ cd /var/log/databend/  
databend@ba30599c4edc:/var/log/databend$ ll  
total 28K  
drwxr-xr-x 1 root     root     4.0K Feb  1 06:23 ..  
-rw-r--r-- 1 databend databend  268 Feb  1 06:30 databend-meta-1@foo_cluster.2024-02-01-06  
-rw-r--r-- 1 databend databend 3.5K Feb  1 06:33 databend-query-default.2024-02-01-06  
drwxr-xr-x 2 databend databend 4.0K Feb  1 07:18 .  
-rw-r--r-- 1 databend databend  402 Feb  1 07:20 databend-meta-1@foo_cluster.2024-02-01-07  
-rw-r--r-- 1 databend databend 2.6K Feb  1 07:21 databend-query-default.2024-02-01-07

12、停止 databend

su - databend  
cd /home/databend/databend_software   
./scripts/stop.sh

13、其他

使用对象存储

或者使用如下方法在deiban 11上安装bendsql

curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 -L -o /usr/share/keyrings/datafuselabs-keyring.gpg https://repo.databend.rs/deb/datafuselabs.gpg 
curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 -L -o /etc/apt/sources.list.d/datafuselabs.list https://repo.databend.rs/deb/datafuselabs.list 
apt-get update
apt-get install -y bendsql

期望 PostgreSQL|开源PolarDB 增加什么功能?

PolarDB 开源数据库

PolarDB 学习图谱

购买PolarDB云服务折扣活动进行中, 55元起

PostgreSQL 解决方案集合

德哥 / digoal's Github - 公益是一辈子的事.

About 德哥

digoal's wechat