digoal
2023-03-18
PostgreSQL , PolarDB , debian , docker
《记录下 debian 在带Nvidia显卡笔记本上的部署 - 用于测试 cuda 应用》
https://docs.docker.com/engine/install/debian/
1、依赖
apt update
apt install -y ca-certificates curl gnupg lsb-release
2、配置docker apt source key
mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
3、配置docker apt source
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
4、部署docker
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
5、检查docker.service是否正常启动
root@localhost:~# systemctl -l|grep -i docker
sys-devices-virtual-net-docker0.device loaded active plugged /sys/devices/virtual/net/docker0
sys-subsystem-net-devices-docker0.device loaded active plugged /sys/subsystem/net/devices/docker0
docker.service loaded active running Docker Application Container Engine
docker.socket loaded active running Docker Socket for the API
如果 docker.service 没启动的话先启动
6、测试 docker
root@localhost:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:ffb13da98453e0f04d33a6eee5bb8e46ee50d08ebe17735fc0779d0349e889e9
Status: Downloaded newer image for hello-world:latest
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
root@localhost:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3fa117a21085 hello-world "/hello" 4 seconds ago Exited (0) 3 seconds ago hardcore_borg
7、清理测试docker容器和镜像
root@localhost:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest feb5d9fea6a5 18 months ago 13.3kB
root@localhost:~# docker rm 3fa117a21085
3fa117a21085
root@localhost:~# docker rmi hello-world
Untagged: hello-world:latest
Untagged: hello-world@sha256:ffb13da98453e0f04d33a6eee5bb8e46ee50d08ebe17735fc0779d0349e889e9
Deleted: sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
Deleted: sha256:e07ee1baac5fae6a26f30cabfe54a36d3402f96afda318fe0a96cec4ca393359
root@localhost:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@localhost:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
1、更多操作系统的参数介绍可参考:
《PostgreSQL on Linux 最佳部署手册 - 珍藏级》
vi /etc/sysctl.conf
vm.swappiness = 0
vm.overcommit_memory = 1
sysctl -p
《microsoft sql server on docker 部署》
《如何用 PolarDB 证明巴菲特的投资理念 - 包括PolarDB on Docker简单部署》
拉取镜像
docker pull debian:stable
创建容器
docker run -d -it --cap-add=SYS_PTRACE --privileged=true --name debian debian:stable
进入容器
docker exec -it -u root --privileged -w /root debian /bin/bash
运行中的容器
docker ps -a
停止容器
docker stop debian
删除容器
docker rm debian
启动容器
docker start debian
查看镜像
docker images
删除镜像
docker rm 3fa117a21085
设置变量(创建容器时)
docker run -d -it --cap-add=SYS_PTRACE --privileged=true --name mssql2022 -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Mssql2o2o-" -e "MSSQL_PID=EnterpriseCore" -p 1433:1433 mcr.microsoft.com/mssql/server:2022-latest
映射端口(创建容器时)
docker run -d -it --cap-add=SYS_PTRACE --privileged=true -p 7860:7860 --name aigc debian:stable
拷贝文件到容器
docker cp /Users/digoal/Downloads/1.html mssql2019:/home/postgres/1.html
将容器的文件拷贝到宿主机
docker cp mssql2019:/home/postgres/1.html /Users/digoal/Downloads/1.html
《Debian学习入门 - (作为服务器使用, Debian 操作系统可能是长期更好的选择?)》
《在debian中部署"人工智能生成内容"(Artificial Intelligence Generated Content,简称 AIGC)》
1、拉取debian stable images
docker pull debian:stable
2、创建容器
docker run -d -it --cap-add=SYS_PTRACE --privileged=true --name debian debian:stable
3、进入容器
docker exec -it -u root --privileged -w /root debian /bin/bash
配置 apt 国内源:
sed -i "s@http://\(deb\|security\).debian.org@http://mirrors.nju.edu.cn@g" /etc/apt/sources.list
apt update
apt-get reinstall apt-transport-https ca-certificates
sed -i "s@http://mirrors.nju.edu.cn@https://mirrors.nju.edu.cn@g" /etc/apt/sources.list
apt update
安装常用包:
apt install -y locales wget glibc-source zlib1g-dev pkg-config libicu-dev icu-devtools libbison-dev libfl-dev git libreadline-dev libedit-dev g++ make cmake man-db vim dnsutils clang libssl-dev default-jdk unixodbc unixodbc-dev bash-completion
配置 bashrc:
# add by digoal
alias rm='rm -i'
alias cp='cp -i'
alias ll='ls -larth'
alias mv='mv -i'
export LC_ALL=en_US.UTF-8
配置 bash 补齐: 在/etc/bash.bashrc中编辑enable bash completion in interactive shells这一项配置，将其注释全部去掉.
配置locale:
https://people.debian.org/~schultmc/locales.html
1、Edit /etc/locale.gen as root. If /etc/locale.gen does not exist, create it. An example /etc/locale.gen is below.
注释en_US.UTF-8 UTF-8
2、Run /usr/sbin/locale-gen as root
locale-gen en_US.UTF-8
创建用户:
useradd -s /bin/bash -d /home/postgres -m postgres
duckdb 测试:
git clone --depth 1 -b v0.7.1 https://github.com/duckdb/duckdb
cd duckdb
env CC=clang CXX=clang++ CMAKE_BUILD_PARALLEL_LEVEL=4 EXTENSION_STATIC_BUILD=1 BUILD_ICU=1 BUILD_TPCH=1 BUILD_TPCDS=1 BUILD_FTS=1 BUILD_VISUALIZER=1 BUILD_HTTPFS=1 BUILD_JSON=1 BUILD_JEMALLOC=1 BUILD_EXCEL=1 BUILD_INET=1 STATIC_OPENSSL=1 BUILD_SQLSMITH=1 BUILD_TPCE=1 make benchmark -j 4
这些基本配置弄好后, 这个容器装东西测试基本上该有的依赖都有, 比较省心.
4、在容器中部署duckdb (由于debian stable镜像中东西太少, 需要先安装一些依赖)
部署duckdb 依赖
apt install -y git libreadline-dev libedit-dev g++ make cmake man-db vim dnsutils clang libssl-dev
部署duckdb 0.7.1 分支
git clone --depth 1 -b v0.7.1 https://github.com/duckdb/duckdb
cd duckdb
env CC=clang CXX=clang++ CMAKE_BUILD_PARALLEL_LEVEL=4 EXTENSION_STATIC_BUILD=1 BUILD_ICU=1 BUILD_TPCH=1 BUILD_TPCDS=1 BUILD_FTS=1 BUILD_VISUALIZER=1 BUILD_HTTPFS=1 BUILD_JSON=1 BUILD_JEMALLOC=1 BUILD_EXCEL=1 BUILD_INET=1 STATIC_OPENSSL=1 BUILD_SQLSMITH=1 BUILD_TPCE=1 make benchmark -j 4
启动测试duckdb
cd build/release/
root@6f6e268beada:~/duckdb/build/release# ./duckdb
v0.7.1 b00b93f
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D .timer on
D create table t (id int, info text, ts timestamp);
Run Time (s): real 0.000 user 0.000192 sys 0.000028
D insert into t select id , md5(random()::text), now() from range(0,10000000) as t(id);
100% ▕████████████████████████████████████████████████████████████▏
Run Time (s): real 3.342 user 3.238521 sys 0.103135
D select count(distinct info) from t;
┌──────────────────────┐
│ count(DISTINCT info) │
│ int64 │
├──────────────────────┤
│ 9988448 │
└──────────────────────┘
Run Time (s): real 0.842 user 5.194127 sys 0.848967
D select count(distinct id) from t;
┌────────────────────┐
│ count(DISTINCT id) │
│ int64 │
├────────────────────┤
│ 10000000 │
└────────────────────┘
Run Time (s): real 0.504 user 3.492460 sys 0.174690
D explain analyze select count(distinct id) from t;
┌─────────────────────────────┐
│┌───────────────────────────┐│
│└───────────────────────────┘│
└─────────────────────────────┘
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Query Profiling Information ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
explain analyze select count(distinct id) from t;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││ Total Time: 0.506s ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌───────────────────────────┐
│ EXPLAIN_ANALYZE │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 0 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ UNGROUPED_AGGREGATE │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ count(DISTINCT #0) │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 1 │
│ (1.46s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ PROJECTION │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ id │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 10000000 │
│ (0.00s) │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│ SEQ_SCAN │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ t │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ id │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ EC: 0 │
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ 10000000 │
│ (0.01s) │
└───────────────────────────┘
Run Time (s): real 0.508 user 3.504543 sys 0.167030
apt install -y libicu-dev icu-devtools libbison-dev libfl-dev
useradd -s /bin/bash -d /home/postgres -m postgres
su - postgres
vi ~/.bashrc
# add by digoal
alias rm='rm -i'
alias cp='cp -i'
alias ll='ls -larth'
alias mv='mv -i'
export PATH=/home/postgres/pg15.2/bin:$PATH
export PGDATA=/home/postgres/pgdata
export PGUSER=postgres
export PGHOST=$PGDATA
export PGPORT=1921
export PGDATABASE=postgres
export LC_ALL=en_US.UTF-8
. ~/.bashrc
wget https://ftp.postgresql.org/pub/source/v15.2/postgresql-15.2.tar.bz2
tar -jxvf postgresql-15.2.tar.bz2
cd postgresql-15.2
env CC=clang CXX=clang++ ./configure --prefix=/home/postgres/pg15.2 --with-icu
make world -j 8
make install-world
initdb -D $PGDATA -U postgres -E UTF8 --locale-provider icu --icu-locale=C --lc-collate=C --lc-ctype=en_US.UTF-8
vi $PGDATA/postgresql.auto.conf
listen_addresses = '0.0.0.0'
port = 1921
max_connections = 1000
superuser_reserved_connections = 3
unix_socket_directories = '., /tmp'
tcp_keepalives_idle = 30
tcp_keepalives_interval = 10
tcp_keepalives_count = 3
client_connection_check_interval = 30
shared_buffers = 1GB
maintenance_work_mem = 1GB
dynamic_shared_memory_type = posix
vacuum_cost_delay = 0
bgwriter_delay = 10ms
bgwriter_lru_maxpages = 500
bgwriter_lru_multiplier = 5.0
effective_io_concurrency = 0
max_parallel_workers_per_gather = 0
max_parallel_maintenance_workers = 2
wal_level = minimal
synchronous_commit = off
wal_compression = on
wal_buffers = -1
wal_writer_delay = 10ms
checkpoint_timeout = 15min
checkpoint_completion_target = 0.5
max_wal_size = 4GB
min_wal_size = 1GB
archive_mode = off
max_wal_senders = 0
random_page_cost = 1.1
effective_cache_size = 8GB
log_destination = 'csvlog'
logging_collector = on
log_directory = 'log'
log_filename = 'postgresql-%Y-%m-%d.log'
log_rotation_age = 1d
log_rotation_size = 10MB
log_truncate_on_rotation = on
log_lock_waits = on
log_recovery_conflict_waits = on
log_statement = 'ddl'
log_timezone = 'Asia/Shanghai'
autovacuum = on
autovacuum_freeze_max_age = 800000000
autovacuum_multixact_freeze_max_age = 900000000
autovacuum_vacuum_cost_delay = 0ms
vacuum_freeze_table_age = 650000000
vacuum_freeze_min_age = 500000000
vacuum_failsafe_age = 1600000000
vacuum_multixact_freeze_table_age = 650000000
vacuum_multixact_freeze_min_age = 50000000
vacuum_multixact_failsafe_age = 1600000000
datestyle = 'iso, mdy'
timezone = 'Asia/Shanghai'
lc_messages = 'en_US.UTF-8'
lc_monetary = 'en_US.UTF-8'
lc_numeric = 'en_US.UTF-8'
lc_time = 'en_US.UTF-8'
default_text_search_config = 'pg_catalog.english'
pg_ctl start
postgres@localhost:~/pgdata$ psql
psql (15.2)
Type "help" for help.
postgres=# \copyright
PostgreSQL Database Management System
(formerly known as Postgres, then as Postgres95)
Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
Portions Copyright (c) 1994, The Regents of the University of California
Permission to use, copy, modify, and distribute this software and its
documentation for any purpose, without fee, and without a written agreement
is hereby granted, provided that the above copyright notice and this
paragraph and the following two paragraphs appear in all copies.
IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR
DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING
LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS
DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES,
INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS
ON AN "AS IS" BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO
PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
postgres=#
https://packages.debian.org/source/bullseye/flex
https://packages.debian.org/stable/libdevel/
