digoal
2024-03-19
PostgreSQL , PolarDB , DuckDB , aws aurora , neon
PostgreSQL neon(开源替代AWS Aurora存算分离PG数据库) 编译和使用测试
参考文档: https://github.com/neondatabase/neon/blob/main/docs/SUMMARY.md
docker image:
docker resource: 10G, 4G swap
后期将直接整合进入这两个镜像.
https://github.com/neondatabase/neon
依赖1
apt-get install -y build-essential libtool libreadline-dev zlib1g-dev flex bison libseccomp-dev libssl-dev clang pkg-config libpq-dev cmake postgresql-client protobuf-compiler libcurl4-openssl-dev openssl lsof libicu-dev libseccomp-dev
https://github.com/protocolbuffers/protobuf/releases
依赖2 protobuf
cd /tmp
curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 --proto '=https' --tlsv1.2 -sSf -L https://github.com/protocolbuffers/protobuf/releases/download/v26.0/protoc-26.0-linux-aarch_64.zip
curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 --proto '=https' --tlsv1.2 -sSf -L https://github.com/protocolbuffers/protobuf/releases/download/v26.0/protoc-26.0-linux-x86_64.zip
cd /tmp
mkdir protoc
mv protoc-26.0-linux-aarch_64.zip /tmp/protoc
cd /tmp/protoc
unzip protoc-26.0-linux-aarch_64.zip
unalias cp
cp -f /tmp/protoc/bin/protoc /usr/local/bin/protoc
cd /tmp/protoc/include
cp -f -r google /usr/local/include/
依赖3 poetry
pip3 install poetry
useradd -b /home/neon -m -d /home/neon -s /bin/bash neon
echo "# add by digoal " >>/home/neon/.bashrc
echo "alias rm='rm -i' " >>/home/neon/.bashrc
echo "alias cp='cp -i' " >>/home/neon/.bashrc
echo "alias ll='ls -larth' " >>/home/neon/.bashrc
echo "alias mv='mv -i' " >>/home/neon/.bashrc
echo "export PGHOME=/usr/lib/postgresql/14 " >>/home/neon/.bashrc
echo "export PATH=$PGHOME/bin:$PATH " >>/home/neon/.bashrc
echo "export LD_LIBRARY_PATH=$PGHOME/lib:/usr/local/lib:/usr/lib/aarch64-linux-gnu:$LD_LIBRARY_PATH " >>/home/neon/.bashrc
echo "export PGUSER=$(whoami) " >>/home/neon/.bashrc
echo "export PGHOST=127.0.0.1 " >>/home/neon/.bashrc
echo "export PGPORT=55432 " >>/home/neon/.bashrc
echo "export PGDATABASE=postgres " >>/home/neon/.bashrc
echo "export LC_ALL=en_US.UTF-8 " >>/home/neon/.bashrc
echo "export PATH=/tmp/cmake-3.27.9/bin:$PATH " >>/home/neon/.bashrc
echo "export MANPATH=/tmp/cmake-3.27.9/man:$MANPATH " >>/home/neon/.bashrc
echo "export PATH=/usr/local/go/bin:/usr/local/go/bin:/home/neon/.cargo/bin:/tmp/cmake-3.27.9/bin:/usr/lib/postgresql/14/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/echo "bin:/sbin:/bin " >>/home/neon/.bashrc
echo "export LD_LIBRARY_PATH=$PGHOME/lib/oracle:$LD_LIBRARY_PATH " >>/home/neon/.bashrc
echo ". $HOME/.cargo/env " >>/home/neon/.bashrc
echo "export RUSTUP_DIST_SERVER=https://mirrors.ustc.edu.cn/rust-static " >>/home/neon/.bashrc
echo "export RUSTUP_UPDATE_ROOT=https://mirrors.ustc.edu.cn/rust-static/rustup " >>/home/neon/.bashrc
echo "export PATH=/home/neon/neon/target/debug:$PATH " >>/home/neon/.bashrc
echo "# export PATH=/home/neon/neon/pg_install/v14/bin:$PATH " >>/home/neon/.bashrc
echo "# export PATH=/home/neon/neon/pg_install/v15/bin:$PATH " >>/home/neon/.bashrc
echo "export PATH=/home/neon/neon/pg_install/v16/bin:$PATH " >>/home/neon/.bashrc
su - neon
rust
curl -Z --connect-timeout 120 -m 36000 --retry 12000 --retry-delay 5 --retry-max-time 1200 --proto '=https' --tlsv1.2 -sSf -L https://sh.rustup.rs -o rustup-init.sh
chmod +x rustup-init.sh
./rustup-init.sh -y
source $HOME/.cargo/env
neon
cd /home/neon
git clone --depth 1 -b release-5090 https://github.com/neondatabase/neon.git
cd /home/neon/neon/vendor
git clone --depth 1 -b REL_16_STABLE_neon https://github.com/neondatabase/postgres.git postgres-v16
git clone --depth 1 -b REL_15_STABLE_neon https://github.com/neondatabase/postgres.git postgres-v15
git clone --depth 1 -b REL_14_STABLE_neon https://github.com/neondatabase/postgres.git postgres-v14
cd /home/neon/neon
make -j`nproc` -s
cargo neon init --pg-version 16
cargo neon start
cargo neon tenant create --set-default --pg-version 16
cargo neon endpoint create main --pg-version 16
cargo neon endpoint start main
psql -h 127.0.0.1 -p 55432 -U cloud_admin
psql (14.11 (Debian 14.11-1.pgdg110+2), server 16.2)
WARNING: psql major version 14, server major version 16.
Some psql features might not work.
Type "help" for help.
postgres=# create role neon superuser login;
CREATE ROLE
部分settings
config_file | /home/neon/neon/.neon/endpoints/main/pgdata/postgresql.conf
data_directory | /home/neon/neon/.neon/endpoints/main/pgdata
hba_file | /home/neon/neon/.neon/endpoints/main/pgdata/pg_hba.conf
neon.console_url |
neon.extension_server_port | 55433
neon.file_cache_path | file.cache
neon.file_cache_size_limit | 0
neon.flush_output_after | 8
neon.forward_ddl | on
neon.logical_replication_max_snap_files | 300
neon.max_cluster_size | -1
neon.max_file_cache_size | 0
neon.max_reconnect_attempts | 60
neon.pageserver_connstring | postgresql://no_user@127.0.0.1:64000
neon.primary_is_running | off
neon.readahead_buffer_size | 128
neon.relsize_hash_size | 65536
neon.safekeeper_connect_timeout | 10000
neon.safekeeper_reconnect_timeout | 1000
neon.safekeepers | 127.0.0.1:5454
neon.stripe_size | 32768
neon.tenant_id | 49b76bb52bbf36c9f64a692320bd8d5d
neon.timeline_id | 7bb35d1a0b8be7b495f99c6b149a3d97
neon_test_evict | off
# cargo neon stop
# 如果stop失败, 建议重新搭建neon, rm -rf .neon
再次启动
# cargo neon start
https://github.com/neondatabase/neon#running-local-installation
https://github.com/neondatabase/neon/blob/main/control_plane/README.md
- Start pageserver and postgres on top of it (should be called from repo root):
# Create repository in .neon with proper paths to binaries and data
# Later that would be responsibility of a package install script
> cargo neon init
Initializing pageserver node 1 at '127.0.0.1:64000' in ".neon"
# start pageserver, safekeeper, and broker for their intercommunication
> cargo neon start
Starting neon broker at 127.0.0.1:50051.
storage_broker started, pid: 2918372
Starting pageserver node 1 at '127.0.0.1:64000' in ".neon".
pageserver started, pid: 2918386
Starting safekeeper at '127.0.0.1:5454' in '.neon/safekeepers/sk1'.
safekeeper 1 started, pid: 2918437
# create initial tenant and use it as a default for every future neon_local invocation
> cargo neon tenant create --set-default
tenant 9ef87a5bf0d92544f6fafeeb3239695c successfully created on the pageserver
Created an initial timeline 'de200bd42b49cc1814412c7e592dd6e9' at Lsn 0/16B5A50 for tenant: 9ef87a5bf0d92544f6fafeeb3239695c
Setting tenant 9ef87a5bf0d92544f6fafeeb3239695c as a default one
# create postgres compute node
> cargo neon endpoint create main
# start postgres compute node
> cargo neon endpoint start main
Starting new endpoint main (PostgreSQL v14) on timeline de200bd42b49cc1814412c7e592dd6e9 ...
Starting postgres at 'postgresql://cloud_admin@127.0.0.1:55432/postgres'
# check list of running postgres instances
> cargo neon endpoint list
ENDPOINT ADDRESS TIMELINE BRANCH NAME LSN STATUS
main 127.0.0.1:55432 de200bd42b49cc1814412c7e592dd6e9 main 0/16B5BA8 running - Now, it is possible to connect to postgres and run some queries:
> psql -p 55432 -h 127.0.0.1 -U cloud_admin postgres
postgres=# CREATE TABLE t(key int primary key, value text);
CREATE TABLE
postgres=# insert into t values(1,1);
INSERT 0 1
postgres=# select * from t;
key | value
-----+-------
1 | 1
(1 row)
- And create branches and run postgres on them:
# create branch named migration_check
> cargo neon timeline branch --branch-name migration_check
Created timeline 'b3b863fa45fa9e57e615f9f2d944e601' at Lsn 0/16F9A00 for tenant: 9ef87a5bf0d92544f6fafeeb3239695c. Ancestor timeline: 'main'
# check branches tree
> cargo neon timeline list
(L) main [de200bd42b49cc1814412c7e592dd6e9]
(L) ┗━ @0/16F9A00: migration_check [b3b863fa45fa9e57e615f9f2d944e601]
# create postgres on that branch
> cargo neon endpoint create migration_check --branch-name migration_check
# start postgres on that branch
> cargo neon endpoint start migration_check
Starting new endpoint migration_check (PostgreSQL v14) on timeline b3b863fa45fa9e57e615f9f2d944e601 ...
Starting postgres at 'postgresql://cloud_admin@127.0.0.1:55434/postgres'
# check the new list of running postgres instances
> cargo neon endpoint list
ENDPOINT ADDRESS TIMELINE BRANCH NAME LSN STATUS
main 127.0.0.1:55432 de200bd42b49cc1814412c7e592dd6e9 main 0/16F9A38 running
migration_check 127.0.0.1:55434 b3b863fa45fa9e57e615f9f2d944e601 migration_check 0/16F9A70 running
# this new postgres instance will have all the data from 'main' postgres,
# but all modifications would not affect data in original postgres
> psql -p 55434 -h 127.0.0.1 -U cloud_admin postgres
postgres=# select * from t;
key | value
-----+-------
1 | 1
(1 row)
postgres=# insert into t values(2,2);
INSERT 0 1
# check that the new change doesn't affect the 'main' postgres
> psql -p 55432 -h 127.0.0.1 -U cloud_admin postgres
postgres=# select * from t;
key | value
-----+-------
1 | 1
(1 row) - If you want to run tests afterwards (see below), you must stop all the running pageserver, safekeeper, and postgres instances
you have just started. You can terminate them all with one command:
> cargo neon stop More advanced usages can be found at Control Plane and Neon Local.
If you encounter errors during setting up the initial tenant, it's best to stop everything (cargo neon stop) and remove the .neon directory. Then fix the problems, and start the setup again.
This crate contains tools to start a Neon development environment locally. This utility can be used with the cargo neon command.
To create and start a local development environment with Postgres 16, you will need to provide --pg-version flag to 3 of the start-up commands.
cargo neon init --pg-version 16
cargo neon start
cargo neon tenant create --set-default --pg-version 16
cargo neon endpoint create main --pg-version 16
cargo neon endpoint start main By default, cargo neon starts an endpoint with cloud_admin and postgres database. If you want to have a role and a database similar to what we have on the cloud service, you can do it with the following commands when starting an endpoint.
cargo neon endpoint create main --pg-version 16 --update-catalog true
cargo neon endpoint start main --create-test-user true The first command creates neon_superuser and necessary roles. The second command creates test user and neondb database. You will see a connection string that connects you to the test user after running the second command.
