静态编译pika #1148

kernelai · 2022-02-11T08:51:17Z

为了减少物理环境依赖包冲突和安装的繁琐，一些同学会尝试静态编译pika。以下是360静态打包脚本。核心思想是分别编译依赖库的静态包，然后再链接到一起。
依赖库：
gflags-v1.3.tar.gz
gperftools-2.7.tar.gz
libunwind-v1.1.zip
snappy-1.1.0.tar.gz
编译脚本：

#!/bin/bash

TOP_DIR=$PWD
CORE_NUM=$(cat /proc/cpuinfo | grep processor | wc -l)

# sudo yum erase libunwind gperftools gflags gflags-devel glog glog-devel snappy snappy-devel
# for f in $(ls /usr/local/lib | grep -P "snappy|tcmalloc|glog|gflags|unwind|profiler"); do sudo rm -f /usr/local/lib/$f; done
# rm -rf gflags-1.3 gperftools-gperftools-2.7 libunwind-1.1 snappy-1.1.0

# libunwind
unzip libunwind-v1.1.zip
cd libunwind-1.1
autoreconf -i
./configure --disable-shared
make -j$CORE_NUM
# sudo make install
cd $TOP_DIR

# gperftool (tcmalloc)
tar xf gperftools-2.7.tar.gz
cd gperftools-gperftools-2.7
./configure --disable-shared
make -j$CORE_NUM
# sudo make install
cd $TOP_DIR

# gflags
tar xf gflags-v1.3.tar.gz
cd gflags-1.3
./configure --disable-shared
make -j$CORE_NUM
# sudo make install
cd $TOP_DIR

# snappy
tar xf snappy-1.1.0.tar.gz
cd snappy-1.1.0
./configure --disable-shared
make -j$CORE_NUM
# sudo make install
cd $TOP_DIR

pika 3.3.X静态连接脚本：

#!/usr/bin/sh
source_path=~/pika-master
dependence_path=/home/xxx/make_static/pika_deps_source
g++ \
  ${source_path}/pika/src/build_version.o \
  ${source_path}/pika/src/pika_stable_log.o \
  ${source_path}/pika/src/pika_consensus.o \
  ${source_path}/pika/src/pika_slave_node.o \
  ${source_path}/pika/src/pika_statistic.o \
  ${source_path}/pika/src/pika_client_processor.o \
  ${source_path}/pika/src/pika_admin.o \
  ${source_path}/pika/src/pika_binlog.o \
  ${source_path}/pika/src/pika_bit.o \
  ${source_path}/pika/src/pika.o \
  ${source_path}/pika/src/pika_client_conn.o \
  ${source_path}/pika/src/pika_command.o \
  ${source_path}/pika/src/pika_pubsub.o \
  ${source_path}/pika/src/pika_conf.o \
  ${source_path}/pika/src/pika_dispatch_thread.o \
  ${source_path}/pika/src/pika_hash.o \
  ${source_path}/pika/src/pika_hyperloglog.o \
  ${source_path}/pika/src/pika_kv.o \
  ${source_path}/pika/src/pika_list.o \
  ${source_path}/pika/src/pika_monitor_thread.o \
  ${source_path}/pika/src/pika_server.o \
  ${source_path}/pika/src/pika_set.o \
  ${source_path}/pika/src/pika_geo.o \
  ${source_path}/pika/src/pika_geohash.o \
  ${source_path}/pika/src/pika_geohash_helper.o \
  ${source_path}/pika/src/pika_binlog_transverter.o \
  ${source_path}/pika/src/pika_binlog_reader.o \
  ${source_path}/pika/src/pika_partition.o \
  ${source_path}/pika/src/pika_repl_bgworker.o \
  ${source_path}/pika/src/pika_repl_client.o \
  ${source_path}/pika/src/pika_repl_client_conn.o \
  ${source_path}/pika/src/pika_repl_client_thread.o \
  ${source_path}/pika/src/pika_repl_server.o \
  ${source_path}/pika/src/pika_repl_server_conn.o \
  ${source_path}/pika/src/pika_repl_server_thread.o \
  ${source_path}/pika/src/pika_cmd_table_manager.o \
  ${source_path}/pika/src/pika_auxiliary_thread.o \
  ${source_path}/pika/src/pika_rm.o \
  ${source_path}/pika/src/pika_table.o \
  ${source_path}/pika/src/pika_rsync_service.o \
  ${source_path}/pika/src/pika_inner_message.pb.o \
  ${source_path}/pika/src/pika_slot.o \
  ${source_path}/pika/src/pika_data_distribution.o \
  ${source_path}/pika/src/pika_meta.o \
  ${source_path}/pika/src/pika_cluster.o \
  ${source_path}/pika/third/slash/slash/lib/libslash.a \
  ${source_path}/pika/third/pink/pink/lib/libpink.a \
  ${source_path}/pika/third/blackwidow/lib/libblackwidow.a \
  ${source_path}/pika/third/rocksdb/librocksdb.a \
  ${dependence_path}//libprotobuf.a\
  ${dependence_path}/libglog.a \
  ${dependence_path}/libgflags.a \
  ${dependence_path}/libsnappy.a \
  ${dependence_path}/libunwind.a \
  ${dependence_path}/libtcmalloc.a \
  ${source_path}/pika/src/pika_zset.o  -o pika -static-libstdc++ -lpthread -lrt -lz -lbz2

lqxhub · 2022-02-12T06:21:51Z

编译 pika

0 准备

需要软件 centos7.x gcc4.8 g++4.8 git cmake
最好用centos7系列, 在别的系统中编译可能会报错, 人生苦短, 别和自己过不去

使用make 编译的时候, 使用 make -jN (N=cpu核数加快编译速度)

1 安装工具

gcc g++ git 一般用 yum 安装
安装cmake

cmake 源码 https://cmake.org/download/

cmake依赖 openssl-devel 先用yum安装 openssl-devel
yum install openssl openssl-devel

编译安装cmake

解压cmake源码

./bootstrap

make && make install

2 解决rocksdb依赖,安装必备的库

rocksdb 依赖库

snappy gflags zlib bzip2 stdc++ lz4 zstd lzma 这些库都要安装为了能静态编译,这些库都要安装静态库

编译snappy

在pika3.3.X版本中, 一定要用 snappy 1.1.0的库

mkdir build && cd build

编译动态库
cmake -DSNAPPY_BUILD_BENCHMARKS=OFF -DSNAPPY_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=ON ..

make && make install

编译静态库
cmake -DSNAPPY_BUILD_BENCHMARKS=OFF -DSNAPPY_BUILD_TESTS=OFF -DBUILD_SHARED_LIBS=OFF ..

make && make install

我没有找到同时编译动态库和静态库的方法，只能分开编译了

编译gflags

./configure

make && make install
编译zlib

./configure

make && make install
bzip2 一般linux都自带, 没有的用 yum 安装一个

yum install bzip2-devel bzip2-libs
stdc++ 这个库需要安装静态库,直接 yum 安装

yum install libstdc++ libstdc++-devel libstdc++-static

安装lz4

解压,进入源码目录

cd build/cmake

mkdir build && build

cmake -DBUILD_STATIC_LIBS=ON -DLZ4_BUILD_CLI=OFF -DLZ4_BUILD_LEGACY_LZ4C=OFF CMAKE_BUILD_TYPE=Relase -D LZ4_POSITION_INDEPENDENT_LIB=OFF ..

make && make install

安装 zstd

解压源码

cd build/cmake/lib

make && make install

安装lzma

解压源码 xz压缩包

./configure --disable-xz --disable-xzdec --disable-lzmadec --disable-lzmainfo --disable-lzma-links --disable-scripts  --disable-doc
make && make install

最后, 所有的库都安装好后,把库加到 ld 目录中。因为有的库会安装到 /usr/local/lib64 目录下，但是有的系统没有把这个目录加到动态库搜索目录中,

解决办法

echo "/usr/local/lib64" >> /etc/ld.so.conf.d/usr.local.conf
echo "/usr/local/lib" >> /etc/ld.so.conf.d/usr.local.conf

执行 ldconfig

安装 protobuf

编译 protobuf, 用cmake编译 protobuf release 地址 https://github.com/protocolbuffers/protobuf/releases
下载 protobuf-cpp-XX.XX.XX.tar.gz

xx.XX.XX 是版本号，需要下载3.X版本的

./configure

make && make install

protobuf 也要安装静态库

到这 protobuf 安装完成

3 拉代码

拉 pika 代码 github https://github.com/OpenAtomFoundation/pika
这里最好通过代理去拉代码,要不然速度感人

git 设置代理

git config --global https.proxy http://127.0.0.1:1080

git config --global https.proxy https://127.0.0.1:1080

git config --global --unset http.proxy

git config --global --unset https.proxy

拉取依赖库代码

cd pika 源码目录

git submodule init
git submodule update

4 编译

预先编译

预先编译一次 glog 和 rocksdb, 因为这两个库直接用 pika 的 MakeFile 编译容易出问题,先预编译一下

先特殊编译一下glog库

cd /third/glog
./configu
make && make install

编译rocksdb

rocksdb编译完不需要安装，只需要复制一下静态库就好

使用cmake构建

cmake 需要 cmake3, cmake命令如下

cmake -DWITH_BZ2=ON -DWITH_MD_LIBRARY=OFF -DWITH_SNAPPY=ON -DWITH_ZLIB=ON -DWITH_ZSTD=ON -DWITH_TOOLS=OFF -DWITH_TESTS=OFF -DCMAKE_BUILD_TYPE=Release ..

make -jN

把编译后的 librocksdb.a 复制到 rocksdb 的根目录

正常编译

按照官方给的文档编译即可,

执行 make

等待拉各个依赖模块的代码然后自动编译

前期工作做好,编译中应该不会出错

检查编译是否正确

后续

编译完成后,这个可执行文件有很多 .so的依赖, 可移植比较差, 通过脚本编译一个依赖较少的.so文件

执行 link_3_3_path-master.sh 脚本等待完成链接

编译完成

正常编译

到 output/bin/ 目录中, 执行 ldd pika 如果没有 notfind 的依赖库说明编译完成了
静态链接
output/bin/ 目录中的是正常编译的, link_3_3_path-master.sh 同级目录下的是静态链接的

静态链接脚本

#!/usr/bin/sh
source_path=/root
dependence_path=/root/pika_static
g++ \
  ${source_path}/pika/src/build_version.o \
  ${source_path}/pika/src/pika_stable_log.o \
  ${source_path}/pika/src/pika_consensus.o \
  ${source_path}/pika/src/pika_slave_node.o \
  ${source_path}/pika/src/pika_statistic.o \
  ${source_path}/pika/src/pika_client_processor.o \
  ${source_path}/pika/src/pika_admin.o \
  ${source_path}/pika/src/pika_binlog.o \
  ${source_path}/pika/src/pika_bit.o \
  ${source_path}/pika/src/pika.o \
  ${source_path}/pika/src/pika_proxy.o \
  ${source_path}/pika/src/pika_proxy_cli.o \
  ${source_path}/pika/src/pika_proxy_conn.o \
  ${source_path}/pika/src/pika_client_conn.o \
  ${source_path}/pika/src/pika_command.o \
  ${source_path}/pika/src/pika_pubsub.o \
  ${source_path}/pika/src/pika_conf.o \
  ${source_path}/pika/src/pika_dispatch_thread.o \
  ${source_path}/pika/src/pika_hash.o \
  ${source_path}/pika/src/pika_hyperloglog.o \
  ${source_path}/pika/src/pika_kv.o \
  ${source_path}/pika/src/pika_list.o \
  ${source_path}/pika/src/pika_monitor_thread.o \
  ${source_path}/pika/src/pika_server.o \
  ${source_path}/pika/src/pika_set.o \
  ${source_path}/pika/src/pika_geo.o \
  ${source_path}/pika/src/pika_geohash.o \
  ${source_path}/pika/src/pika_geohash_helper.o \
  ${source_path}/pika/src/pika_binlog_transverter.o \
  ${source_path}/pika/src/pika_binlog_reader.o \
  ${source_path}/pika/src/pika_partition.o \
  ${source_path}/pika/src/pika_repl_bgworker.o \
  ${source_path}/pika/src/pika_repl_client.o \
  ${source_path}/pika/src/pika_repl_client_conn.o \
  ${source_path}/pika/src/pika_repl_client_thread.o \
  ${source_path}/pika/src/pika_repl_server.o \
  ${source_path}/pika/src/pika_repl_server_conn.o \
  ${source_path}/pika/src/pika_repl_server_thread.o \
  ${source_path}/pika/src/pika_cmd_table_manager.o \
  ${source_path}/pika/src/pika_auxiliary_thread.o \
  ${source_path}/pika/src/pika_rm.o \
  ${source_path}/pika/src/pika_table.o \
  ${source_path}/pika/src/pika_rsync_service.o \
  ${source_path}/pika/src/pika_inner_message.pb.o \
  ${source_path}/pika/src/pika_slot.o \
  ${source_path}/pika/src/pika_data_distribution.o \
  ${source_path}/pika/src/pika_meta.o \
  ${source_path}/pika/src/pika_cluster.o \
  ${source_path}/pika/third/slash/slash/lib/libslash.a \
  ${source_path}/pika/third/pink/pink/lib/libpink.a \
  ${source_path}/pika/third/blackwidow/lib/libblackwidow.a \
  ${source_path}/pika/third/rocksdb/librocksdb.a \
  ${source_path}/pika/src/pika_zset.o -static-libstdc++ -Wl,-Bstatic -lprotobuf -llz4 -lzstd -llz4 -lglog -lgflags -llzma -lsnappy -Wl,-Bdynamic -lpthread -lrt -lz -lbz2 -Wl,--dynamic-linker=/lib64/ld-linux-x86-64.so.2 -o pika

-Wl,-Bstatic 后面的是要 静态链接 的库

-Wl,-Bdynamic 后面的是要 动态链 接的库

kernelai added the compile label Feb 11, 2022

kernelai mentioned this issue Aug 24, 2022

pika源码编译coredump #1190

Closed

wanghenshui closed this as completed Feb 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

静态编译pika #1148

静态编译pika #1148

kernelai commented Feb 11, 2022 •

edited

lqxhub commented Feb 12, 2022 •

edited

静态编译pika #1148

静态编译pika #1148

Comments

kernelai commented Feb 11, 2022 • edited

lqxhub commented Feb 12, 2022 • edited

编译 pika

0 准备

1 安装工具

2 解决rocksdb依赖,安装必备的库

rocksdb 依赖库

安装 protobuf

3 拉代码

4 编译

预先编译

正常编译

后续

编译完成

静态链接脚本

kernelai commented Feb 11, 2022 •

edited

lqxhub commented Feb 12, 2022 •

edited