Skip to content

Latest commit

 

History

History
283 lines (185 loc) · 13.2 KB

20240124_01.md

File metadata and controls

283 lines (185 loc) · 13.2 KB
Raw

用chatgpt排查PG create extension duckdb_fdw异常: could not load library...undefined symbol...

作者

digoal

日期

2024-01-24

标签

PostgreSQL , PolarDB , DuckDB , chatgpt , linux , lib , so , undefined symbol , nm , objdump , ldd

背景

使用gpt4_turbo_128K 排查PG create extension duckdb_fdw异常: could not load library...undefined symbol... .

create extension duckdb_fdw 报错

ERROR:  could not load library "/usr/lib/postgresql/14/lib/duckdb_fdw.so": /usr/lib/postgresql/14/lib/duckdb_fdw.so: undefined symbol: _ZN6duckdb9Exception25ConstructMessageRecursiveERKSsRSt6vectorINS_20ExceptionFormatValueESaIS4_EE

使用的是如下2个pg/duckdb学习docker image环境

duckdb_fdw安装过程:

cd $TEMP_DIR    
git clone --depth 1 https://github.com/alitrack/duckdb_fdw  
cd $TEMP_DIR/duckdb_fdw  
  
# 使用了已编译好的libduckdb.so    
wget -T 36000 -t 0 --waitretry=5 https://github.com/duckdb/duckdb/releases/download/v0.9.2/libduckdb-linux-aarch64.zip    
unzip -n -d . libduckdb-linux-aarch64.zip    
cp libduckdb.so $(pg_config --libdir)    
USE_PGXS=1 make    
USE_PGXS=1 make install

1

gpt4_turbo_128K回复使用ldd, nm, objdump查一下duckdb_fdw.so. 可以看到依赖libduckdb.so , 怀疑和编译好的libduckdb.so有关.

nm and objdump are powerful tools used for inspecting binary files such as shared libraries and object files. They can help diagnose issues with symbols and dependencies within binaries.

Here's how you can use these tools to diagnose the undefined symbol issue with duckdb_fdw.so:

Using nm

nm lists symbols from object files. If you want to check whether a specific symbol is present in a binary, you can use nm and then grep for the symbol.

  1. List symbols in duckdb_fdw.so:

    nm -C /usr/lib/postgresql/14/lib/duckdb_fdw.so | grep 'ConstructMessageRecursive'

    The -C flag demangles (decodes) the symbol names, which makes them more readable. Look for symbols related to ConstructMessageRecursive. If the symbol is undefined, it will have a U next to it.

  2. Check for the symbol in DuckDB's library:

    nm -C /path/to/libduckdb.so | grep 'ConstructMessageRecursive'

    Replace /path/to/libduckdb.so with the actual path to the DuckDB library file. You're looking to see if this symbol is defined in the DuckDB library, which should not have the U next to it if it's properly defined.

Using objdump

objdump displays information about object files. You can use it to examine the dynamic symbol table, which contains the symbols needed for dynamic linking.

  1. Inspect dynamic symbols of duckdb_fdw.so:

    objdump -T /usr/lib/postgresql/14/lib/duckdb_fdw.so | grep 'ConstructMessageRecursive'

    The -T option displays the dynamic symbol table of the shared object. If the symbol is undefined, it will be marked with *UND*.

  2. Inspect dynamic symbols of DuckDB's library:

    objdump -T /path/to/libduckdb.so | grep 'ConstructMessageRecursive'

    Again, replace /path/to/libduckdb.so with the actual path to the DuckDB library file. The symbol should be defined here and should not show *UND*.

Interpretation

If you see that the symbol is listed as undefined (U with nm or *UND* with objdump) in duckdb_fdw.so but is defined in libduckdb.so, this means that the linker is unable to resolve this symbol when attempting to load duckdb_fdw.so. This could be due to issues such as library path misconfiguration, version incompatibility, or improper building or installation procedures.

If the symbol is not found in libduckdb.so, this indicates that the duckdb_fdw extension might be compiled against a different version of the DuckDB library than the one you have installed, or there might be some corruption or issue with the library itself.

Based on the information you gather from nm and objdump, you can take appropriate action, such as recompiling duckdb_fdw against the correct version of DuckDB, fixing library paths, or reinstalling DuckDB if necessary.

2

下载libduckdb-src.zip重新编译, 而不是使用编译好的libduckdb.so

https://github.com/duckdb/duckdb/releases

https://github.com/duckdb/duckdb/releases/download/v0.9.2/libduckdb-src.zip

但是怎么把h,cpp,hpp变易成so? gpt4_turbo_128K回复:

如果你只有 .h, .cpp, .hpp 文件，并且想要将它们编译成 .so 文件（共享库），你将需要创建一个适当的编译命令或者使用 CMake 等构建系统来管理编译过程。这里，我将演示如何手动使用 g++ （GCC 的 C++ 编译器）来编译这些文件为 .so 文件。

  1. 安装编译工具：
    确保你的系统上安装了 g++ 和其他构建工具。在 Ubuntu 上，你可以使用下面的命令来安装：

    sudo apt-get update  
sudo apt-get install build-essential

  2. 编译源代码为对象文件：
    对于每个 .cpp 文件，你需要使用 g++ 来编译它们为对象文件（.o）。使用 -fPIC 选项（Position Independent Code）是创建共享库的要求，因为它允许代码在内存中任意位置运行。

    g++ -c -fPIC source1.cpp -o source1.o  
g++ -c -fPIC source2.cpp -o source2.o  
...

    可以用通配符来简化这个过程，假设当前目录下所有 .cpp 文件都是你想要编译的：

    g++ -c -fPIC *.cpp

  3. 创建共享库：
    将所有的对象文件链接成一个 .so 文件。使用 -shared 选项告诉编译器创建一个共享库。

    g++ -shared -o libmylibrary.so source1.o source2.o ...

    如果你有很多对象文件，可以使用通配符：

    g++ -shared -o libmylibrary.so *.o

  4. (可选) 设置库版本和soname：
    如果你想要设置版本或者 soname（共享对象名称），你可以使用 -Wl 选项。

    g++ -shared -Wl,-soname,libmylibrary.so.1 -o libmylibrary.so.1.0 source1.o source2.o ...

    这里，libmylibrary.so.1 是 soname，而 libmylibrary.so.1.0 是库文件的实际名称。

  5. 测试共享库：
    为了测试你的共享库，你可以创建一个简单的测试程序，然后使用 g++ 并链接你的库来编译它。

    g++ -L. -lmylibrary -o testprogram testprogram.cpp

    这里，-L. 表示告诉编译器在当前目录寻找库，而 -lmylibrary 链接到 libmylibrary.so。

  6. 运行时库路径：
    如果你在非标准库路径中创建了 .so 文件，运行时你可能需要设置 LD_LIBRARY_PATH 环境变量来指定共享库的位置。

    export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

    现在你可以运行你的测试程序，它应该会使用你的共享库。

请记住，这个过程是通用的，并且假定你所有的源文件都是独立的，不需要额外的依赖或者复杂的构建步骤。如果库有复杂的依赖关系，创建一个 CMakeLists.txt 文件来使用 CMake 构建系统可能更合适。

3

clang++ -c -fPIC duckdb.cpp -o duckdb.o   
  
clang++ -shared -o libduckdb.so *.o

编译完, 把libduckdb.so替换后重新安装duckdb_fdw, 报了另一个错, 看起来只是符号_ZN6duckdb9ExceptionC1ENS_13ExceptionTypeERKSs变了.

cd duckdb_fdw  
grep -r _ZN6duckdb9ExceptionC1ENS_13ExceptionTypeERKSs *  
  
duckdb_fdw.so  
sqlite3_api_wrapper.o

The grep command you've run indicates that the symbol _ZN6duckdb9ExceptionC1ENS_13ExceptionTypeERKSs is found in both duckdb_fdw.so and sqlite3_api_wrapper.o. This means that this specific symbol, which is a mangled C++ name, is indeed present in those binary files.

Given that the symbol is there, but PostgreSQL is reporting an "undefined symbol" error, the issue might be related to the following:

  1. Linkage Issues: The symbol may not be correctly linked. This could be due to the duckdb_fdw.so extension being compiled against a different version of the DuckDB library than the one currently installed on your system.

  2. Mismatched ABIs: The Application Binary Interface (ABI) expected by duckdb_fdw.so might differ from the ABI provided by the installed DuckDB library. This could occur if they were compiled with different compiler versions or settings.

  3. Incorrect Load Order: It's possible that the dependencies are not being loaded in the correct order, causing the symbol to appear undefined when duckdb_fdw.so is loaded.

To resolve the issue, you can try the following:

  1. Ensure Consistent Versions: Double-check that you have consistent versions of DuckDB and duckdb_fdw installed. They should be compatible with each other.

  2. Recompile duckdb_fdw: Recompile the duckdb_fdw extension with the correct and currently installed version of DuckDB to ensure they are compatible. Make sure you clean any previous build artifacts before recompiling.

  3. Install DuckDB Correctly: Verify that DuckDB is installed correctly and the headers and libraries are consistent with what duckdb_fdw expects.

  4. Set Proper Library Paths: Make sure that the library path (LD_LIBRARY_PATH) is set correctly, and PostgreSQL can find all necessary libraries at runtime.

  5. Use ldd to Check Dependencies: You can use the ldd command to check the shared library dependencies of duckdb_fdw.so:

    ldd /usr/lib/postgresql/14/lib/duckdb_fdw.so

    This will list the shared libraries that duckdb_fdw.so depends on and whether they are found.

  6. Look for Help: If you're still stuck, look for help in the DuckDB community or reach out to the maintainers.

  7. File an Issue: If you suspect this is a bug or regression, you may file an issue on the GitHub repository of DuckDB or duckdb_fdw with detailed information about your situation.

Remember, after making any changes to the system's libraries or PostgreSQL's configuration, it's important to restart the PostgreSQL server to ensure that changes take effect.

4

说明是编译so的参数和duckdb_fdw src Makefile 不一致, 修改后重新编译正常.

# 编译 libduckdb.so 可能比较费内存, 建设调大docker 内存限制到8GB以上    
  
cd /tmp   
wget -T 36000 -t 0 --waitretry=5 https://github.com/duckdb/duckdb/releases/download/v0.9.2/libduckdb-src.zip  
git clone --depth 1 https://github.com/alitrack/duckdb_fdw  
  
mkdir /tmp/duckdb_fdw/libduckdb   
mv /tmp/libduckdb-src.zip /tmp/duckdb_fdw/libduckdb   
cd /tmp/duckdb_fdw/libduckdb   
unzip libduckdb-src.zip  
  
unalias cp  
cp -f duckdb.h ../  
cp -f duckdb.hpp ../  
  
# 编译参数需要 -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=0 与 duckdb_fdw src Makefile 一致   
clang++ -c -fPIC -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=0 duckdb.cpp -o duckdb.o   
  
clang++ -shared -o libduckdb.so *.o   
  
cp -f libduckdb.so $(pg_config --libdir)   
cp -f libduckdb.so /tmp/duckdb_fdw/   
cd /tmp/duckdb_fdw   
USE_PGXS=1 make uninstall  
USE_PGXS=1 make clean  
USE_PGXS=1 make distclean  
USE_PGXS=1 make    
USE_PGXS=1 make install     

postgres=# create extension duckdb_fdw ;  
正常.

参考

  • man nm
  • man objdump
  • ldd
  • clang

期望 PostgreSQL|开源PolarDB 增加什么功能?

PolarDB 开源数据库

PolarDB 学习图谱

购买PolarDB云服务折扣活动进行中, 55元起

PostgreSQL 解决方案集合

德哥 / digoal's Github - 公益是一辈子的事.

About 德哥

digoal's wechat