Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.11.0 doesn't find ncbi-vdb: configure: error: required ncbi-vdb package not found #489

Closed
yurivict opened this issue Apr 13, 2021 · 27 comments

Comments

@yurivict
Copy link

I call configure with these arguments:

--with-ngs-sdk-prefix=/usr/local --with-ncbi-vdb-sources=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0 --with-xml2-prefix=/usr/local --with-hdf5-prefix=/usr/local CC=cc CXX=c++ TOOLS= --without-debug --prefix=/usr/local --build=amd64-portbld-freebsd12.2

It breaks after saying that it found it:

checking for ngs-sdk package...
	includes... /usr/local
	libraries... /usr/local/lib
includes: /usr/local/include
libraries: /usr/local/lib
checking for ncbi-vdb package source files and build results...
	includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
	src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.
Please report the problem to mzaki@niid.go.jp [maintainer] and attach the
"/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).
@klymenko
Copy link
Contributor

klymenko commented Apr 16, 2021

How did you configure ncbi-vdb and what did it print?

@yurivict
Copy link
Author

configuration command is:

MAKE=gmake ac_cv_path_PERL=/usr/local/bin/perl ac_cv_path_PERL_PATH=/usr/local/bin/perl  PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin SHELL=/bin/sh CONFIG_SHELL=/bin/sh CMAKE_PREFIX_PATH="/usr/local" CONFIG_SITE=/disk-samsung/freebsd-ports/Templates/config.site lt_cv_sys_max_cmd_len=524288 \
./configure \
--with-xml2-prefix=/usr/local  --with-hdf5-prefix=/usr/local  --with-mbedtls-prefix=/usr/local  CC="cc" CXX="c++" TOOLS="clang" --prefix=/usr/local

It prints:

===>  Configuring for ncbi-vdb-2.11.0_1
Configuring NCBI-VDB package
checking system type... FreeBSD
checking machine architecture... amd64
checking for supported architecture... amd64 (64 bits) is supported
checking for supported OS... FreeBSD (bsd) is supported
checking for supported tool chain... clang tool chain is supported
checking for c++... yes
checking for Python 3... skipped
checking for fuse library... no
checking for hdf5 library... yes
checking for magic library... yes
checking for mbedtls library... yes
checking for xml2 library... yes
checking for ngs-sdk package...
	includes... ../ngs/ngs-sdk
	includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ngs/ngs-sdk
	libraries... no
configure: ngs-sdk package: found interface files but not libraries.
includes: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ngs-2.11.0/ngs-sdk
checking for ngs-java package...
	jar... configure: optional ngs-java package not found: skipped.

configure: creating 'build/Makefile.config.bsd.x86_64'
configure: creating 'Makefile.config'
configure: creating 'reconfigure'
build type: release
build prefix: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb
build output path: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel
includedir: /usr/local/include
bindir: /usr/local/bin
libdir: /usr/local/lib
CC = cc -c
CPP = c++
configured with: "'--with-xml2-prefix=/usr/local' '--with-hdf5-prefix=/usr/local' '--with-mbedtls-prefix=/usr/local' 'CC=cc' 'CXX=c++' 'TOOLS=clang' '--prefix=/usr/local' '--build=amd64-portbld-freebsd12.2'"

@klymenko
Copy link
Contributor

Try to add --with-ncbi-vdb-build=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2

@yurivict
Copy link
Author

Try to add --with-ncbi-vdb-build=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2

No, this didn't help.

checking for ncbi-vdb package source files and build results...
	includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
	src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
	libraries... no
	src... 	libraries... no
	src... configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.

What exactly does the build need?

I have ncbi-vdb installed:

$ pkg info -l ncbi-vdb
ncbi-vdb-2.11.0_1:
	/usr/local/include/ncbi-vdb/NGS.hpp
	/usr/local/lib/libkdf5-static.a
	/usr/local/lib/libkdf5.a
	/usr/local/lib/libkdf5.a.2
	/usr/local/lib/libkdf5.a.2.11.0
	/usr/local/lib/libkdf5.so
	/usr/local/lib/libkdf5.so.2
	/usr/local/lib/libkdf5.so.2.11.0
	/usr/local/lib/libncbi-ngs-c++-static.a
	/usr/local/lib/libncbi-ngs-c++.a
	/usr/local/lib/libncbi-ngs-c++.a.2
	/usr/local/lib/libncbi-ngs-c++.a.2.11.0
	/usr/local/lib/libncbi-ngs-static.a
	/usr/local/lib/libncbi-ngs.a
	/usr/local/lib/libncbi-ngs.a.2
	/usr/local/lib/libncbi-ngs.a.2.11.0
	/usr/local/lib/libncbi-vdb-static.a
	/usr/local/lib/libncbi-vdb.a
	/usr/local/lib/libncbi-vdb.a.2
	/usr/local/lib/libncbi-vdb.a.2.11.0
	/usr/local/lib/libncbi-vdb.so
	/usr/local/lib/libncbi-vdb.so.2
	/usr/local/lib/libncbi-vdb.so.2.11.0
	/usr/local/lib/libncbi-wvdb-static.a
	/usr/local/lib/libncbi-wvdb.a
	/usr/local/lib/libncbi-wvdb.a.2
	/usr/local/lib/libncbi-wvdb.a.2.11.0
	/usr/local/lib/libncbi-wvdb.so
	/usr/local/lib/libncbi-wvdb.so.2
	/usr/local/lib/libncbi-wvdb.so.2.11.0
	/usr/local/lib/ncbi/certs.kfg
	/usr/local/lib/ncbi/default.kfg
	/usr/local/share/licenses/ncbi-vdb-2.11.0_1/LICENSE
	/usr/local/share/licenses/ncbi-vdb-2.11.0_1/PD
	/usr/local/share/licenses/ncbi-vdb-2.11.0_1/catalog.mk

Is anything missing (was forgotten to be installed) in the above file list that sra-tools needs so that it also needs a full build directory?

@klymenko
Copy link
Contributor

Yes, sra-tools needs the full build of ncbi-vdb.

Do you have lib/libncbi-vdb.a and ilib/libkapp.a in /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel?

@yurivict
Copy link
Author

Do you have lib/libncbi-vdb.a and ilib/libkapp.a in /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel?

Both files are present.

@klymenko
Copy link
Contributor

Please add -d option to configure and post the output.

@yurivict
Copy link
Author

===>  Configuring for sra-tools-2.11.0
Configuring SRA-TOOLS package
checking system type... FreeBSD
checking machine architecture... amd64
checking for supported architecture... amd64 (64 bits) is supported
checking for supported OS... FreeBSD (FreeBSD) is supported
checking for supported tool chain... clang tool chain is supported
checking for c++... 
		running c++ --help
	yes
checking for Python 3... no
checking for QMake... 
		running qmake... QMake version 3.1
 qmake
checking for fuse library... 
		running echo -e '#include <fuse.h>  \n int main() { fuse_get_context(); }\n' | clang -xc -D_FILE_OFFSET_BITS=64 -I/usr/include  - -lfuse
<stdin>:1:10: fatal error: 'fuse.h' file not found
#include <fuse.h>  
         ^~~~~~~~
1 error generated.
	no
checking for hdf5 library... '/usr/local/lib': 
		running echo -e '#include <hdf5.h>  \n int main() { H5close         (); }\n' | clang -xc  -I/usr/local/include -L/usr/local/lib - -Wl,-Bstatic -lhdf5 -Wl,-Bdynamic -ldl -lm -lz
ld: error: undefined symbol: SZ_encoder_enabled
>>> referenced by H5Z.o:(H5Z__init_package) in archive /usr/local/lib/libhdf5.a

ld: error: undefined symbol: SZ_BufftoBuffCompress
>>> referenced by H5Zszip.o:(H5Z_filter_szip) in archive /usr/local/lib/libhdf5.a

ld: error: undefined symbol: SZ_BufftoBuffDecompress
>>> referenced by H5Zszip.o:(H5Z_filter_szip) in archive /usr/local/lib/libhdf5.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
	no
checking for magic library... 
		running echo -e '#include <magic.h> \n int main() { magic_open     (0); }\n' | clang -xc  -I/usr/include  - -lmagic
	yes
		HAVE_MAGIC = 1
		MAGIC_INCDIR = /usr/include
checking for xml2 library... '/usr/local/lib': 
		running echo -e '#include <libxml/xmlreader.h>\nint main() { xmlInitParser  ( ); }\n' | clang -xc  -I/usr/local/include/libxml2 -L/usr/local/lib - -lxml2
	yes
		HAVE_XML2 = 1
		XML2_INCDIR = /usr/local/include/libxml2
		XML2_LIBDIR = /usr/local/lib
checking for ngs-sdk package...
	includes... /usr/local
	libraries... 
		checking /usr/local/FreeBSD/clang/amd64/rel/lib/libngs-c++.a
		checking /usr/local/lib/libngs-c++.a
	/usr/local/lib
includes: /usr/local/include
libraries: /usr/local/lib
checking for ncbi-vdb package source files and build results...
	includes... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
	src... /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0
	libraries... 
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/FreeBSD/clang/amd64/rel/lib/libncbi-vdb.a
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/lib/libncbi-vdb.a
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/FreeBSD/clang/amd64/dbg/lib/libncbi-vdb.a
	no
	src... 	libraries... 
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/FreeBSD/clang/amd64/rel/lib/libncbi-vdb.a
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/lib/libncbi-vdb.a
		checking /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/FreeBSD/clang/amd64/dbg/lib/libncbi-vdb.a
	no
	src... 		not found /usr/local/ncbi/ncbi-vdb
		not found /disk-samsung/freebsd-ports/biology/sra-tools/work/ncbi/ncbi-vdb
		ncbi-vdb: includes: /disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces; libs: ; ilibs: 
configure: error: required ncbi-vdb package not found.
===>  Script "configure" failed unexpectedly.
Please report the problem to mzaki@niid.go.jp [maintainer] and attach the
"/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
(e.g. a /usr/local/sbin/pkg-static info -g -Ea).
*** Error code 1

@klymenko
Copy link
Contributor

Copy setup/os-arch.prl from ncbi-vdb to sra-tools.

@yurivict
Copy link
Author

Now it passed the configure stage, and breaks at the build stage:

gcc -c  -std=c11  -DNDEBUG -m64   -DBSD -DUNIX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=bsd64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/override -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/bsd -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/linux -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/unix -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc/amd64 -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/bsd -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/linux -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/unix -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ext -I/usr/local/include -I. -MD -o vdb-dump-tools.o /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-tools.c -D__mod_name__=tools/vdb-dump -D__file_name__=vdb-dump-tools -D__file_ext__=c
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.config.bsd.x86_64:114: target '/usr/local/lib' given more than once in the same rule
gcc -c  -std=c11  -DNDEBUG -m64   -DBSD -DUNIX -D_REENTRANT -D_FILE_OFFSET_BITS=64 -DPKGNAME=bsd64 -D_ARCH_BITS=__SIZEOF_POINTER__*__CHAR_BIT__ -DLIBPREFIX=lib -DSHLIBEXT=so  -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/override -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/bsd -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/linux -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/unix -I/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc/amd64 -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/cc/gcc -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/bsd -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/linux -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/os/unix -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/interfaces/ext -I/usr/local/include -I. -MD -o vdb-dump-str.o /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-str.c -D__mod_name__=tools/vdb-dump -D__file_name__=vdb-dump-str -D__file_ext__=c
gmake[8]: Entering directory '/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/driver-tool/secure'
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.c:27:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-redir.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-str.c:34:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
1 error generated.
gmake[5]: *** [/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.rules:31: vdb-dump-str.o] Error 1
gmake[5]: *** Waiting for unfinished jobs....
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-coldefs.c:27:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-helper.h:55:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-context.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-redir.h:40:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file.h:39:
In file included from /disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfs/file-v2.h:35:
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/interfaces/kfc/refcount.h:39:10: fatal error: 'atomic32.h' file not found
#include <atomic32.h>
         ^~~~~~~~~~~~
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/tools/vdb-dump/vdb-dump-tools.c:41:10: fatal error: 'bitstr.h' file not found
#include <bitstr.h>
         ^~~~~~~~~~
/disk-samsung/freebsd-ports/biology/sra-tools/work/sra-tools-2.11.0/build/Makefile.config.bsd.x86_64:114: target '/usr/local/lib' given more than once in the same rule
1 error generated.

@klymenko
Copy link
Contributor

What command did you run to build ncbi-vdb?

@yurivict
Copy link
Author

configure:

MAKE=gmake ac_cv_path_PERL=/usr/local/bin/perl ac_cv_path_PERL_PATH=/usr/local/bin/perl  PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin SHELL=/bin/sh CONFIG_SHELL=/bin/sh CMAKE_PREFIX_PATH="/usr/local" CONFIG_SITE=/disk-samsung/freebsd-ports/Templates/config.site lt_cv_sys_max_cmd_len=524288 \
./configure \
--with-xml2-prefix=/usr/local  --with-hdf5-prefix=/usr/local  --with-mbedtls-prefix=/usr/local  CC="cc" CXX="c++" TOOLS="clang" --prefix=/usr/local

make:

PERL_USE_UNSAFE_INC=1 XDG_DATA_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  XDG_CONFIG_HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work  HOME=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work PATH=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/.bin:/home/yuri/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin NO_PIE=yes MK_DEBUG_FILES=no MK_KERNEL_SYMBOLS=no SHELL=/bin/sh NO_LINT=YES PREFIX=/usr/local  LOCALBASE=/usr/local  CC="cc" CFLAGS="-O2 -pipe -fno-omit-frame-pointer  -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing "  CPP="cpp" CPPFLAGS="-fno-omit-frame-pointer -isystem /usr/local/include"  LDFLAGS=" -fstack-protector-strong -L/usr/local/lib " LIBS=""  CXX="c++" CXXFLAGS="-O2 -pipe -fno-omit-frame-pointer -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -fno-omit-frame-pointer  -isystem /usr/local/include "  MANPREFIX="/usr/local" BSD_INSTALL_PROGRAM="install  -s -m 555"  BSD_INSTALL_LIB="install  -s -m 0644"  BSD_INSTALL_SCRIPT="install  -m 555"  BSD_INSTALL_DATA="install  -m 0644"  BSD_INSTALL_MAN="install  -m 444" \
gmake \
CCNAME=cc CXXNAME=c++ PKGCFLAGS="-O2 -pipe -fno-omit-frame-pointer  -I/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/lib -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing " DESTDIR=/disk-samsung/freebsd-ports/biology/ncbi-vdb/work/stage

@klymenko
Copy link
Contributor

I cannot build ncbi-vdb with your PR.
See ncbi/ncbi-vdb#37 (comment)

@outpaddling
Copy link

Any more progress on this?
I fixed a couple of build issues in sra-tools a while back:
https://github.com/outpaddling/freebsd-ports-wip/tree/master/sra-tools/files
Still getting link errors, though:
g++ -o /usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/bin/sratools.2.11.0 -DNDEBUG -m64 -L/usr/local/lib sratools.o sratools2.o imp_fasterq_dump.o imp_fastq_dump.o imp_prefetch.o imp_sam_dump.o imp_sra_pileup.o imp_srapath.o imp_vdb_dump.o cmdline.o config.o proc.o run-source.o uuid.o service.o -L/usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/lib -L/usr/ports/wip/sra-tools/work/sra-tools-2.11.0/amd64-portbld-freebsd12.2/sra-tools/bsd/clang/x86_64/rel/ilib -L/usr/local/lib -L/usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib -L/usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/ilib -Wl,-Bstatic -lncbi-vdb -ljson -Wl,-Bdynamic -lepoll-shim -lncbi-vdb -lmbedtls -lexecinfo -lz -lbz2 -lpthread -lm
/usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function MakeAwsAuthenticationHeader': aws-auth.c:(.text+0x76): undefined reference to vdb_mbedtls_md_info_from_type'
/usr/local/bin/ld: aws-auth.c:(.text+0xb1): undefined reference to vdb_mbedtls_md_hmac' /usr/local/bin/ld: aws-auth.c:(.text+0xe6): undefined reference to vdb_mbedtls_base64_encode'
/usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function Base64InIdentityDocument': aws-auth.c:(.text+0x603): undefined reference to vdb_mbedtls_base64_encode'
/usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function Base64InIdentityPkcs7': aws-auth.c:(.text+0x6ba): undefined reference to vdb_mbedtls_base64_encode'
/usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-aws-auth.pic.o): in function MakeLocation': aws-auth.c:(.text+0x77c): undefined reference to vdb_mbedtls_base64_encode'
/usr/local/bin/ld: aws-auth.c:(.text+0x7c6): undefined reference to vdb_mbedtls_base64_encode' /usr/local/bin/ld: /usr/ports/biology/ncbi-vdb/work/ncbi-vdb-2.11.0/amd64-portbld-freebsd12.2/ncbi-vdb/bsd/clang/x86_64/rel/lib/libncbi-vdb.a(cloud-gcp.pic.o): in function Sign_RSA_SHA256':
gcp.c:(.text+0x9e): undefined reference to vdb_mbedtls_entropy_init' /usr/local/bin/ld: gcp.c:(.text+0xad): undefined reference to vdb_mbedtls_ctr_drbg_init'
/usr/local/bin/ld: gcp.c:(.text+0xb7): undefined reference to vdb_mbedtls_entropy_func' /usr/local/bin/ld: gcp.c:(.text+0xc9): undefined reference to vdb_mbedtls_ctr_drbg_seed'
/usr/local/bin/ld: gcp.c:(.text+0xde): undefined reference to vdb_mbedtls_pk_init' /usr/local/bin/ld: gcp.c:(.text+0xfb): undefined reference to vdb_mbedtls_pk_parse_key'

@outpaddling
Copy link

outpaddling commented Jun 19, 2021

Looks like the mbedtls code is not being built in ncbi_vdb, even though configure is finding the library:

<<<ROOT@coral.acadix>>> /usr/ports/biology/ncbi-vdb 1083 # find work/ -name '*.a' -print -exec objdump -t '{}' \; | fgrep vdb_mbedtls_
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_base64_encode
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_md_hmac
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_md_info_from_type
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_ctr_drbg_free
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_ctr_drbg_init
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_ctr_drbg_random
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_ctr_drbg_seed
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_entropy_free
0000000000000000         *UND*	0000000000000000 vdb_mbedtls_entropy_func
...

===> ncbi-vdb-2.11.0_1 depends on shared library: libmbedtls.so - found (/usr/local/lib/libmbedtls.so)
checking for mbedtls library... yes
configured with: "'--with-xml2-prefix=/usr/local' '--with-hdf5-prefix=/usr/local' '--with-mbedtls-prefix=/usr/local' 'CC=cc' 'CXX=c++' 'TOOLS=clang' '--prefix=/usr/local' '--build=amd64-portbld-freebsd12.2'"

@outpaddling
Copy link

Looks like all the vdb_mbedtls_ functions are defined in the bundled libs/ext/mbedtls in ncbi-vdb.

build/mbedtls-lib-name-replace.sed was used at some point (not during build) to add the vdb_ prefix to all the symbols.

So I'm guessing that the bundled mbedtls is heavily modified and cannot be easily replaced with a separate mbedtls. Andrew?

If that's the case, we'll have to revert the patches that unbundle mbedtls.

@kwrodarmer
Copy link
Contributor

Use the included mbedtls. It has been modified to allow for static inclusion in libvdb without conflicting with other mbedtls linkage (if we included it verbatim, then anyone linking against it in addition to libvdb would get symbol conflicts).

I did not read the trail above with great detail, but why would you unbundle mbedtls? Please just build libvdb the way it was designed. Thanks!

@outpaddling
Copy link

Standard reasons: To get regular security updates and other improvements. Bundled libraries invariably fall behind upstream.

Thanks for verifying. I've already had a successful build with the bundled mbedtls and will post more info once I've cleaned up and tested more thoroughly.

@outpaddling
Copy link

I just committed to the FreeBSD ports tree. The one big issue remaining is how the build interacts with ncbi-vdb. Andrew mentioned that sra-tools requires access to the ncbi-vdb source tree. I got around this by treating ncbi-vdb like a Github submodule, building it as part of the sra-tools port instead of building and installing as a separate port.
Are there plans to fully separate ncbi-vdb so dependent apps can build off a standard install?

@outpaddling
Copy link

I did not read the trail above with great detail, but why would you unbundle mbedtls? Please just build libvdb the way it was designed. Thanks!

A little more about this: In addition to being a good practice for security and reliability, unbundling is actually policy for many mainstream package managers, e.g.

https://fedoraproject.org/wiki/Bundled_Libraries?rd=Packaging:Bundled_Libraries
https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles
https://wiki.gentoo.org/wiki/Why_not_bundle_dependencies
https://docs.freebsd.org/en/books/porters-handbook/special/#bundled-libs
You can add pkgsrc to the list as well. I can't find an official document, but I'm also a pkgsrc committer and it was part of my training.
We can make exceptions if unbundling proves too difficult, but we have to make a sincere effort or our colleagues will throw tomatoes at us.
I'm not sure how much NCBI values inclusion of its tools in package managers like Debian packages, FreeBSD ports, Yum, etc., but making it easier to work with external libraries would make our jobs as package maintainers easier and probably increase availability of NCBI software via these deployment methods.

@klymenko
Copy link
Contributor

We switched to cmake build since you created this issue.

@outpaddling
Copy link

Thanks, does this mean the build should work with separately installed ngs and ncbi-vdb now?

@klymenko
Copy link
Contributor

It means that you probable will have to retry your steps.
ngs is not needed anymore.
Recommended build sequence is:

git clone https://github.com/ncbi/ncbi-vdb.git
git clone https://github.com/ncbi/sra-tools.git
cd ncbi-vdb && ./configure && make
cd ../sra-tools && ./configure && make

@outpaddling
Copy link

My question is, does sra-tools still require access to the ncbi-vdb source tree during build? Or can we build and install ncbi-vdb and point the sra-tools build to the installation prefix?
The former is problematic for building within a package manager, but I can make it happen as I did with 2.x.

@outpaddling
Copy link

Also, you stated earlier that you switched to cmake, but your recommended build sequence above uses the old configure scripts. Do you recommend one method over the other? I built ncbi-vdb successfully using standard cmake commands.

@klymenko
Copy link
Contributor

klymenko commented Feb 3, 2023

sra-tools should not require ncbi-vdb source tree.

You can configure/make or build using cmake commands.
There is no preference.

@outpaddling
Copy link

OK, I should be able to take it from here. This can probably be closed now. I don't have the authority since Yuri opened it.

@klymenko klymenko closed this as completed Feb 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants