Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MySQL source wasn't initialized. Wrong name in dlopen? #155

Closed
obikenoby opened this issue Jan 12, 2019 · 11 comments

Comments

@obikenoby
Copy link

commented Jan 12, 2019

Manticore Search version:
2.7.5
OS version:
Debian 9
Build version:

* in case an official prebuild package is used, write down the full name of the package file
* in case it's compiled from sources, you need to provide the debug files and revision of code

Describe the problem

After upgrading MySQL to version 8.0 I get error message:
"ERROR: index 'main': sql_connect: MySQL source wasn't initialized. Wrong name in dlopen? (DSN=mysql://login:***@localhost:3306/database)."

Messsages from log files:
No messages found related to this issue.

@tomatolog

This comment has been minimized.

Copy link
Contributor

commented Jan 12, 2019

could you provide example config (source and index sections) that reproduces this issue?

@adriannuta

This comment has been minimized.

Copy link
Contributor

commented Jan 13, 2019

On Stretch Manticore is built against libmariaclient 18 (the default mysql client library). Upgrading to MySQL 8 must have removed it.
There are 2 options:

  • you can install libmariadbclient18 package. However in this case you need to set MySQL 8 to use mysql_native_password as default authentification plugin as older lib clients don't know to work with the new auth plugin

  • if you want to use MySQL 8 with the default authentification plugin (caching_sha2_password), Manticore needs to be recompiled against libmysqlclient21 ( follow the compiling guide in the docs - install libmysqlclient-dev should also install libmysqlclient21 if you are using the official MySQL apt repository).

@obikenoby

This comment has been minimized.

Copy link
Author

commented Jan 13, 2019

Hi, I resolved it by installing default-libmysqlclient-dev. Installation of libmysqlclient-dev didn't help. It might help to others when this problem emerge.

@obikenoby obikenoby closed this Jan 13, 2019

@sigaev-pro

This comment has been minimized.

Copy link

commented Jan 20, 2019

I have Ubuntu 18.04 and MariaDB 10.3 installed and have the same problem.

using config file './sphinx.conf'...
indexing index 'cars'...
ERROR: index 'cars': sql_connect: MySQL source wasn't initialized. Wrong name in dlopen? (DSN=mysql://ap_nescis_ru:***@localhost:3306/ap_nescis_ru).

and have following connection properties in sphinx.conf

	sql_host		= localhost
	sql_user		= ap_nescis_ru
	sql_pass		= ********************
	sql_db			= ap_nescis_ru
	sql_port		= 3306	# optional, default is 3306

Neither installing default-libmysqlclient-dev, nor libmysqlclient-dev have effect.

I have installed ManticoreSearch from following installation instructions

@klirichek

This comment has been minimized.

Copy link
Contributor

commented Jan 20, 2019

Let me tell a bit about different versions of libs.
When building from sources usually we need 'development' versions of different libraries (as mysql, posgres, etc.). They usually came in special packages suffixed with 'dev' or 'devel'.
Inside apart binary library itself also included some necessary stuff as C headers and m.b. another things. And - essential thing - libraries themselves usually aliased into non-versioned name. Say, if you work with libmysqlclient.so.20 - it will be aliased also as libmysqlclient.so (without '.20'). So that during the build we would not care about specific version, but can rely just on 'libmysqlclient.so', and that is enough.

However different versions may provide different possibilities and even different API and ABI. So, for seamless work version is important, and if an application built for 'v.20' it is not the fact that it will same seamless work with, say, 'v.16'.

On final (user) installation such aliases is no more available, but the application itself instead is linked against right version of the lib and will just not start in case the system lacks it. But such dependency is not a good choice, since, say, if we build with support of mysql, posgres, expat and zoo of other libs that means that we either have to provide also zoo of builds, or our only build will require this zoo of libs despite the fact that you don't want to index all kind of sources (that is simple - link dependency means that the program will just not start without all dependencies).

Instead we provide another approach: exact names of the libs is stored, and then we try to dynamically load necessary one exactly at the moment you try to index the source supported by this lib. That allows us to provide support to all sources in one binary, and at the same time not require any of them till the moment you want to use it.

List of the libraries you can look by running 'indexer' alone, without any params. Perform indexer | grep mysql - and you'll see what exact lib we require. If it looks for 'libmysqlclient.so.20' - then you need this very library being available on your system. So, if you have another one - you can try to substitute it, but take care that substitution library must be the same version (other cases may work, but no guarantee). If you have a kind of 'libmariadbclient.so' - try to exec objdump -p libmariadbclient.so | grep SONAME over it, and it it shows the same name as need by indexer (as mentioned libmysqlclient.so.20) - then you can make a symlink to the lib named the same as required soname. So, it would solve your problem.

(however, if version is different and you still make a symlink - may be it would work, but no guarantee. We are not rely on very-very-modern versions of mysql, so features of v.16 for us is enough and we don't really need possibilities of v.20, but apart feature requests they also may provide different ABIs we can't control)

@manticoresearch

This comment has been minimized.

Copy link
Contributor

commented Jan 21, 2019

@sigaev-pro Manticore works fine for me on a clean Ubuntu 18.04 with a clean Mariadb 10.3 installed:

root@76d53dbbcce5:/# cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
root@76d53dbbcce5:/# indexer --print-queries -c sphinx_min.conf --all
Manticore 2.7.5 4a31c54@181204 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2018, Manticore Software LTD (http://manticoresearch.com)

using config file 'sphinx_min.conf'...
indexing index 'idx_min'...
SQL-CONNECT: ok
SQL-QUERY: select 1, 'dog' doc, 1 group_id, 'red' color, 3 size         union select 2, 'dog' doc, 1 group_id, 'red' color, 8 size         union select 3, 'dog' doc, 2 group_id, 'green' color, 1 size         union select 4, 'dog' doc, 2 group_id, 'green' color, 3 size         union select 5, 'dog' doc, 3 group_id, 'red' color, 9 size         union select 6, 'dog' doc, 4 group_id, 'blue' color, 3 size         union select 7, 'dog > <' doc, 5 group_id, 'red' color, 3 size: ok
SQL-DISCONNECT
collected 7 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 7 docs, 25 bytes
total 0.189 sec, 132 bytes/sec, 37.02 docs/sec
total 7 reads, 0.000 sec, 13.7 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
root@76d53dbbcce5:/#
root@76d53dbbcce5:/# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 55
Server version: 10.3.12-MariaDB-1:10.3.12+maria~bionic-log mariadb.org binary distribution
root@76d53dbbcce5:/# mysql --version
mysql  Ver 15.1 Distrib 10.3.12-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
root@76d53dbbcce5:/# apt list --installed|grep mariadb

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

mariadb-client/unknown,now 1:10.3.12+maria~bionic all [installed]
mariadb-client-10.3/unknown,now 1:10.3.12+maria~bionic amd64 [installed,automatic]
mariadb-client-core-10.3/unknown,now 1:10.3.12+maria~bionic amd64 [installed,automatic]
mariadb-common/unknown,now 1:10.3.12+maria~bionic all [installed,automatic]
mariadb-server/unknown,now 1:10.3.12+maria~bionic all [installed]
mariadb-server-10.3/unknown,now 1:10.3.12+maria~bionic amd64 [installed,automatic]
mariadb-server-core-10.3/unknown,now 1:10.3.12+maria~bionic amd64 [installed,automatic]
root@76d53dbbcce5:/# indexer
Manticore 2.7.5 4a31c54@181204 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2018, Manticore Software LTD (http://manticoresearch.com)

Built by gcc/clang v 7,

Built on Linux 0fb09d7498d2 4.15.0-30-generic #32~16.04.1-Ubuntu SMP Thu Jul 26 20:25:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=bionic -DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DDATADIR=/var/data -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_ODBC=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON -DSYSCONFDIR=/etc/sphinxsearch
root@76d53dbbcce5:/# find .|grep libmysqlclient.so.20
./usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.11
./usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
root@76d53dbbcce5:/# find .|grep libmysqlclient.so
./usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.11
./usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
root@76d53dbbcce5:/# find .|grep libmariadbclient
root@76d53dbbcce5:/#

Do you have libmysqlclient.so.20 ? What's different in your case comparing to the above versions?

@mlb5000

This comment has been minimized.

Copy link

commented Apr 3, 2019

@manticoresearch I am having this problem on Mac, both when installing from binaries as in the documentation, and when building tarball. MySQL is at v8.0.13.

The failing indexer call:

Matthews-MBP-2:sphinxwrapper matt$ indexer --config /Users/matt/sphinx.conf --rotate proj_8_index
Manticore 2.8.1 3684198@190306 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

using config file '/Users/matt/sphinx.conf'...
indexing index 'proj_8_index'...
ERROR: index 'proj_8_index': sql_connect: MySQL source wasn't initialized. Wrong name in dlopen? (DSN=mysql://username:***@localhost:3306/odintext).
total 0 docs, 0 bytes
total 0.000 sec, 0 bytes/sec, 0.00 docs/sec
total 0 reads, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
total 0 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg

The indexer output as it pertains to libraries:

Matthews-MBP-2:sphinxwrapper matt$ indexer
Manticore 2.8.1 3684198@190306 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

Built by gcc/clang v 4.2.1,

Built on Darwin Matthews-MBP-2 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019

Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.dylib -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.dylib -DDATADIR=/usr/local/var/data -DFULL_SHARE_DIR=/usr/local/share/manticore -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_RE2=1 -DWITH_ZLIB=ON

The relevant MySQL portion from make -j4

-- Option WITH_MYSQL 1
-- mysql_config was found /usr/local/mysql/bin/mysql_config
-- MySQL Include dir: /usr/local/mysql/include
-- MySQL Library    : mysqlclient;ssl;crypto
-- MySQL Library dir: /usr/local/mysql/lib
-- MySQL CXXFLAGS: -I/usr/local/mysql/include 
-- MySQL Library Name   : libmysqlclient.dylib
-- MYSQL will be loaded dynamically in runtime as libmysqlclient.dylib
@mlb5000

This comment has been minimized.

Copy link

commented Apr 3, 2019

And as always, simply stating the question gets me 99% the way there. Using the --verbose debugvv flag on indexer, I got the error message DEBUG: dlopen(libmysqlclient.dylib) failed. Which led me to ask if maybe my DYLD path was messed up. Added this to my ~/.bash_profile and now it connects:

export DYLD_LIBRARY_PATH=/usr/local/mysql/lib:$DYLD_LIBRARY_PATH

Note that this matches the path described in my cmake output.

@manticoresearch

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2019

So can we conclude it's a MySQL installation issue? I've checked on my mac. What I have is:

$ ls -la /usr/local/lib/libmysqlclient.dylib
lrwxr-xr-x  1 snikolaev  staff  47 Sep 16  2015 /usr/local/lib/libmysqlclient.dylib -> ../Cellar/mysql/5.6.23/lib/libmysqlclient.dylib

and indexer of both 2.8.1 and 2.8.2 released yesterday works fine out of the box. What it gives is:

$ ./indexer | grep mysql
Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=macos -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.dylib -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.dylib -DDL_PGSQL=1 -DPGSQL_LIB=libpq.dylib -DDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
$ ./indexer -c sphinx_min.conf --all
Manticore 2.8.1 3684198c@190306 release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2019, Manticore Software LTD (http://manticoresearch.com)

using config file 'sphinx_min.conf'...
indexing index 'idx_min'...
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 1 docs, 3 bytes
total 0.189 sec, 15 bytes/sec, 5.28 docs/sec
total 7 reads, 0.000 sec, 13.7 kb/call avg, 0.0 msec/call avg
total 12 writes, 0.000 sec, 0.0 kb/call avg, 0.0 msec/call avg
@mlb5000

This comment has been minimized.

Copy link

commented Apr 3, 2019

@manticoresearch Yes, and no. Specifically is has to do with the ability to actually find libmysqlclient.dylib inside your search paths. Without DYLD_LIBRARY_PATH set and including the MySQL lib directory for your install, it won't be able to find the MySQL library at runtime. That turned out to be the real important part. Probably worth noting in the documentation for Mac users.

@klirichek

This comment has been minimized.

Copy link
Contributor

commented Jun 15, 2019

@mlb5000 if you interested: I've investigated the way how to configure and provide dylib-id (the full path to real library) and implemented it in 4ad8067
Btw, with this approach library used for build is 'sticked' by absolute path (so, instead of just libmysqlclient.dylib it will use in case of homebrew to long /usr/local/opt/mysql-client/lib/libmysqlclient.20.dylib), but it however works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.