UnixODBC+FreeTDS broken on FR3 on RHEL7

[bluikko]

radiusd -X gives error when trying to use rlm_sql_unixodbc:

rlm_sql_unixodbc: 01000 [unixODBC][Driver Manager]Can't open lib '/usr/lib64/libtdsodbc.so.0' : ??????: undefined symbol: get_vtable

This is NOT ODBC/TDS problem with libraries or so, since I can connect to the database with the "isql" tool just fine.

CentOS 7, tested with following UnixODBC and FreeTDS versions:

• UnixODBC: 2.3.4 and
• FreeTDS 0.95.19 and 0.91
  With RHEL7 standard FreeRADIUS 3.0.4

Many people have raised this issue on various mailing lists over the past few years and nobody seems to have an answer!

[alandekok]

There's nothing in the server which uses the "get_vtable" API. So the problem is in the libraries.

My guess is that the libraries suddenly require an additional library to be linked to the application. Or, the libraries are broken.

A simple google search yields this page:

http://fossies.org/dox/unixODBC-2.3.4/shl__load_8c.html

Which says:

#define     get_vtable   shl_load_LTX_get_vtable

So it's a library problem. The libraries were built such that they didn't define get_vtable correctly.

Please file a bug with CentOS and/or RedHat. Ask them to supply libraries which aren't broken.

There is nothing we can do to FreeRADIUS which will fix this problem.

[bluikko]

Thank you!
I saw you discussed this repeatedly in the mailing list but did not see this information. I will try to find out more and it is interesting nobody else has reported this outside of the mailing list posts.

[gustavoheredia]

Same issue in Ubuntu 14.04 with Freeradius 3.1.0

rlm_sql (sql): Opening additional connection (0), 1 of 32 pending slots used
rlm_sql_unixodbc: 01000 [unixODBC][Driver Manager]Can't open lib '/usr/lib/odbc/libtdsodbc.so' : ?F?!??: undefined symbol: get_vtable

[bluikko]

Can you please give more information - what version of UnixODBC? Seems you use FreeTDS with it, what version? Also how did you install UnixODBC/FreeTDS, compile from source or install from a repository?

[gustavoheredia]

UnixODBC 2.2.14p2-5ubuntu5 and FreeTDS 0.91-5 installed from a repository . With this version of UnixODBC and FreeTDS in the same machine works without issue FreeRadius 2.1.12 with rlm_sql_unixodbc

[arr2036]

Check the linkages on 2.1.12 and 3.0.x, see how they differ?

[bluikko]

That (gustavoheredia) does sound like a problem in FreeRADIUS or problem in build of FreeRADIUS...

edit: removed broken link

[gustavoheredia]

bluiko i can`t see the link that you mentioned (Page not found)

[bluikko]

Thanks, fixed

[arr2036]

What was the conclusion with this?

[arr2036]

Apparently nothing, i'll close the issue then...

[bluikko]

No conclusion, your closing of the bug seems to be the only reply from FR side?

[arr2036]

arr2036 commented on Oct 7
Check the linkages on 2.1.12 and 3.0.x, see how they differ?

arr2036 commented 26 days ago
What was the conclusion with this?

[arr2036]

Did we receive any linkage information? no. If you're not willing to help diagnose this issue, i'm not going to keep it hanging around in the bug tracker.

[bluikko]

I can do troubleshooting but I am not a developer. If there are some specific commands I can run to help, please list them. Otherwise I understand closing the bug if nobody would work on it anyways.

[arr2036]

Run ldd <path to rlm_unix_odbc.so> on the two systems, and compare/post the output.

[bluikko]

Not working:

$ cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)
$ rpm -qa |fgrep -i radius
freeradius-unixODBC-3.0.4-6.el7.x86_64
freeradius-3.0.4-6.el7.x86_64
$ rpm -qa |fgrep -i odbc
unixODBC-2.3.1-10.el7.x86_64
freeradius-unixODBC-3.0.4-6.el7.x86_64
$ rpm -qa |fgrep -i tds
freetds-0.91-12.git0a42888.el7.x86_64
$ ldd /usr/lib64/freeradius/rlm_sql_unixodbc.so
linux-vdso.so.1 => (0x00007ffe75f59000)
libodbc.so.2 => /usr/lib64/libodbc.so.2 (0x00007f9dbb41b000)
libc.so.6 => /usr/lib64/libc.so.6 (0x00007f9dbb05a000)
libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00007f9dbae4f000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f9dbac4b000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f9dbaa2f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f9dbb888000)

Working:

$ cat /etc/centos-release
CentOS release 6.7 (Final)
$ rpm -qa |fgrep -i radius
freeradius-2.2.6-6.el6_7.x86_64
freeradius-utils-2.2.6-6.el6_7.x86_64
freeradius-unixODBC-2.2.6-6.el6_7.x86_64
$ rpm -qa |fgrep -i odbc
unixODBC-2.2.14-14.el6.x86_64
freeradius-unixODBC-2.2.6-6.el6_7.x86_64
$ rpm -qa |fgrep -i tds
freetds-0.91-2.el6.x86_64
$ ldd /usr/lib64/freeradius/rlm_sql_unixodbc-2.2.6.so
linux-vdso.so.1 => (0x00007ffe42ff6000)
libodbc.so.2 => /usr/lib64/libodbc.so.2 (0x00007f82e8034000)
libc.so.6 => /lib64/libc.so.6 (0x00007f82e7ca0000)
libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00007f82e7a96000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f82e7879000)
/lib64/ld-linux-x86-64.so.2 (0x0000003170800000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f82e7675000)

[gthyni]

https://bugs.centos.org/view.php?id=9746

[bluikko]

Seems to be the solution, so this bug can be closed.

[spbnick]

@alandekok, @arr2036, it appears there's more to this. We did a bit of research for our RHEL customers, and it appears the problem is that FreeRADIUS defines symbols lt_dlopenext, lt_dlsym, lt_dlclose, and lt_dlerror, which are also defined by libltdl, which is used by libodbc.

As a result, when rlm_sql_unixodbc loads libodbc, which in turn loads libltdl, it gets to use the FreeRADIUS version of those symbols, not the ones from libltdl. From this chaos ensues, including linker data corruption, as seen in those question marks in the output in the first message here.

My question is: does anything else use these, except the explicit uses in FreeRADIUS code, and can we perhaps rename these to avoid the clash? If yes, then I'll do the patch. If no, then what else uses this and what options do we have? Thank you.

[spbnick]

You can reproduce the problem with this command:

LD_PRELOAD=/usr/sbin/radiusd isql -v MSSQLTestServer

Provided you added this to /etc/odbcinst.ini:

[FreeTDS]
Description=v0.63 with protocol v8.0
Driver=/usr/lib64/libtdsodbc.so.0
UsageCount=1

And put this into /etc/odbc.ini:

[MSSQLTestServer]
Driver    = FreeTDS
Description = Sample database
Trace     = No
Server    = 127.0.0.1
Port      = 1433
Database  = Sample

Here is an excerpt from LD_DEBUG=all output:

  2324:     symbol=lt_dlsym;  lookup in file=isql [0]
  2324:     symbol=lt_dlsym;  lookup in file=/usr/sbin/radiusd [0]
  2324:     binding file /lib64/libltdl.so.7 [0] to /usr/sbin/radiusd [0]: normal symbol `lt_dlsym'

[arr2036]

Ah, that's what's going on. I don't think they're used, just vestigial @alandekok?

[alandekok]

They're used for the local dlopen wrappers. They could just as well be named fr_foo instead of lt_foo.

[spbnick]

Alright, thanks Alan, Arran. I'll do a patch for v3.0.x renaming them to fr_* today or tomorrow, if you don't do it first.

[spbnick]

Thanks, Alan!

[praiskup]

Thanks! Just a suggestion, it is always worth limiting the number of "exported symbols" as much as possible, so you avoid such linking issues in future. IOW, it is very rare case some symbol is going to be used (from binary), for example [1]. But again, thank you -> changing name-space should work for us.

[1] https://blog.flameeyes.eu/2008/02/why-would-an-executable-export-symbols/

[arr2036]

The symbols aren't exported in the radiusd bindary. They're exported in libfreeradius-server, which modules link to and is linked to by the radiusd binary.

[praiskup]

That would mean the library is linked statically? But anyways:

$ readelf -s /usr/sbin/radiusd  | grep lt_dl
   611: 0000000000024550     5 FUNC    GLOBAL DEFAULT   13 lt_dlsym
   697: 0000000000024560    19 FUNC    GLOBAL DEFAULT   13 lt_dlclose
   753: 0000000000023da0   764 FUNC    GLOBAL DEFAULT   13 lt_dlopenext
   757: 0000000000024580     5 FUNC    GLOBAL DEFAULT   13 lt_dlerror

While

$ readelf -s /usr/lib64/freeradius/libfreeradius-server.so | grep lt_dl
[empty]

[arr2036]

It changed in v3.1.x/v4.0.x.

[praiskup]

Great, thanks for the info!