Problem using Python firebird-driver with either Intel or M1 Mac builds with version 4.0.3 or 5.0+

[pfrantz]

When i try and build version 5.0 of firebird on a mac (either intel or m1 based), everything works fine. I can also make the installer and install everything fine on either style of mac. I can use all the tools like isql and can connect via database tools like datagrip (which uses jdbc). The only problem is when i try and access the database from python using firebird-driver everything dies.

There are 2 problems. These are:-

1. the firebird driver can't find the libfbclient.dylib file, but this is easy to fix by doing the following:-

driver_config.fb_client_library.value = "/Library/Frameworks/Firebird.framework/Resources/lib/libfbclient.dylib"

2. when you try to connect to a database or do any operation on the database it fails with the following

firebird.driver.types.DatabaseError: Error loading plugin Engine13
-Module /Library/Frameworks/Firebird.framework/Versions/A/Resources/plugins/libEngine13.dylib exists but can not be loaded
-dlopen(/Library/Frameworks/Firebird.framework/Versions/A/Resources/plugins/libEngine13.dylib, 0x0001): symbol not found in flat namespace '_fb_cancel_operation'
None

This is due to the firebird driver connecting to the libfbclient.dylib and making direct calls to the exported functions. It appears that all the functions are exported in the libfbclient.dylib (via the nm -gC tool), but then in libEngine13.dylib none of these symbols are exported. nm -gC looks like this for libEngine13.dylib:

                 U _fb_cancel_operation
                 U _fb_database_crypt_callback
                 U _fb_dsql_set_timeout
                 U _fb_get_master_interface
                 U _fb_interpret
                 U _fb_print_blr
                 U _fb_shutdown
                 U _fb_shutdown_callback
                 U _fb_sqlstate
                .....
                 U _find_cipher
                 U _find_hash
00000000001deba0 T _firebird_plugin
                 U _flock
                 U _fopen
                ....

I am assuming that libfbclient.dylib calls the libEngine13.dylib but the functions aren't exported. Is this something that is broken in the mac build ? I am assuming people using firebird with python on other platforms.

To build the firebird I follow the readme.build.macosx doco and everything builds successfully, tests run fine, installation is fine.

[aafemt]

Your assumption is wrong. It is libEngine13.dylib is trying to load libfbclient.dylib and cannot find it. Perhaps your solution for problem 1 is incorrect and you should make libfbclient.dylib searchable by dlopen. May be this can help: https://apple.stackexchange.com/questions/376227/how-to-see-library-search-paths-macos

[asfernandes]

Can you try exporting DYLD_LIBRARY_PATH to your $FIREBIRD/lib directory?

[pfrantz]

Tried pretty much everything, but nothing seems to work, even put all the lib files in the same directory. I am either doing something very wrong or firebird 4+ / python / mac combos just don't work. Has anyone successfully built a version of firebird on the mac and tried to access it via python ?

[asfernandes]

Try to log some debug information with this running before the application:

export DYLD_PRINT_LIBRARIES="1"
export DYLD_PRINT_LIBRARIES_POST_LAUNCH="1"
export DYLD_PRINT_APIS="1"
export DYLD_PRINT_INITIALIZERS="1"
export DYLD_PRINT_RPATHS="1"

[pfrantz]

Thanks for that advice - found out there was a incorrect symbolic link which was fixed by:-
'''
sudo ln -s Versions/Current/Libraries/libfbclient.dylib Firebird
'''
after this it finds the correct libfbclient.dylib file but still fails with the libEngine13 call. I don't know where this is called at all - i don't think its in the python code or firebird driver but not sure.

i have attached the debug logs of the load libraries
trace.txt

[mrotteveel]

On 08-11-2023 08:07, Phillip Frantz wrote: Thanks for that advice - found out there was a incorrect symbolic link which was fixed by:- ''' sudo ln -s Versions/Current/Libraries/libfbclient.dylib Firebird ''' after this it finds the correct libfbclient.dylib file but still fails with the libEngine13 call. I don't know where this is called at all - i don't think its in the python code or firebird driver but not sure.

Depending on the connection string used, fbclient will try and create an embedded connection instead of connecting to a server process. Embedded connections use libEngine13, as that is the actual database engine. Mark -- Mark Rotteveel

[asfernandes]

Looks like this line https://github.com/FirebirdSQL/python3-driver/blob/1aaa016468e7ef3eb72616340c3682a706c84b36/src/firebird/driver/fbapi.py#L2045 (or maybe this https://github.com/FirebirdSQL/python3-driver/blob/1aaa016468e7ef3eb72616340c3682a706c84b36/src/firebird/driver/fbapi.py#L2047) need to be changed to pass ctypes.RTLD_GLOBAL.

Can you try that?

[pfrantz]

On 08-11-2023 08:07, Phillip Frantz wrote: Thanks for that advice - found out there was a incorrect symbolic link which was fixed by:- ''' sudo ln -s Versions/Current/Libraries/libfbclient.dylib Firebird ''' after this it finds the correct libfbclient.dylib file but still fails with the libEngine13 call. I don't know where this is called at all - i don't think its in the python code or firebird driver but not sure.
Depending on the connection string used, fbclient will try and create an embedded connection instead of connecting to a server process. Embedded connections use libEngine13, as that is the actual database engine. Mark -- Mark Rotteveel

Thank you @mrotteveel - that helped a lot. If i specify inet and specify the full path of the database, it defaults to tcpip mode and everything works as expected - yay!! I still think embedded mode is broken for the mac.

So really the first problem of not finding the libfbclient is with the installer, as it does not create the firebird link correctly while the second problem is still the undefined symbols in libEngine.

This gets past my problem for now but will still help diagnose this problem if you all want me too.

[asfernandes]

And maybe we should remove usage of -flat_namespace in Firebird linkage.

[pfrantz]

Looks like this line https://github.com/FirebirdSQL/python3-driver/blob/1aaa016468e7ef3eb72616340c3682a706c84b36/src/firebird/driver/fbapi.py#L2045 (or maybe this https://github.com/FirebirdSQL/python3-driver/blob/1aaa016468e7ef3eb72616340c3682a706c84b36/src/firebird/driver/fbapi.py#L2047) need to be changed to pass ctypes.RTLD_GLOBAL.

Can you try that?

Yes that works !!!! Thank you so much @asfernandes - it looks like the python firebird driver has the bug. I will raise an issue with that library maintainers of it. So to fix needed to :-

        if sys.platform in ('win32', 'cygwin', 'os2', 'os2emx'):
            self.client_library: ctypes.CDLL = ctypes.WinDLL(str(filename))
        else:
            self.client_library: ctypes.CDLL = ctypes.CDLL(str(filename), ctypes.RTLD_GLOBAL)

interesting , the python doco says GLOBAL is the default for macos 10.3, but we are upto 13.x

thanks again

[pfrantz]

And maybe we should remove usage of -flat_namespace in Firebird linkage.

dunno - most libs are linked using the flat namespace but i am by no means an expert

[AlexPeshkoff]

On 11/8/23 13:25, Phillip Frantz wrote: So really the first problem of not finding the libfbclient is with the installer, as it does not create the firebird link correctly while the second problem is still the undefined symbols in libEngine.

I doubt there is something actually undefined. For network server to work absolutely same fbclient is loading absolutely same engine13, just all that happens not in your python process but in network server process. So some difference in environment, etc. to be found here.

[pfrantz]

On 11/8/23 13:25, Phillip Frantz wrote: So really the first problem of not finding the libfbclient is with the installer, as it does not create the firebird link correctly while the second problem is still the undefined symbols in libEngine.
I doubt there is something actually undefined. For network server to work absolutely same fbclient is loading absolutely same engine13, just all that happens not in your python process but in network server process. So some difference in environment, etc. to be found here.

i am wrong here about the undefined symbol - you fixed this problem via adding the RTLD_GLOBAL flag to the ctypes.cdll call. That was the problem. It also exists in the fdb library. So only one thing to fix in the install script and another tweak to the python library. I will try the same fixes on my m1 machine and work tomorrow.

[asfernandes]

sudo ln -s Versions/Current/Libraries/libfbclient.dylib Firebird

I pushed this fix in v5. It will appear https://github.com/FirebirdSQL/firebird/actions/runs/6797205572 as firebird-macos artifact once it is built.

Can you test it?

[pfrantz]

Sure - i will give it a test tomorrow at work. (its getting late here in Aus) - have a good day.

[asfernandes]

And in https://github.com/FirebirdSQL/firebird/actions/runs/6797262441, a version without -flat_namespace, to be tested with unmodified firebird-python.

[pfrantz]

And in https://github.com/FirebirdSQL/firebird/actions/runs/6797262441, a version without -flat_namespace, to be tested with unmodified firebird-python.

HI @asfernandes , Have tried it on my intel mac and all works well. The firebird driver works without modification. The library is found by python and all is good. On Monday i will try out on my m1 mac at work (i will build from your tag).

There is one more enhancement that would be good to put in the installer but not sure if its possible. Right now the firebird client lib (or main process) seems to create /tmp/firebird directory that the current mac user has no permission to access as the directory permission is limited to the firebird group. This causes a failure when trying to open a database. This can be easily fixed by the following command:-

sudo dseditgroup -o edit -a {current usename goes here} -t user firebird

would this command be able to be added to the installer if appropriate ?

cheers,

Phillip

[asfernandes]

Thanks for testing.

@AlexPeshkoff can better confirm it, but AFAIK the same happens in Linux and we do not want to make an installed server insecure by default. Adding users to Firebird group for embedded usage should be done manually.

[pfrantz]

Thanks for testing.

@AlexPeshkoff can better confirm it, but AFAIK the same happens in Linux and we do not want to make an installed server insecure by default. Adding users to Firebird group for embedded usage should be done manually.

Thats fair enough - i will just put this in the developer notes for us.

Cheers,

Phillip

[pfrantz]

And in https://github.com/FirebirdSQL/firebird/actions/runs/6797262441, a version without -flat_namespace, to be tested with unmodified firebird-python.

I just built this commit using the M1 mac and everything works fine. I think we can close this ticket whenever you want now.

[asfernandes]

And in https://github.com/FirebirdSQL/firebird/actions/runs/6797262441, a version without -flat_namespace, to be tested with unmodified firebird-python.

I just built this commit using the M1 mac and everything works fine. I think we can close this ticket whenever you want now.

I will go with that (flat_namespace) then if others (@AlexPeshkoff) do not see a problem.

Anyway, a change in the firebird-driver would be welcome to load previous versions.

[pcisar]

Fix for firebird-driver was committed.