Possible incompatibility with macOS

[AndreFCruz]

Description

When importing fairgbm on a mac I immediately get the following error message:

OSError: dlopen(<env-dir>/python3.9/site-packages/fairgbm/lib_lightgbm.so, 0x0006): tried: '<env-dir>/python3.9/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file)

Screenshot:

I have tried this with Python 3.8, 3.9, and 3.10 (I haven't tried it with earlier Python versions due to incompatibility with arm64 CPUs, but I have no reason to believe the bug would be exclusive to this architecture).

Reproducible example

1. Install fairgbm on a mac
2. Run import fairgbm
3. See error message.

Environment info

• fairgbm==0.9.13
• macOS
• arm64 (M1 Max)

Additional Comments

It also does not work when using a linux python3.9 environment via docker containers (on macOS).
With this setup, the following (different but related to the same file) error is shown:

OSError: /usr/local/lib/python3.9/site-packages/fairgbm/lib_lightgbm.so: cannot open shared object file: No such file or directory

[joaoleveiga]

Hey @AndreFCruz :

1. Does it work on non-M1 macs?
2. Can you import lightgbm with the same version being used here in your M1?

Cheers 😊

Btw you mention Python 3.9 and above, but the screenshot is showing Python 3.8

[AndreFCruz]

Thanks for the reply Veiga 😄

1. I just tested it and it does not work on my older Intel mac. It crashes with the same exact error:
   

The above screenshot is on a clean Python 3.7 environment, which should be the most stable version (but with 3.9 we get the same error on Intel as well).

2. Good point, I just tried it and I can't even pip install lightgbm on M1 (neither the older 3.2.1 nor the current version), but I can pip install fairgbm (but it crashes later on import).

However, the exception thrown when doing pip install lightgbm on M1 is different than the bug reported above, and I think it's a lightgbm bug. The general incompatibility with macs doesn't seem to be related to this one M1 thing, as I can install lightgbm fine on my Intel mac.

My current theory is that the Microsoft LightGBM setup builds the .so library for the current platform, and our pip package somehow has a pre-compiled library for linux only (fairgbm also installs much much faster than lightgbm). The exception is macOS asking for the mac-equivalent mach-o file.

---

The exception thrown when running pip install lightgbm on M1 is different and this one is likely due to the arm architecture:

Assertion failed: (_file->_atomsArrayCount == computedAtomCount && "more atoms allocated than expected")

(but one bug at a time; the first one definitely affects all macOS platforms).

PS: w.r.t. the first comment: I tested it on 3.8 and above, the repeated 3.9 was a typo.

[AndreFCruz]

In the meantime I commented the tags for compatibility with MacOS and Windows from PyPI and linked this issue in the respective code block.

Plus added a line to the readme that currently only Linux compatibility is maintained.

[tmcarvalho]

Hi @AndreFCruz ! I'm having the same issue :/ have you solved it in the meantime?

[AndreFCruz]

Hi @tmcarvalho

Are you able to install and use the standard lightgbm package? (should be possible on intel-based macs)

If so, then you can install fairgbm directly from source instead of using pip install. Instructions are the same as with lightgbm (but using this repository as the source): https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-from-github

If you're using an arm mac, then (at least in my case) lightgbm can't be installed, which means fairgbm can't be installed either unfortunately.

Another option is getting access to a linux machine, as pip install fairgbm will work fine then.

[eustomaqua]

I encounter the same issue, and weirdly, lightgbm works well but fairgbm does not. Here is my error info

$ python
Python 3.8.17 | packaged by conda-forge | (default, Jun 16 2023, 07:11:32)
[Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm
>>> import fairgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset, register_logger
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 95, in <module>
    _LIB = _load_lib()
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 86, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so, 0x0006): tried: '/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (no such file), '/Users/elise/Software/miniconda3/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so' (not a mach-o file)
>>>

My operation system is macOS Ventura 13.4, and I have tried

brew install libomp
brew install open-mpi

Everything I found on the Internet is for lightgbm, and nearly nothing is relevent to the similar issue of fairgbm. I also tried deleted the previous conda environment and created new ones, but it didn't work. So I was wondering if anyone may face the same issue or happen to know what I should do next to fix this issue? Thanks in advance

[AndreFCruz]

Hi @eustomaqua

Unfortunately compatibility with macOS is not maintained. Please try to use a linux machine.
For example, Google Colab is free and works fine with fairgbm.

Here's an example colab notebook that trains a few FairGBM models with hyperparameter-tuning: https://colab.research.google.com/drive/1OElgST6yLpcgsC2i4BE_dJxVny_bLN6F?usp=sharing

[eustomaqua]

@AndreFCruz Thanks for your reply!

I actually tried using docker (continuumio/anaconda3) as well, but it didn't work

(envs) root@addc1df49268:~# python
Python 3.8.18 (default, Sep 11 2023, 13:19:25)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm
>>> import fairgbm
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset, register_logger
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 95, in <module>
    _LIB = _load_lib()
  File "/opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/basic.py", line 86, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/opt/conda/envs/ensem/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/opt/conda/envs/ensem/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /opt/conda/envs/ensem/lib/python3.8/site-packages/fairgbm/lib_lightgbm.so: cannot open shared object file: No such file or directory
>>>
[1]+  Stopped                 python

So I was wondering perhaps using docker was still somehow dependent on the local operation system? But I didn't figure it out yet.

And thank you for sharing the colab example with me, but I'm using fairgbm as a baseline method for my project, so I feel it would be more convenient for me to use it on our server/local environment.

[eustomaqua]

I built a docker image myself and it works now. Here is my Dockerfile

FROM --platform=linux/amd64 ubuntu:latest

RUN apt-get update && apt-get install -y wget
RUN apt-get update && apt-get install vim -y

RUN wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
RUN mv Miniconda3-latest-Linux-x86_64.sh /root/

Then run this built image and install miniconda3 and other packages including fairgbm, and it works

[AndreFCruz]

Thanks @eustomaqua !

I'll check it out and add it as an option for non-linux users.

[AndreFCruz]

This dockerfile should work for non-linux users: https://github.com/AndreFCruz/fairgbm-fork/blob/add-colab-example/examples/FairGBM-python-notebooks/Dockerfile

Image is public and can be ran with docker run -p 8888:8888 ndrcrz/fairgbm-miniconda

[AndreFCruz]

PR #46 provided a docker example on how to run fairgbm on macOS