XGBoost and MacOS incompatibility

[NeoChaos12]

This issue has been reported by multiple users when they attempted to install and use JAHS-Bench-201 on MacOS. To the best of my knowledge, this boils down to an incompatibility between XGBoost and MacOS, but it needs to be investigated further and should be addressed somehow at the earliest convenience in order to enable Mac users to also use the benchmark.

The following is an error log that was sent to me in response to the installation instructions in our ReadMe:

WARNING: /Users/runner/work/xgboost/xgboost/src/gbm/gbtree.cc:386: Loading from a raw memory buffer on CPU only machine.  Changing tree_method to hist.
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/jahs_bench_examples/minimal.py", line 13, in <module>
    run()
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/jahs_bench_examples/minimal.py", line 5, in run
    benchmark = jahs_bench.Benchmark(task="cifar10", kind="surrogate", download=True)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/jahs_bench/api.py", line 96, in __init__
    loaders[kind]()
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/jahs_bench/api.py", line 106, in _load_surrogate
    self._surrogates[o] = XGBSurrogate.load(model_path / str(o))
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/jahs_bench/surrogate/model.py", line 462, in load
    model = joblib.load(outdir / cls.__model_filename)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 587, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 506, in _unpickle
    obj = unpickler.load()
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/pickle.py", line 1088, in load
    dispatch[key[0]](self)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 331, in load_build
    Unpickler.load_build(self)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/pickle.py", line 1552, in load_build
    setstate(state)
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/xgboost/core.py", line 1452, in __setstate__
    _LIB.XGBoosterUnserializeFromBuffer(handle, ptr, length))
  File "/usr/local/Caskroom/miniconda/base/envs/meta_neps_02/lib/python3.7/site-packages/xgboost/core.py", line 218, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [10:43:09] /Users/runner/work/xgboost/xgboost/src/tree/tree_updater.cc:20: Unknown tree updater grow_gpu_hist
Stack trace:
  [bt] (0) 1   libxgboost.dylib                    0x00000001230ce4a4 dmlc::LogMessageFatal::~LogMessageFatal() + 116
  [bt] (1) 2   libxgboost.dylib                    0x0000000123217f39 xgboost::TreeUpdater::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, xgboost::GenericParameter const*) + 729
  [bt] (2) 3   libxgboost.dylib                    0x0000000123159d8a xgboost::gbm::GBTree::LoadConfig(xgboost::Json const&) + 2634
  [bt] (3) 4   libxgboost.dylib                    0x0000000123178246 xgboost::LearnerConfiguration::LoadConfig(xgboost::Json const&) + 742
  [bt] (4) 5   libxgboost.dylib                    0x0000000123179827 xgboost::LearnerIO::Load(dmlc::Stream*) + 743
  [bt] (5) 6   libxgboost.dylib                    0x00000001230c9101 XGBoosterUnserializeFromBuffer + 145
  [bt] (6) 7   libffi.6.dylib                      0x000000010196a934 ffi_call_unix64 + 76
  [bt] (7) 8   ???                                 0x00007ff7beb9eb80 0x0 + 140702033505152

[sdaulton]

+1. I also was unable to install on MacOS

[nilskober]

Same issue on my machine with MacOS. Here a few details:

MacBook Pro 15" 2018
MacOS 13.0 (Ventura) but happened on MacOS 12.* (Monterey) as well

I created a new conda environment (conda 4.13.0) with Python 3.7.15 (but same error with Python 3.10.) and installed jahsbench with pip install jahs-bench. Running python -m jahs_bench_examples.minimal leads to the following error:

[20:03:24] WARNING: /Users/runner/work/xgboost/xgboost/src/gbm/gbtree.cc:386: Loading from a raw memory buffer on CPU only machine.  Changing tree_method to hist.
Traceback (most recent call last):
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/jahs_bench_examples/minimal.py", line 13, in <module>
    run()
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/jahs_bench_examples/minimal.py", line 5, in run
    benchmark = jahs_bench.Benchmark(task="cifar10", kind="surrogate", download=True)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/jahs_bench/api.py", line 135, in __init__
    loaders[kind]()
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/jahs_bench/api.py", line 148, in _load_surrogate
    self._surrogates[o] = XGBSurrogate.load(pth) if not self._lazy else \
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/jahs_bench/surrogate/model.py", line 462, in load
    model = joblib.load(outdir / cls.__model_filename)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 587, in load
    obj = _unpickle(fobj, filename, mmap_mode)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 506, in _unpickle
    obj = unpickler.load()
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/pickle.py", line 1088, in load
    dispatch[key[0]](self)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/joblib/numpy_pickle.py", line 331, in load_build
    Unpickler.load_build(self)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/pickle.py", line 1552, in load_build
    setstate(state)
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/xgboost/core.py", line 1452, in __setstate__
    _LIB.XGBoosterUnserializeFromBuffer(handle, ptr, length))
  File "/usr/local/Caskroom/miniconda/base/envs/jahs_bench_test/lib/python3.7/site-packages/xgboost/core.py", line 218, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [20:03:24] /Users/runner/work/xgboost/xgboost/src/tree/tree_updater.cc:20: Unknown tree updater grow_gpu_hist
Stack trace:
  [bt] (0) 1   libxgboost.dylib                    0x000000012a9fa4a4 dmlc::LogMessageFatal::~LogMessageFatal() + 116
  [bt] (1) 2   libxgboost.dylib                    0x000000012ab43f39 xgboost::TreeUpdater::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, xgboost::GenericParameter const*) + 729
  [bt] (2) 3   libxgboost.dylib                    0x000000012aa85d8a xgboost::gbm::GBTree::LoadConfig(xgboost::Json const&) + 2634
  [bt] (3) 4   libxgboost.dylib                    0x000000012aaa4246 xgboost::LearnerConfiguration::LoadConfig(xgboost::Json const&) + 742
  [bt] (4) 5   libxgboost.dylib                    0x000000012aaa5827 xgboost::LearnerIO::Load(dmlc::Stream*) + 743
  [bt] (5) 6   libxgboost.dylib                    0x000000012a9f5101 XGBoosterUnserializeFromBuffer + 145
  [bt] (6) 7   libffi.8.dylib                      0x00000001075cda22 ffi_call_unix64 + 82
  [bt] (7) 8   ???                                 0x00007ff7b8e931e0 0x0 + 140701935940064

Details on versions of any packages installed in the conda environment (conda list):

ca-certificates           2022.10.11           hecd8cb5_0  
certifi                   2022.9.24        py37hecd8cb5_0  
charset-normalizer        2.1.1                    pypi_0    pypi
configspace               0.4.21                   pypi_0    pypi
cython                    0.29.32                  pypi_0    pypi
idna                      3.4                      pypi_0    pypi
jahs-bench                1.0.2                    pypi_0    pypi
joblib                    1.1.1                    pypi_0    pypi
libffi                    3.4.2                hecd8cb5_6  
ncurses                   6.3                  hca72f7f_3  
numpy                     1.21.6                   pypi_0    pypi
openssl                   1.1.1s               hca72f7f_0  
pandas                    1.3.5                    pypi_0    pypi
pip                       22.3.1           py37hecd8cb5_0  
pyparsing                 3.0.9                    pypi_0    pypi
python                    3.7.15               h218abb5_1  
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2022.6                   pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  hca72f7f_0  
requests                  2.28.1                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scipy                     1.7.3                    pypi_0    pypi
setuptools                65.5.0           py37hecd8cb5_0  
six                       1.16.0                   pypi_0    pypi
sqlite                    3.40.0               h880c91c_0  
threadpoolctl             3.1.0                    pypi_0    pypi
tk                        8.6.12               h5d9f67b_0  
urllib3                   1.26.13                  pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
xgboost                   1.5.2                    pypi_0    pypi
xz                        5.2.8                h6c40b1e_0  
yacs                      0.1.8                    pypi_0    pypi
zlib                      1.2.13               h4dc903c_0 

[NeoChaos12]

As per the response of the authors of XGBoost, the correct way to handle this issue on our end will require a very significant change to the way surrogate models are saved and loaded, possibly requiring a complete rewrite of the interface between jahs_bench.surrogate.model.XGBSurrogate and sklearn.pipeline.Pipeline. Thus, I will defer this to a future update to the repo.