Boost.Python converter problem for Python 3.8.10 h3a889e0_1_cpython build for osx-64

[bkpoon]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

For the h3a889e0_1_cpython build of Python 3.8.10 for osx-64, the package I maintain has a problem with one of our conversions between Python and C++ using Boost.Python. To reproduce the issue on an Intel Mac,

1. conda create -n test cctbx-base python=3.8.10=h3a889e0_1_cpython
2. Run these commands in an interactive shell

conda activate test
python
>>> from scitbx.array_family import flex
>>> a = flex.complex_double([None])
>>> a[0]
(4e-323+0j)
>>>

What should happen is a TypeError because None should not be converted into a number for constructing a complex number. Sometimes, the value is (5e-323+0j). Generally, the argument is a list of numbers (e.g. [1, 2, 3j, 4+5j]). I have tested this on macOS 10.14.6, 10.15.7, and 11.4. They all show the same behavior.

To get the expected behavior, you can change the Python to the earlier build by running,

1. conda activate test
2. conda install python=3.8.10=h0e5c897_0_cpython
3. Run the same commands in an interactive shell. The output should look like

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: No registered converter was able to produce a C++ rvalue of type std::__1::complex<double> from this Python object of type NoneType

So I thought the problem might be that the Boost.Python package is not compiled the same way as the latest Python 3.8.10 with macOS SDK 11.0 and linking to CoreFoundation. But I have rebuilt boost and cctbx-base with the latest SDK, with and without linking to CoreFoundation, but the problematic behavior persists for Python 3.8.10 (h3a889e0_1_cpython).

The normal behavior occurs in the osx-arm64 builds, which was another reason why I though boost was the issue, but Python 3.9.4 is fine even though it is built with macOS SDK 11 and links to CoreFoundation while boost is built with macOS SDK 10.9 (from looking at the build logs, but I could be misinterpreting the output).

Lastly, this different behavior is only for Python 3.8.10 (h3a889e0_1_cpython) on osx-64. The other platforms and other versions of Python behave properly. I am running nightly tests here (status badges at the bottom of the README).

The issue is also discussed here
conda-forge/cctbx-base-feedstock#22

Any suggestions for debugging are welcome. Thanks!

Environment (conda list):

$ conda list

# packages in environment at /Users/builder/software/miniconda3/envs/test:
#
# Name                    Version                   Build  Channel
biopython                 1.78             py38h96a0964_2    conda-forge
boost                     1.74.0           py38h692b87f_3    conda-forge
boost-cpp                 1.74.0               hff03dee_4    conda-forge
brotlipy                  0.7.0           py38h5406a74_1001    conda-forge
bzip2                     1.0.8                h0d85af4_4    conda-forge
ca-certificates           2021.5.30            h033912b_0    conda-forge
cctbx-base                2021.5           py38hfaf3399_0    conda-forge
certifi                   2021.5.30        py38h50d1736_0    conda-forge
cffi                      1.14.5           py38ha97d567_0    conda-forge
chardet                   4.0.0            py38h50d1736_1    conda-forge
cryptography              3.4.7            py38h1fa4640_0    conda-forge
cycler                    0.10.0                     py_2    conda-forge
freetype                  2.10.4               h4cff582_1    conda-forge
future                    0.18.2           py38h50d1736_3    conda-forge
icu                       68.1                 h74dc148_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
jbig                      2.1               h0d85af4_2003    conda-forge
jpeg                      9d                   hbcb3906_0    conda-forge
kiwisolver                1.3.1            py38hd9c93a9_1    conda-forge
lcms2                     2.12                 h577c468_0    conda-forge
lerc                      2.2.1                h046ec9c_0    conda-forge
libblas                   3.9.0                9_openblas    conda-forge
libcblas                  3.9.0                9_openblas    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libdeflate                1.7                  h35c211d_5    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
libgfortran               5.0.0           9_3_0_h6c81a4c_22    conda-forge
libgfortran5              9.3.0               h6c81a4c_22    conda-forge
liblapack                 3.9.0                9_openblas    conda-forge
libopenblas               0.3.15          openmp_h5e1b9a4_1    conda-forge
libpng                    1.6.37               h7cec526_2    conda-forge
libsvm                    325                  he49afe7_0    conda-forge
libtiff                   4.3.0                h1167814_1    conda-forge
libwebp-base              1.2.0                h0d85af4_2    conda-forge
llvm-openmp               11.1.0               hda6cdc1_1    conda-forge
lz4-c                     1.9.3                h046ec9c_0    conda-forge
matplotlib-base           3.4.2            py38h6152e83_0    conda-forge
mrcfile                   1.3.0              pyh44b312d_0    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
numpy                     1.20.3           py38had91d27_1    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                h6e7aa92_1    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
pillow                    8.2.0            py38h83525de_1    conda-forge
pip                       21.1.2             pyhd8ed1ab_0    conda-forge
psutil                    5.8.0            py38h96a0964_1    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysocks                   1.7.1            py38h50d1736_3    conda-forge
python                    3.8.10          h3a889e0_1_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python.app                1.3              py38h5406a74_4    conda-forge
python_abi                3.8                      1_cp38    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
reportlab                 3.5.67           py38hf6ac518_0    conda-forge
requests                  2.25.1             pyhd3deb0d_0    conda-forge
setuptools                49.6.0           py38h50d1736_3    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.35.5               h44b9ce1_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
tornado                   6.1              py38h5406a74_1    conda-forge
urllib3                   1.26.5             pyhd8ed1ab_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge
zstd                      1.5.0                h582d3a0_0    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : None
            shell level : 0
       user config file : /Users/builder/.condarc
 populated config files : /Users/builder/.condarc
          conda version : 4.10.1
    conda-build version : not installed
         python version : 3.8.10.final.0
       virtual packages : __osx=10.14.6=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /Users/builder/software/miniconda3  (writable)
      conda av data dir : /Users/builder/software/miniconda3/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/cctbx/osx-64
                          https://conda.anaconda.org/cctbx/noarch
                          https://conda.anaconda.org/phenix-project/osx-64
                          https://conda.anaconda.org/phenix-project/noarch
                          https://conda.anaconda.org/ambermd/osx-64
                          https://conda.anaconda.org/ambermd/noarch
          package cache : /Users/builder/software/miniconda3/pkgs
                          /Users/builder/.conda/pkgs
       envs directories : /Users/builder/software/miniconda3/envs
                          /Users/builder/.conda/envs
               platform : osx-64
             user-agent : conda/4.10.1 requests/2.25.1 CPython/3.8.10 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 502:20
             netrc file : None
           offline mode : False

[bkpoon]

Sorry for the late update. This ended up being a Python 3.8 issue as found by @isuruf. Building Python 3.8 with the newest macOS SDK is not supported when targeting an older version of macOS.

conda-forge/python-feedstock#479
python/cpython#26474