Bus Error (core dumped) with anaconda.org/anaconda and anaconda.org/conda-forge

[ericdill]

Hi constructor devs,

I am getting a bus error when using conda.anaconda.org/anaconda and conda.anaconda.org/conda-forge as my channels in a construct.yaml. I have pared down the construct.yaml as far as it can go

name: python
version: 3.5 
channels:
    - https://conda.anaconda.org/conda-forge
    - https://conda.anaconda.org/anaconda
specs:
    - python 3.5*

Running the constructor that is produced by this construct.yaml produces the following error:

$ bash ./python-3.5-Linux-x86_64.sh -b -p /tmp/foo
PREFIX=/tmp/foo
installing: python-3.5.2-4 ...
Python 3.5.2
./python-3.5-Linux-x86_64.sh: line 161: 20122 Bus error               (core dumped) $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT

I tried to do a diff of the installer scripts produced when I remove conda-forge as a channel, but, as expected, that diff is pretty much inscrutable and filled with binary diffs 😞

cc @msarahan

[edit] Running constructor without conda-forge as a channel does work

[ericdill]

oh, right, conda info:

conda info -a

Current conda install:

               platform : linux-64
          conda version : 4.2.13
       conda is private : False
      conda-env version : 4.2.13
    conda-build version : not installed
         python version : 3.5.2.final.0
       requests version : 2.12.4
       root environment : /tmp/mc  (writable)
    default environment : /tmp/mc
       envs directories : /tmp/mc/envs
          package cache : /tmp/mc/pkgs
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://conda.anaconda.org/ericdill/linux-64
                          https://conda.anaconda.org/ericdill/noarch
            config file : /home/edill/.condarc
           offline mode : False

# conda environments:
#
root                  *  /tmp/mc

sys.version: 3.5.2 | packaged by conda-forge | (defau...
sys.prefix: /tmp/mc
sys.executable: /tmp/mc/bin/python
conda location: /tmp/mc/lib/python3.5/site-packages/conda
conda-build: /home/edill/miniconda/bin/conda-build
conda-env: /tmp/mc/bin/conda-env
user site dirs: 

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: <not set>
CONDA_ENVS_PATH: <not set>
LD_LIBRARY_PATH: <not set>
PATH: /tmp/mc/bin:/home/edill/miniconda/bin:/home/edill/bin:/usr/local/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/edill/bin://home/edill/autobin
PYTHONHOME: <not set>
PYTHONPATH: <not set>

WARNING: could not import _license.show_info
# try:
# $ conda install -n root _license

condarc

always_yes: true
binstar_upload: false
channels:
- conda-forge
- defaults
- ericdill
show_channel_urls: true
allow_softlinks: false
track_features:
- nomkl

conda list

# packages in environment at /tmp/mc:
#
ca-certificates           2016.9.26                     0    conda-forge
certifi                   2016.9.26                py35_0    conda-forge
conda                     4.2.13                   py35_0    conda-forge
conda-env                 2.6.0                         0    conda-forge
constructor               1.5.1                    py35_0    defaults
libconda                  4.0.0                    py35_0    conda-forge
ncurses                   5.9                          10    conda-forge
openssl                   1.0.2h                        3    conda-forge
pip                       9.0.1                    py35_0    conda-forge
pycosat                   0.6.1                    py35_0    conda-forge
pycrypto                  2.6.1                    py35_0    conda-forge
python                    3.5.2                         4    conda-forge
pyyaml                    3.11                     py35_0    conda-forge
readline                  6.2                           0    conda-forge
requests                  2.12.4                   py35_0    conda-forge
ruamel_yaml               0.11.14                  py35_1    defaults
setuptools                32.3.1                   py35_0    conda-forge
sqlite                    3.13.0                        1    conda-forge
tk                        8.5.19                        1    conda-forge
wheel                     0.29.0                   py35_0    conda-forge
xz                        5.2.2                         0    conda-forge
yaml                      0.1.6                         0    conda-forge
zlib                      1.2.8                         3    conda-forge

[msarahan]

Sorry @ericdill I'm not the right one for this. I'll make sure @ilanschnell sees this issue, though.

[ilanschnell]

The problem is that the Python binary from conda-forge contains binary placeholders, which causes a bootstrapping issue. The solution is to fix the conda-forge recipe by adding:

build:
  detect_binary_files_with_prefix: False

[msarahan]

Thanks @ilanschnell, that should improve the conda-forge python in more ways than just this.

[jjhelmus]

I've run into this issue as well, the issue is that during install the constructor install script re-writes shared libraries used by a process (python). These are files are typically memory mapped, rewriting causes a Bus Error.

There are two solutions that are think are possible

1. Build the Python package without any binary prefixes.
2. Update constructor to unlink files prior to writing the binary file with the replaced prefix.

From personal experience, I know that the first option works but I think the second is more general and a better solution. There are files in most Python packages which have binary prefixes that should be replaced, I do not think misrepresenting this fact is the correction solution.

[msarahan]

Thanks @jjhelmus. I agree that 2 is a nicer solution. How does the unlinking work? Is copying particular files an adequate substitute? Should we instead be setting the no_link meta.yaml setting for any files that are causing this issue?

[ericdill]

fwiw this problem started showing up in my nightly constructor creation last night. The difference is that I started pulling in python 3.5.2-4 from conda-forge. There is no bus error with python 3.5.2-2 from conda-forge

[edit] bad typo in python version (was 3.4.2 instead of 3.5.2)

[jjhelmus]

Stack overflow has a nice write up on this topic: How to upgrade shared library without crash?.

I think that adding os.unlink(path) to the update_prefix function in install.py right before writing to the file should fix the problem. The end of the function would look as follows:

    if new_data == data:
        return
    st = os.lstat(path)
    os.unlink(path)
    with exp_backoff_fn(open, path, 'wb') as fo:
        fo.write(new_data)
    os.chmod(path, stat.S_IMODE(st.st_mode))

Unlinking the file keeps a copy of the the memory mapped file until the process end. A new file is created rather than replacing the contents of the file inplace.

[jjhelmus]

@ericdill The change is that python 3.5.2-4 was built with conda-build 2.x where binary prefix detection is opt-out rather than opt-in. The newer builds have binary prefixes in some of the shared libraries which need to be replaced when the package is installed in an environment. The older builds did not have these files correctly marked even though binary prefixes were present in the files.

[ericdill]

@jjhelmus Ahhh I see. Thanks

[jjhelmus]

Tested the unlink change and submitted as PR #58.

I tested this on a linux-64 machine and was able to successfully create and install an installer which uses the conda-forge python-3.5.2-4 package which has binary prefixes. Without this change the installer would cause a bus error when executed.

Without the change in PR #58:

$ ./python-3.5-Linux-x86_64.sh 

Welcome to python 3.5

python will now be installed into this location:
/home/jhelmus/python

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/home/jhelmus/python] >>> 
PREFIX=/home/jhelmus/python
installing: python-3.5.2-4 ...
Python 3.5.2
./python-3.5-Linux-x86_64.sh: line 161:  2664 Bus error               (core dumped) $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT

With change in PR #58:

$ ./python-3.5-Linux-x86_64.sh 

Welcome to python 3.5

python will now be installed into this location:
/home/jhelmus/python

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/home/jhelmus/python] >>> 
PREFIX=/home/jhelmus/python
installing: python-3.5.2-4 ...
Python 3.5.2
installing: ncurses-5.9-10 ...
installing: openssl-1.0.2j-0 ...
installing: readline-6.2-2 ...
installing: sqlite-3.13.0-1 ...
installing: tk-8.5.19-1 ...
installing: xz-5.2.2-0 ...
installing: zlib-1.2.8-3 ...
installation finished.
Do you wish the installer to prepend the python install location
to PATH in your /home/jhelmus/.bashrc ? [yes|no]
[no] >>> no

You may wish to edit your .bashrc or prepend the python install location:

$ export PATH=/home/jhelmus/python/bin:$PATH

Thank you for installing python!

[ilanschnell]

I just merged #58, and am going to release 1.5.2 later today.

[ericdill]

@jjhelmus Thank you for figuring this out! @ilanschnell thanks for moving quickly on this!

[hammerdirt]

Just received the same return "Bus error (core dumped) when running conda update conda

conda version: 4.3.23
ubuntu: 16.04.3, xenial

I am not understanding the fix here.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!