[conda-build] issue with package pinning ?

[neodark]

Hi,

Could you please help ?

We are having a strange issue related to conda-build and package pinning:

This is the error:

raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.
(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"

Here is the complete output:

/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/environ.py:418: UserWarning: The environment variable 'DOCSERVER' is being passed through with value <website>.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/environ.py:418: UserWarning: The environment variable 'DOCSERVER' is being passed through with value website.  If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
  UserWarning
Traceback (most recent call last):
  File "/local/builds/beat/beat.editor/miniconda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 438, in main
    execute(sys.argv[1:])
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 408, in execute
    action(recipe, config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 348, in test_action
    return api.test(recipe, move_broken=False, config=config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/api.py", line 227, in test
    stats=stats)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1835, in test
    config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1805, in construct_metadata_for_test
    m, hash_input = _construct_metadata_for_test_from_recipe(recipedir_or_package, config)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/build.py", line 1691, in _construct_metadata_for_test_from_recipe
    metadata = finalize_metadata(metadata)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/render.py", line 457, in finalize_metadata
    output = m.get_rendered_output(m.name())
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 2047, in get_rendered_output
    permit_undefined_jinja=permit_undefined_jinja):
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 2037, in get_rendered_outputs_section
    self.parse_until_resolved()
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 980, in parse_until_resolved
    bypass_env_check=bypass_env_check)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 924, in parse_again
    bypass_env_check=bypass_env_check),
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/metadata.py", line 1494, in _get_contents
    rendered = template.render(environment=env)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/local/builds/beat/beat.editor/conda/meta.yaml", line 11, in top-level template code
    - {{ pin_subpackage(name) }}
  File "/local/builds/beat/beat.editor/miniconda/lib/python3.7/site-packages/conda_build/jinja_context.py", line 313, in pin_subpackage
    raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.
(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"

Thanks for your help

[msarahan]

Please post the information requested in the template. We also need a minimal reproducible test case.

[neodark]

@msarahan thanks for your quick answer.

Here some extra information:

So basically the system is running on a CI build on a gitlab repository:

It seems to fail when executing this line:
(11:37:15.224) Error: Command Failed "/local/builds/beat/beat.editor/miniconda/bin/conda build --no-anaconda-upload --variant-config-files /local/builds/beat/beat.editor/_ci/conda_build_config.yaml --python=3.6 --test conda"

The example of this failed job can be found here: https://gitlab.idiap.ch/beat/beat.editor/-/jobs/149048

It uses this conda build configuration file:

macos_min_version:
  - 10.9
macos_machine:
  - x86_64-apple-darwin13.4.0
MACOSX_DEPLOYMENT_TARGET:
  - 10.9
CONDA_BUILD_SYSROOT:            # [osx]
  - /opt/MacOSX10.9.sdk         # [osx]
# makes autotools verbose
VERBOSE_AT:
  - V=1
# makes cmake verbose
VERBOSE_CM:
  - VERBOSE=1

# the blas implementations that we build against
blas_impl:
  - mkl

pin_run_as_build:
  libboost:
    max_pin: x.x.x
  py_boost:
    max_pin: x.x.x
  boost:
    max_pin: x.x.x

## the dependencies that we build against multiple versions
python:
  - 3.6

zip_keys:
  -                             # [win]
    - vc                        # [win]
    - c_compiler                # [win]
    - cxx_compiler              # [win]
    - fortran_compiler_version  # [win]
    - python                    # [win]


# Here is the version of dependencies are used when building packages (build
# and host requirements). We keep a list of **all of them** here to make sure
# everything goes as expected in our conda build process. For the version of
# packages that are used for testing packages, see the recipe of bob-devel.
# The version here do not necessarily match the versions in bob-devel.

# This version of bob-devel will be used at test time of packages:
bob_devel:
  - 2018.09.24

# This version of beat-devel will be used at test time of packages. Notice it
# uses bob-devel and should have a version that is greater or equal its value
beat_devel:
  - 2018.09.24

# The build time only dependencies (build requirements).
# Updating these to the latest version all the time is OK and a good idea.
# These versions should match the versions inside bob-devel as well (if they
# overlap) so update them in both places.
cmake:
  - 3.12.2
pkg_config:
  - 0.29.2
cython:
  - 0.28.1

# The dependencies that we link against (host requirements).
# Ideally we want to build against the oldest possible version of packages when
# we are linking against them. It is best to keep this in sync with:
# https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml
numpy:
  - 1.14.5
boost:
  - 1.65.1
cyvlfeat:
  - 0.4.6
ffmpeg:
  - 3.4
freetype:
  - 2.8
giflib:
  - 5.1.4
hdf5:
  - 1.10.1
jpeg:
  - 9b
libblitz:
  - 1.0.1
libmatio:
  - 1.5.11
libogg:
  - 1.3.2
libpng:
  - 1.6.32
libsvm:
  - 3.22
libtiff:
  - 4.0.9
mkl:
  - 2018.0.3
openfst:
  - 1.6.1
sox:
  - 14.4.2
speex:
  - 1.2.0
speexdsp:
  - 1.2rc3
sqlite:
  - 3.20.1
vlfeat:
  - 0.9.21
xz:
  - 5.2.3
zlib:
  - 1.2.11

# The dependencies that are needed for runtime only (run requirements).
# These versions **should** match the versions inside bob-devel recipe.
caffe:
  - 1.0
click:
  - 6.7
click_plugins:
  - 1.0.3
coverage:
  - 4.5.1
dlib:
  - 19.7
docopt:
  - 0.6.2
jinja2:
  - 2.10
kaldi:
  - 2017.03.13
matplotlib:
  - 2.2.3
menpo:
  - 0.8.1
menpofit:
  - 0.5.0
mne:
  - 0.15.2
mr_developer:
  - 1.38
nose:
  - 1.3.7
opencv:
  - 3.3.1
pillow:
  - 5.2.0
pyedflib:
  - 0.1.11
pytorch:
  - 0.4.1
pyyaml:
  - 3.13
requests:
  - 2.19.1
schema:
  - 0.6.7
scikit_image:
  - 0.19.2
scipy:
  - 1.1.0
setuptools:
  - 40.2.0
six:
  - 1.11.0
sphinx:
  - 1.7.9
sphinx_rtd_theme:
  - 0.4.1
sqlalchemy:
  - 1.2.11
tabulate:
  - 0.8.2
tensorflow:
  - 1.9.0
torchvision:
  - 0.2.1
zc_buildout:
  - 2.11.2
zc_recipe_egg:
  - 2.0.5

# The dependencies that are needed for runtime only (run requirements) of BEAT
# packages. These versions **should** match the versions inside beat-devel
# recipe.
docker_py:
  - 3.2.1
jsonschema:
  - 2.6.0
oset:
  - 0.1.3
python_graphviz:
  - 0.8.2
pyzmq:
  - 17.0.0
simplejson:
  - 3.13.2
termcolor:
  - 1.1.0
sphinxcontrib_programoutput:
  - 0.11
nodejs:
  - 8.9.3
flask:
  - 0.12.2
flask_cors:
  - 3.0.3
flask_restful:
  - 0.3.6

For more information:
This is the .gitlab.yml file that runs on our CI to launch the job

# Definition of global variables (all stages)
variables:
  CONDA_ROOT: "${CI_PROJECT_DIR}/miniconda"


# Definition of our build pipeline order
stages:
  - build
  - browser-tests
  - deploy
  - pypi


# Build targets
.build_template: &build_job
  stage: build
  before_script:
    - mkdir _ci
    - curl --silent "https://gitlab.idiap.ch/bob/bob.admin/raw/master/gitlab/install.sh" > _ci/install.sh
    - chmod 755 _ci/install.sh
    - ./_ci/install.sh _ci master #installs ci support scripts
    - ./_ci/before_build.sh
  script:
    - ./_ci/build.sh
  after_script:
    - ./_ci/after_build.sh
  cache: &build_caches
    paths:
      - miniconda.sh
      - ${CONDA_ROOT}/pkgs/*.tar.bz2
      - ${CONDA_ROOT}/pkgs/urls.txt


.build_linux_template: &linux_build_job
  <<: *build_job
  tags:
    - docker
  image: continuumio/conda-concourse-ci
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - ${CONDA_ROOT}/conda-bld/linux-64/*.tar.bz2
  cache:
    <<: *build_caches
    key: "linux-cache"


.build_macosx_template: &macosx_build_job
  <<: *build_job
  tags:
    - macosx
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - ${CONDA_ROOT}/conda-bld/osx-64/*.tar.bz2
  cache:
    <<: *build_caches
    key: "macosx-cache"


build_linux_27:
  <<: *linux_build_job
  variables:
    PYTHON_VERSION: "2.7"


build_linux_36:
  <<: *linux_build_job
  variables:
    PYTHON_VERSION: "3.6"
    BUILD_EGG: "true"
  artifacts:
    expire_in: 1 week
    paths:
      - _ci/
      - dist/*.zip
      - sphinx
      - ${CONDA_ROOT}/conda-bld/linux-64/*.tar.bz2


build_macosx_27:
  <<: *macosx_build_job
  variables:
    PYTHON_VERSION: "2.7"


build_macosx_36:
  <<: *macosx_build_job
  variables:
    PYTHON_VERSION: "3.6"


# Docker host based testing (must be run inside dind or docker-enabled host)
.browser_test_linux_template: &linux_browser_test_job
  stage: browser-tests
  image: docker.idiap.ch/beat/ci.env.editor:0.0.1r0
  before_script:
    # safe keep artifacts as before_build.sh will erase those...
    - apt-get update > /dev/null
    - apt-get install -y locales > /dev/null
    - echo "en_US UTF-8" > /etc/locale.gen
    - locale-gen en_US.UTF-8
    - export LANG=en_US.UTF-8
    - export LANGUAGE=en_US:en
    - export LC_ALL=en_US.UTF-8
    - mv ${CONDA_ROOT}/conda-bld .
    - ./_ci/install.sh _ci master #updates ci support scripts
    - ./_ci/before_build.sh
    - mv conda-bld ${CONDA_ROOT}
  script:
    - export BEAT_BROWSER_TESTS=true
    - BOB_TEST_ONLY=true ./_ci/build.sh
  after_script:
    - ./_ci/after_build.sh


browser_linux_36:
  <<: *linux_browser_test_job
  variables:
    PYTHON_VERSION: "3.6"
  dependencies:
    - build_linux_36
  tags:
    - docker


# Deploy targets
.deploy_template: &deploy_job
  stage: deploy
  before_script:
    - ./_ci/install.sh _ci master #updates ci support scripts
  script:
    - ./_ci/deploy.sh
  dependencies:
    - build_linux_27
    - build_linux_36
    - build_macosx_27
    - build_macosx_36
  tags:
    - deployer


deploy_beta:
  <<: *deploy_job
  environment: beta
  only:
    - master


deploy_stable:
  <<: *deploy_job
  environment: stable
  only:
    - /^v\d+\.\d+\.\d+([abc]\d*)?$/  # PEP-440 compliant version (tags)
  except:
    - branches


pypi:
  stage: pypi
  environment: pypi
  only:
    - /^v\d+\.\d+\.\d+([abc]\d*)?$/  # PEP-440 compliant version (tags)
  except:
    - branches
  before_script:
    - ./_ci/install.sh _ci master #updates ci support scripts
  script:
    - ./_ci/pypi.sh
  dependencies:
    - build_linux_36
  tags:
    - deployer

[anjos]

@msarahan, @183amir: Here is a minimal example that reproduces the issue, using git@github.com:AnacondaRecipes/zlib-feedstock.git:

$ git clone git@github.com:AnacondaRecipes/zlib-feedstock.git
$ cd zlib-feedstock
$ conda update -n base conda conda-build
# just to ensure we've got the lastest
$ conda build --no-test recipe
# works w/o issues
$ conda build --test recipe
TEST START: recipe
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults

Leaving build/test directories:
  Work:
 /idiap/user/aanjos/conda/conda-bld/work
  Test:
 /idiap/user/aanjos/conda/conda-bld/test_tmp
Leaving build/test environments:
  Test:
source activate  /idiap/user/aanjos/conda/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_
  Build:
source activate  /idiap/user/aanjos/conda/conda-bld/_build_env


Traceback (most recent call last):
  File "/idiap/user/aanjos/conda/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 438, in main
    execute(sys.argv[1:])
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 408, in execute
    action(recipe, config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 348, in test_action
    return api.test(recipe, move_broken=False, config=config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/api.py", line 227, in test
    stats=stats)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1835, in test
    config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1805, in construct_metadata_for_test
    m, hash_input = _construct_metadata_for_test_from_recipe(recipedir_or_package, config)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/build.py", line 1691, in _construct_metadata_for_test_from_recipe
    metadata = finalize_metadata(metadata)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/render.py", line 457, in finalize_metadata
    output = m.get_rendered_output(m.name())
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 2047, in get_rendered_output
    permit_undefined_jinja=permit_undefined_jinja):
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 2037, in get_rendered_outputs_section
    self.parse_until_resolved()
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 980, in parse_until_resolved
    bypass_env_check=bypass_env_check)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 924, in parse_again
    bypass_env_check=bypass_env_check),
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/metadata.py", line 1494, in _get_contents
    rendered = template.render(environment=env)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "/remote/idiap.svm/user.active/aanjos/work/bob/zlib-feedstock/recipe/meta.yaml", line 15, in top-level template code
    # mostly OK, but some scary symbol removal.  Let's try for trusting them.
  File "/idiap/user/aanjos/conda/lib/python3.6/site-packages/conda_build/jinja_context.py", line 313, in pin_subpackage
    raise ValueError("Bug in conda-build: we need to have info about other outputs in "
ValueError: Bug in conda-build: we need to have info about other outputs in order to allow pinning to them.  It's not here.

The problem seems to happen when you test only (and not build the package). This is currently breaking a lot of our infrastructure and issuing (unnecessary) rebuilds, as we cannot just test already built packages.

[183amir]

This is probably a regression issue. This used to work for us before.

[anjos]

I checked - conda-build 3.10.3 is the last to work correctly in this regard. Changes in 3.10.4 likely created the bug. All versions following that have it.

[msarahan]

I will fix this, but I am deprecating this mode of operation. Pass tarball paths to conda-build for testing instead.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!