Errors with render and recipes with multiple outputs

[bnemanich]

Actual Behavior

Calling conda_build.render on recipes with multiple outputs does not always generate the correct build requirements. For example, if I call conda_build.render on this recipe: https://github.com/open-ce/xgboost-feedstock/blob/main/recipe/meta.yaml , the generated metadata doesn't not include cmake. This means that passing the metadata to conda_build.build fails.

Expected Behavior

The generated metadata should include all of the build requirements listed in the recipe.

Steps to Reproduce

git clone https://github.com/open-ce/xgboost-feedstock.git

From a python script

import conda_build

meta_data = conda_build.render("xgboost-feedstock/recipe")
conda_build.build(meta_data)

conda info

conda info

 active environment : base
    active env location : /tmp/nemanich/miniconda3
            shell level : 1
          conda version : 4.10.1
    conda-build version : 3.21.4
         python version : 3.7.6.final.0
       virtual packages : __cuda=10.2=0
                          __linux=4.14.0=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=ppc64le
       base environment : /tmp/nemanich/miniconda3  (writable)
      conda av data dir : /tmp/nemanich/miniconda3/etc/conda
  conda av metadata url : https://repo.anaconda.com/pkgs/main
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-ppc64le
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-ppc64le
                          https://repo.anaconda.com/pkgs/r/noarch
               platform : linux-ppc64le
             user-agent : conda/4.10.1 requests/2.22.0 CPython/3.7.6 Linux/4.14.0-115.18.1.el7a.ppc64le rhel/7.6 glibc/2.17
                UID:GID : 581379:1059966
             netrc file : None
           offline mode : False

[chenghlee]

Is there a conda_build_config.yaml that's used with this recipe? We are trying to replicate this issue, but without the conda_config_config.yaml you're using, we don't have appropriate values for various variables.

[npanpaliya]

@chenghlee - Here is the conda_build_config.yaml that the above test needs. https://github.com/open-ce/open-ce/blob/main/envs/conda_build_config.yaml.

[chenghlee]

Are you calling render and build from a script, rather than directly using conda render and conda build from the command line?

[npanpaliya]

We've calls to conda_build.api.build, conda_build.api.debug and conda_build.api.render.

[chenghlee]

Can you try with the CLI tools (conda render and/or conda build) to see if the issue still appears there?

[npanpaliya]

Okay

[npanpaliya]

With the latest conda-build 3.21.8, we are seeing another issue with conda_build.api.render method. It gives the following error -

    metas = conda_build.api.render(path,
  File "~/anaconda3/lib/python3.8/site-packages/conda_build/api.py", line 42, in render
    metadata_tuples = render_recipe(recipe_path, bypass_env_check=bypass_env_check,
  File "~/anaconda3/lib/python3.8/site-packages/conda_build/render.py", line 838, in render_recipe
    m.config.compute_build_id(m.name(), m.version(), reset=reset_build_id)
  File "~/anaconda3/lib/python3.8/site-packages/conda_build/config.py", line 611, in compute_build_id
    shutil.move(old_dir, work_dir)
  File "~/anaconda3/lib/python3.8/shutil.py", line 802, in move
    copy_function(src, real_dst)
  File "~/anaconda3/lib/python3.8/shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "~/anaconda3/lib/python3.8/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '~/anaconda3/conda-bld/work'

This is just a part of the error but pointing to the problem in conda-build.
This used to work fine with conda-build 3.21.7.

Further investigation shows that 001bcc4#diff-eaaf464431ee59c1575fecf5aa57498e16bb0694fd2cb6ee1638d3ebd682e68f change causes the problem.

And when I tried conda render CLI command instead of conda_build.api.render, it worked fine even with 3.21.8 version.

[chenghlee]

It's likely you need additional code to get and pass values for the config, variants, and possibly other arguments to the render and/or build API functions. I would suggest looking at the code in cli/main_render.py and cli/main_build.py as a start.

Is there a reason you're writing your own script rather than using the existing conda render & conda build CLI tools?

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!