Broken ruamel_yaml build!

[dhirschfeld]

After installing Miniconda3 on a Windows NanoServer container conda doesn't work out of the box:

PS C:\> conda info
Traceback (most recent call last):
  File "C:\Miniconda3\lib\site-packages\conda\common\configuration.py", line 49, in <module>
    from ruamel_yaml.comments import CommentedSeq, CommentedMap
  File "C:\Miniconda3\lib\site-packages\ruamel_yaml\__init__.py", line 9, in <module>
    from .cyaml import *
  File "C:\Miniconda3\lib\site-packages\ruamel_yaml\cyaml.py", line 5, in <module>
    from ruamel_yaml._ruamel_yaml import CParser, CEmitter
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\lib\site-packages\conda\exceptions.py", line 616, in conda_exception_handler
    return_value = func(*args, **kwargs)
  File "C:\Miniconda3\lib\site-packages\conda\cli\main.py", line 85, in _main
    from ..base.context import context
  File "C:\Miniconda3\lib\site-packages\conda\base\context.py", line 20, in <module>
    from ..common.configuration import (Configuration, LoadError, MapParameter, PrimitiveParameter,
  File "C:\Miniconda3\lib\site-packages\conda\common\configuration.py", line 52, in <module>
    from ruamel.yaml.comments import CommentedSeq, CommentedMap  # pragma: no cover
ModuleNotFoundError: No module named 'ruamel'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\lib\site-packages\conda\common\configuration.py", line 49, in <module>
    from ruamel_yaml.comments import CommentedSeq, CommentedMap
  File "C:\Miniconda3\lib\site-packages\ruamel_yaml\__init__.py", line 9, in <module>
    from .cyaml import *
  File "C:\Miniconda3\lib\site-packages\ruamel_yaml\cyaml.py", line 5, in <module>
    from ruamel_yaml._ruamel_yaml import CParser, CEmitter
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Miniconda3\Scripts\conda-script.py", line 5, in <module>
    sys.exit(conda.cli.main())
  File "C:\Miniconda3\lib\site-packages\conda\cli\main.py", line 167, in main
    return conda_exception_handler(_main, *args)
  File "C:\Miniconda3\lib\site-packages\conda\exceptions.py", line 632, in conda_exception_handler
    print_unexpected_error_message(e)
  File "C:\Miniconda3\lib\site-packages\conda\exceptions.py", line 542, in print_unexpected_error_message
    from conda.base.context import context
  File "C:\Miniconda3\lib\site-packages\conda\base\context.py", line 20, in <module>
    from ..common.configuration import (Configuration, LoadError, MapParameter, PrimitiveParameter,
  File "C:\Miniconda3\lib\site-packages\conda\common\configuration.py", line 52, in <module>
    from ruamel.yaml.comments import CommentedSeq, CommentedMap  # pragma: no cover
ModuleNotFoundError: No module named 'ruamel'

AFAICT it's because yaml.dll is incorrectly compiled with VS2008 rather than VS2015 which it should be for Python 3.6:

C:\vcpp2008\continuum\py36>dumpbin /dependents yaml.dll
Microsoft (R) COFF/PE Dumper Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file yaml.dll

File Type: DLL

  Image has the following dependencies:

    MSVCR90.dll  <======== VS2008 DLL !!!
    KERNEL32.dll

  Summary

        3000 .data
        1000 .pdata
        2000 .rdata
        1000 .reloc
        1000 .rsrc
       34000 .text

In contrast, the conda-forge package appears to be correctly compiled with VS2015 but they don't yet have a py36 package and if conda is broken it would be difficult to install anyway.

C:\vcpp2008\ruamel_yaml\ext>dumpbin /dependents yaml.dll
Microsoft (R) COFF/PE Dumper Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file yaml.dll

File Type: DLL

  Image has the following dependencies:

    VCRUNTIME140.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    KERNEL32.dll

  Summary

        1000 .data
        1000 .gfids
        1000 .pdata
        4000 .rdata
        1000 .reloc
        1000 .rsrc
       16000 .text

xref: #1298

[dhirschfeld]

I think on "normal" windows the required deps are always available so people don't run into a problem having ruamel_yaml compiled against the incorrect runtime. They're not there on nanoserver though which has exposed the latent issue.

[dhirschfeld]

Thanks for looking into it @mingwandroid - I can confirm that conda works fine on nanoserver with a correctly built ruamel_yaml.

Tested with https://anaconda.org/dhirschfeld/ruamel_yaml which was built from the conda-forge recipe.

[scw]

I've also hit this issue in some Windows contexts where VS2008 DLLs aren't provided. The conda-forge packages of ruaml_yaml is correctly linked against the VS2015 DLLs in the versions of Python I've tested (h/t @dhirschfeld), but would be great to have to have this fixed in the Continuum maintained version of the package.

[dhirschfeld]

Just a note to mention the buggy version is ruamel_yaml: 0.11.14-py36_1 and it hasn't yet been fixed.

[scw]

I did some further testing with the conda-forge version, and in that case the recipe is 1) requiring the base yaml package, and manually copying it from Library\bin into the built package. This doesn't guarantee a correct version, I created a short PR which links the ruamel_yaml build directly to the related VC environment. There may be a more elegant way to do this, but that was enough to get the correct yaml.dll.

[mingwandroid]

Is this really still broken? If so I'm somewhat amazed given the amount of stuff we've changed!

[dhirschfeld]

AFAICS this library hasn't been updated since? It's not urgent as I'm fully capable of money-patching my own install but it would be really good if this fix could make it into the next Anaconda/Miniconda package.

[dhirschfeld]

I didn't check if conda still depends on this package though, but if it does it's broken on Windows Containers (and anywhere else without the 2008 C++ libraries installed)

[nehaljwani]

@dhirschfeld ruamel_yaml: 0.11.14-py36_1 is quite old. Did you try the newer builds of this package?

[scw]

It looks like the packages @kalefranz uploaded in the past couple weeks resolve this issue. I see the current version of yaml, 0.1.7-hc54c509_2, and the previous release hdd19a5b_2 both resolve this issue. The latest ruamel_yaml is also bound explicitly to the correct runtime with the vc metapackage, so installing a build which requires VS2008 / VC 9 will force the downgrade of the environment to Python 2.7, which is expected.

[dhirschfeld]

Right, it seems 0.11.14=py36h9b16331_2 is built correctly. I'll now have to find out why my version didn't update despite my having done an update --all...

[dhirschfeld]

So, the reason my ruamel_yaml wasn't updating was a channel priority issue I believe. Because the new build has the same version number the channel priority came into play and the new build wasn't in my own (higher priority) channels.

Don't know there's anything that can be done about that - just one of those corner cases.

NB: I have channel_priority=false so that if a higher version number is available in defaults it will get installed in preference to higher priority channels.