charm generate seems to be broken with ruamel.yaml 0.10.13

[gnuoy]

I'm trying to assemble a charm which has its own config.yaml and inherits some config.yaml from a layer. When I try and build the charm I get: http://paste.ubuntu.com/14470761/

Adding some debug seems to indicate that its exploding at the point of creating the config.yaml.

If I downgrade the version of ruamel.yaml to 0.10.12 it fixes the issue

[chuckbutler]

@bcsaller @marcoceppi we may need to pin at 0.10.12 until we have a proper fix. this appears to still be the case with 0.10.17 of ruamel, starting with 0.10.13

[marcoceppi]

Packaging and the PPA has this pinned. 0.10.2 is what gets installed from
the archive. We could potentially pin this as well for pip installers but
very few people should be using the pip package directly.

On Sat, Jan 30, 2016 at 8:44 AM Charles Butler notifications@github.com
wrote:

@bcsaller https://github.com/bcsaller @marcoceppi
https://github.com/marcoceppi - Is there a way we can blacklist this
version of ruamel? I've asked gnuoy to try with 0.10.17 the current release
to see if this is a breaking bug that has persisted, or if we need to pin
at 0.10.12 until we have a proper investigative fix.

—
Reply to this email directly or view it on GitHub
#88 (comment).

[chuckbutler]

@marcoceppi well, interesting thing here. I just built the charm from the provided layers, on the most recent ruamel and it worked. soooo.... i'm not sure where the disconnect is, but i'll work with liam to figure out whats happening here and why its angry.

[ajkavanagh]

Some details on how to reproduce the bug. I've tracked it down to something in the ruamel.yaml handling PreservedScalarString objects (ruamel.yaml.scalarstring.PreservedScalarString) which is triggered by a '|' (pipe) character in the yaml file. This gets converted internally to a unicode string, but the rest of the yaml file is ordinary Py2 strings. e.g. the config.yaml

options:
  os-data-network:
    type: string
    default:
    description: |
      The IP address and netmask of the OpenStack Data network (e.g.,
      192.168.0.0/24)
      .
      This network will be used for tenant network traffic in overlay
      networks.
  mac-network-map:
    default:
    type: string
    description: |
      Map of physical nic mac address to configured VLAN's.

becomes (with some debugging added):

<-- SerializedTactic.read(ConfigYAML: /home/ubuntu/charms/openvswitch-odl/charm/config.yaml -> /home/ubuntu/charms/openvswitch-odl/build/trusty/openvswitch-od
l/config.yaml)
/home/ubuntu/charms/openvswitch-odl/charm/config.yaml
construct_yaml_str <type 'unicode'> options
construct_yaml_str <type 'unicode'> os-data-network
construct_yaml_str <type 'unicode'> mac-network-map
construct_yaml_str <type 'unicode'> type
construct_yaml_str <type 'unicode'> string
construct_yaml_str <type 'unicode'> default
construct_yaml_str <type 'unicode'> description
construct_yaml_str <class 'ruamel.yaml.scalarstring.PreservedScalarString'> The IP address and netmask of the OpenStack Data network (e.g.,
192.168.0.0/24)
.
This network will be used for tenant network traffic in overlay
networks.

construct_yaml_str <type 'unicode'> default
construct_yaml_str <type 'unicode'> type
construct_yaml_str <type 'unicode'> string
construct_yaml_str <type 'unicode'> description
construct_yaml_str <class 'ruamel.yaml.scalarstring.PreservedScalarString'> Map of physical nic mac address to configured VLAN's.

ordereddict([('options', ordereddict([('os-data-network', ordereddict([('type', 'string'), ('default', None), ('description', u'The IP address and netmask of
the OpenStack Data network (e.g.,\n192.168.0.0/24)\n.\nThis network will be used for tenant network traffic in overlay\nnetworks.\n')])), ('mac-network-map',
ordereddict([('default', None), ('type', 'string'), ('description', u"Map of physical nic mac address to configured VLAN's.\n")]))]))])
-->

Notice the u" in front of "Map of ..." above.

Anyway, this results in a blow-up (ultimately) in ruamel.yaml.emiter.write_comment() due to trying to combine unicode and strs (Py2). It probably doesn't happen in Py3. The caller (from charm-helpers) is charm-tools.build.tactics.YamlTactic.dump().

To reproduce it:

juju deploy ubuntu
juju ssh ubuntu/0

and then

sudo apt-get install language-pack-en
sudo apt-get install git
git clone https://git.launchpad.net/~thedac/charms/+source/openvswitch-odl
cd openvswitch-odl/
sudo apt-get install make
mkdir layers
sudo apt-get install python-tox
sudo apt-get install libpython-dev
export http_proxy=http://squid.internal:3128/
export https_proxy=https://squid.internal:3128/
make generate

This happens with ruamel.yaml 0.10.20 (latest version available on PiPy).

However, I can't seem to reproduce it in ruamel.yaml's tests (as an extra test). So it looks like something that charm-tools is doing with ruamel.yaml ??

[javacruft]

@marcoceppi we're using the pip package for new layered charms as we're using tox and venv to generate the final charm - so if this could be pinned elsewhere as well that would be great!

[johnsca]

This was duplicated in #118 and I was able to track it down to an interaction between ruamel.yaml and path.py (see that issue for details and a minimal test case). I submitted the above PR which has been merged but I don't think has been released. I believe that will resolve the issue without having to pin ruamel.yaml.