Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompatible changes in version 5.1 #265

Open
perlpunk opened this issue Mar 14, 2019 · 4 comments

Comments

Projects
None yet
4 participants
@perlpunk
Copy link
Member

commented Mar 14, 2019

The PyYAML release 5.1 adds some incompatible changes.
If you are getting problems with the new version, that can have several reasons, which are listed here.

Also note that there are things that still need to be fixed. Please wait for release 5.2 and see details at the bottom

The full list of merged PRs can be found here: https://github.com/yaml/pyyaml/blob/master/announcement.msg#L30

The two incompatible changes are:

  • #257 -- Deprecate yaml.load and add FullLoader and UnsafeLoader classes
  • #256 -- Make default_flow_style=False

load: Deprecate yaml.load and add FullLoader and UnsafeLoader classes

For the full story read #257 and #207

YAML is a serialization language, and the yaml.load method allowed to execute arbitrary function calls and import modules, which made it unsafe when loading untrusted YAML.

With this release, it is deprecated to use yaml.load without a Loader argument and will issue a warning:

data = yaml.load("some yaml") # deprecated, warns. It uses FullLoader behind the scenes
data = yaml.load("some yaml", Loader=yaml.SafeLoader) # ok

use safe_load/SafeLoader

yaml.safe_load() always existed for that purpose, and most of you which are getting a warning now should be able to just replace yaml.load with yaml.safe_load:

yaml.safe_load(...)
yaml.load(..., Loader=yaml.SafeLoader)

use full_load/FullLoader

If you are actually using the serialization features (e.g. !!python/object/... tags) and want to load python objects, you should be fine with using the FullLoader:

yaml.load(..., Loader=yaml.FullLoader)
yaml.full_load(...)

FullLoader is pretty safe, but not completely. For an attack, you would need to

  • find a module that does something destructive/exploitable in its constructor
  • rely on the fact that this module is installed and already loaded

use unsafe_load/UnsafeLoader

There are two things that the FullLoader won't do, though:

  • Automatically import the modules for the objects that you want to load
  • Execute arbitrary function/method calls

If you need this then you must:

yaml.load(..., Loader=yaml.UnsafeLoader)
yaml.unsafe_load(...)
# or be backwards compatible
yaml.load(..., Loader=yaml.Loader)

yaml.Loader will behave just like yaml.UnsafeLoader

dump: Make default_flow_style=False

#256

YAML has two styles for collections: block and flow style.
The flow style looks a bit similar to JSON or to Python itself:
block key: { flow: mapping, more: items }

To save some space, the default for dumping was to use flow style on lists and dicts at the lowest level of a data structure, that means, that have no child nodes themselves:
yaml.dump(data, default_flow_style=None)

However, many people were using default_flow_style=False to avoid this, so the default didn't seem to be the best.
We decided that we change the default in a major release:
yaml.dump(data, default_flow_style=False)

You can still get the old behaviour by using None instead of False.

Other important changes

  • #45 -- Allow colon in a plain scalar in a flow context
  • #63 -- Adding support to Unicode characters over codepoint 0xffff
  • #143 -- Allow to turn off sorting keys in Dumper
  • #158 -- Support escaped slash in double quotes "/"

Breakages to be fixed in 5.2

When the default loader yaml.Loader for yaml.load() was changed to FullLoader, it was forgotten that there are two other places where we need to change the default:

  • in yaml.add_constructor
  • in the class YAMLObject which you can inherit from so that from_yaml and to_yaml are called

Also the logic for detecting special characters was broken, which results in strings with tabs (for example) to be dumped in plain style on python 2.7 on systems with sys.maxunicode <= 0xffff.

PRs are ready:
#273
#274
#276

@perlpunk perlpunk pinned this issue Mar 14, 2019

jmaupetit added a commit to openfun/arnold that referenced this issue Mar 17, 2019

📌(lint) pin PyYAML to 3.13
Latest PyYAML (5.1) release breaks ansible-lint execution [1]. Let's
stick to an older release to now.

[1] yaml/pyyaml#265

jmaupetit added a commit to openfun/arnold that referenced this issue Mar 18, 2019

📌(lint) pin PyYAML to 3.13
Latest PyYAML (5.1) release breaks ansible-lint execution [1]. Let's
stick to an older release to now.

[1] yaml/pyyaml#265

jmaupetit added a commit to openfun/arnold that referenced this issue Mar 18, 2019

📌(lint) pin PyYAML to 3.13
Latest PyYAML (5.1) release breaks ansible-lint execution [1]. Let's
stick to an older release to now.

[1] yaml/pyyaml#265
@perlpunk

This comment has been minimized.

Copy link
Member Author

commented Mar 19, 2019

Please see the update. If you are using yaml.add_constructor without specifying a loader, and same for using YAMLObject/from_yaml/to_yaml please wait for release 5.2

@henrikef henrikef referenced this issue Mar 20, 2019

Closed

pyyaml 5.1 #97

dis-xcom added a commit to Mirantis/tcp-qa that referenced this issue Mar 22, 2019

Do not install PyYAML version 5.1
yaml.load() has been deprecated in the patch
yaml/pyyaml#265

Change-Id: I24dff0d25d6f8ae4a636b8c64e083fa7ff8603e7

@erseco erseco referenced this issue Mar 26, 2019

Closed

Support pyyaml 5.1 #2

@wimglenn

This comment has been minimized.

Copy link

commented Apr 1, 2019

Where are the docs maintained? Doesn't seem to be a part of this repository directly?

Although the source link was updated for PyYAML-5.1.tar.gz on PyYAMLDocumentation the info remaining about default_flow_style is old (and now, incorrect).

@pombredanne

This comment has been minimized.

Copy link

commented Apr 11, 2019

I maintain this small library called saneyaml that is a wrapper of PyYAML to get "sane" defaults (mainly safe loading and block style defaults) at https://github.com/nexB/saneyaml
I pushed yesterday a bunch of updates that deal with making this compatible with both PyYAML 3.1x and 5.1x after the report of installation issues from @dankegel in nexB/saneyaml#1

This may be of help to some to see the changes needed to support both versions:
nexB/saneyaml@v0.3...v0.4

As a side note, I wished I would not have had the need to write such a wrapper ;)

ngfgrant added a commit to cloudreach/sceptre that referenced this issue Apr 16, 2019

Upgrade to PyYaml 5.1
Updates the PyYaml version to 5.1.

Adds in YamlLoader as per yaml/pyyaml#292

Other incompatible changes were reviewed
(yaml/pyyaml#265) and the yaml.Loader appears
to the be only concern for now.

[Resolves #665]

ngfgrant added a commit to cloudreach/sceptre that referenced this issue Apr 23, 2019

Upgrade to PyYaml 5.1 (#681)
Updates the PyYaml version to 5.1.

Adds in YamlLoader as per yaml/pyyaml#292

Other incompatible changes were reviewed
(yaml/pyyaml#265) and the yaml.Loader appears
to the be only concern for now.

[Resolves #665]

laurentb added a commit to laurentb/weboob that referenced this issue Apr 25, 2019

ngfgrant added a commit to cloudreach/sceptre that referenced this issue May 2, 2019

Upgrade to PyYaml 5.1 (#681)
Updates the PyYaml version to 5.1.

Adds in YamlLoader as per yaml/pyyaml#292

Other incompatible changes were reviewed
(yaml/pyyaml#265) and the yaml.Loader appears
to the be only concern for now.

[Resolves #665]

@jamesewoo jamesewoo referenced this issue May 3, 2019

Closed

PyYAML upgrade #689

@graingert

This comment has been minimized.

Copy link

commented May 21, 2019

PRs are ready:
#273

Is now in "Won't fix"

#274

Replaced with #287 (which is merged)

#276

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.