ImportError: cannot import name yaml2 #319

Closed
ojos opened this Issue Feb 4, 2016 · 28 comments

Projects

None yet
@ojos
ojos commented Feb 4, 2016

The following errors occured in Python2 environment.

> invoke
Traceback (most recent call last):
  File "/Users/ojos/Workspace/hoge/python/bin/invoke", line 7, in <module>
    from invoke.main import program
  File "/Users/ojos/Workspace/hoge/python/lib/python2.7/site-packages/invoke/__init__.py", line 2, in <module>
    from .collection import Collection  # noqa
  File "/Users/ojos/Workspace/hoge/python/lib/python2.7/site-packages/invoke/collection.py", line 7, in <module>
    from .config import merge_dicts
  File "/Users/ojos/Workspace/hoge/python/lib/python2.7/site-packages/invoke/config.py", line 11, in <module>
    from .vendor import yaml2 as yaml

There is not invoke/vender/yaml2 package to version 0.12.1.
Please include.

@raimonesteve

I found same error. It's when install with "pip install invoke" and 0.12.1. Before version, 0.12.0 install successfully.

@raimonesteve

I try to install with "python setup.py install" and yaml2 directory found it, but not yaml3.

~/virtualenv/invprova$ ls lib/python2.7/site-packages/invoke-0.12.1-py2.7.egg/invoke/vendor/ fluidity/ __init__.py __init__.pyc lexicon/ six.py six.pyc yaml2/

I'm working with py2.7.

@ojos
ojos commented Feb 4, 2016

I installed in pip 7.1.2(python 2.7)

@denisgarci

Same here with Python 2. I don't have the issue on 0.12.0.

@noirbizarre

Same here, downgraded on 0.12.0 where the problem does not occurs.

@Benoss
Benoss commented Feb 5, 2016

Same here
python 2.7.6
pip 8.0.2
fresh virtualenv

>>> import invoke
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/benoit/.virtualenvs/test_invoke2/local/lib/python2.7/site-packages/invoke/__init__.py", line 2, in <module>
    from .collection import Collection  # noqa
  File "/home/benoit/.virtualenvs/test_invoke2/local/lib/python2.7/site-packages/invoke/collection.py", line 7, in <module>
    from .config import merge_dicts
  File "/home/benoit/.virtualenvs/test_invoke2/local/lib/python2.7/site-packages/invoke/config.py", line 11, in <module>
    from .vendor import yaml2 as yaml
ImportError: cannot import name yaml2

@alfakini
alfakini commented Feb 5, 2016

Same problem here while using streamparse, I downgraded on 0.12.0 and everything is now working.

$ sparse run -n test_topology
Traceback (most recent call last):
  File "/Users/user/miniconda2/bin/sparse", line 11, in <module>
    sys.exit(main())
  File "/Users/user/miniconda2/lib/python2.7/site-packages/streamparse/cli/sparse.py", line 51, in main
    load_subparsers(subparsers)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/streamparse/cli/sparse.py", line 27, in load_subparsers
    module = importlib.import_module('streamparse.cli.{}'.format(mod_name))
  File "/Users/user/miniconda2/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/user/miniconda2/lib/python2.7/site-packages/streamparse/cli/jar.py", line 9, in <module>
    from invoke import run
  File "/Users/user/miniconda2/lib/python2.7/site-packages/invoke/__init__.py", line 2, in <module>
    from .collection import Collection  # noqa
  File "/Users/user/miniconda2/lib/python2.7/site-packages/invoke/collection.py", line 7, in <module>
    from .config import merge_dicts
  File "/Users/user/miniconda2/lib/python2.7/site-packages/invoke/config.py", line 11, in <module>
    from .vendor import yaml2 as yaml
ImportError: cannot import name yaml2
@alfakini alfakini referenced this issue in Parsely/streamparse Feb 5, 2016
Closed

ImportError: cannot import name yaml2 #223

@aaron-trout

+1 :D

@RazerM
RazerM commented Feb 5, 2016

The problem is that wheels don't execute setup.py, which excludes either yaml2 or yaml3 at runtime. So we can surmise that the wheel was built on Python 3.

You can get around this by disabling the wheel when installing invoke with pip

pip install --no-binary invoke invoke
@thedrow
Contributor
thedrow commented Feb 5, 2016

@RazerM Workaround works. Thanks
Is the blame on the wheel package? Can we workaround it while still using wheels?

@RazerM
RazerM commented Feb 5, 2016

I looked at the wheel for 0.12.0 and it has both yaml2 and yaml3 included. I don't know how it was generated unless it was done before the code to exclude yaml2 or yaml3 was added.

@jenisys
jenisys commented Feb 7, 2016

REGARDING: wheel
The wheel may be currently the problem, but the fault location is actually in the "setup.py" file which is lacking the necessary description when you build/publish wheels. You need something like the following in "setup.py" to build your "universal" wheels correctly:

# -- FILE: setup.py
...
setup(
    ...
    extras_require={
        # -- SUPPORT-WHEELS: Extra packages for Python2.6
        # SEE: https://bitbucket.org/pypa/wheel/ , CHANGES.txt (v0.24.0)
        ':python_version=="2.7"': py27_extra,
        ':python_version=="3.3"': py33_extra,
    },
    ...
)

In case of invoke, the following line is the problem:

# -- FILE: setup.py
# PACKAGING-/PUBLISHER-VIEW, but not necessarily PACKAGE-INSTALLER VIEW:
exclude = ['*.yaml3' if sys.version_info[0] == 2 else '*.yaml2']

If you build universal wheels for both python2 and python3, the wheel must include everything. Even stuff that is not needed for python3, for example. But it would be nice, if the wheel would only the install parts (meaning either yaml2 or yaml3 variant).

@bitprophet
Member

Well that's great :( Yes, this is due to a "fix" added with that exclude line recently, in...#254 / #255 (though there had been reports of the earlier issue prior). Thanks to all for reporting & digging!

The problem of universal wheels makes sense offhand. My memory is rusty but I don't think we actually need universal wheels, they are just a convenience for projects where the 2 + 3 versions are identical, which is no longer the case for us (or rather, it causes the problems reported in #254).

I'm updating my packaging tasks to do some basic testing of build & install at CI time, so I'll try to replicate it there & test out some fixes, either non-universal wheels (which I think means I'll need to run the build task under both Python versions? will see) or the extras_require that @jenisys dug up (thanks!).

@bitprophet
Member

Also, upon double-checking #255, I'm able to recreate the errors there, but there's no actual failure, just ugly-looking output. So it's possible the best solution is to just roll that back and add an FAQ stating "hey, you may get these scary errors thanks to pyyaml, but if you see 'success' at the end, don't worry". Hopefully we can get the best of both worlds though.

@bitprophet
Member

This is getting confusing fast, in part due to the two conflicting errors (old and new) & the difficulty in reproducing them reliably.

  • The universal .whl on PyPI for 0.12.1 only seems to have the yaml3 subtree. This is what's causing all the problems for users in this thread, presumably. I can confirm that installing that wheel under Python 2 causes inv to explode.
  • The same wheel for 0.12.0 has both YAML subtrees.
  • When I started poking this, I thought #255 was merged between 0.12.0 and 0.12.1, but this is not the case, it's in the changelog for 0.12.0. That means #255 is probably not the problem.
  • Building a wheel using my normal build method, under either version of Python, currently results in a .whl that has both yaml2 and yaml3.
  • Therefore, somehow my build & distribution of 0.12.1 was abnormal and omitted the yaml2 tree. I have no idea how that could have happened. Perhaps it's related to how I was testing out improved build scripts at the time.

Additionally, trying to apply @jenisys' approach of extras_require doesn't appear to make a difference - it feels like it only applies to regular, external dependencies (the same type that are placed into install_requires) and our issue here is which sub-packages of our source tree end up in the archives. I haven't dug super deep into this yet but that's what I'm getting from the wheel docs, google results, etc.


So for now I am chalking this up to me screwing up the 0.12.1 wheel build somehow. I'm going to publish 0.12.2 tonight and doublecheck the distributed result for existence of both yaml trees.

When I do I will comment here and ask all that posted to please try installing 0.12.2 and report your results.

Thanks!

@bitprophet
Member

0.12.2 is on PyPI now, and has the below contents. Please report back when you can @ojos @raimonesteve @denisgarci @noirbizarre @Benoss @jenisys @RazerM @thedrow @aaron-trout @alfakini !

Wheel:

Archive:  /Users/jforcier/Downloads/invoke-0.12.2-py2.py3-none-any.whl
  Length     Date   Time    Name
 --------    ----   ----    ----
     1074  02-07-16 20:02   invoke/__init__.py
       81  02-07-16 21:34   invoke/_version.py
    15287  02-07-16 19:52   invoke/cli.py
    15319  01-12-16 18:46   invoke/collection.py
     3938  02-07-16 20:02   invoke/complete.py
    21790  02-07-16 20:02   invoke/config.py
     2215  12-11-15 09:14   invoke/context.py
     4013  09-17-15 19:17   invoke/env.py
     5198  02-07-16 20:02   invoke/exceptions.py
     8711  02-07-16 20:02   invoke/executor.py
     3579  02-07-16 20:02   invoke/loader.py
      199  02-07-16 20:02   invoke/main.py
     5509  02-07-16 20:02   invoke/platform.py
    21175  02-07-16 20:02   invoke/program.py
    36626  02-07-16 20:02   invoke/runners.py
    16588  02-07-16 20:02   invoke/tasks.py
     2844  02-07-16 20:02   invoke/util.py
      198  02-07-16 20:02   invoke/parser/__init__.py
     3278  03-30-15 15:26   invoke/parser/argument.py
     9145  02-07-16 20:02   invoke/parser/context.py
    13644  02-07-16 20:02   invoke/parser/parser.py
        0  06-23-13 10:40   invoke/vendor/__init__.py
    23143  09-18-15 10:27   invoke/vendor/six.py
      196  06-23-13 10:40   invoke/vendor/fluidity/__init__.py
      135  06-23-13 10:40   invoke/vendor/fluidity/backwardscompat.py
     8686  06-23-13 10:40   invoke/vendor/fluidity/machine.py
     1073  07-16-13 20:17   invoke/vendor/lexicon/__init__.py
     3395  07-16-13 20:17   invoke/vendor/lexicon/alias_dict.py
      332  06-23-13 10:40   invoke/vendor/lexicon/attribute_dict.py
     9776  03-30-15 15:26   invoke/vendor/yaml2/__init__.py
     4921  03-30-15 15:26   invoke/vendor/yaml2/composer.py
    25145  03-30-15 15:26   invoke/vendor/yaml2/constructor.py
     3290  03-30-15 15:26   invoke/vendor/yaml2/cyaml.py
     2719  03-30-15 15:26   invoke/vendor/yaml2/dumper.py
    43298  03-30-15 15:26   invoke/vendor/yaml2/emitter.py
     2559  03-30-15 15:26   invoke/vendor/yaml2/error.py
     2445  03-30-15 15:26   invoke/vendor/yaml2/events.py
     1132  03-30-15 15:26   invoke/vendor/yaml2/loader.py
     1440  03-30-15 15:26   invoke/vendor/yaml2/nodes.py
    25542  03-30-15 15:26   invoke/vendor/yaml2/parser.py
     6746  03-30-15 15:26   invoke/vendor/yaml2/reader.py
    17642  03-30-15 15:26   invoke/vendor/yaml2/representer.py
     8972  03-30-15 15:26   invoke/vendor/yaml2/resolver.py
    52630  03-30-15 15:26   invoke/vendor/yaml2/scanner.py
     4171  03-30-15 15:26   invoke/vendor/yaml2/serializer.py
     2573  03-30-15 15:26   invoke/vendor/yaml2/tokens.py
     9607  03-30-15 15:26   invoke/vendor/yaml3/__init__.py
     4881  03-30-15 15:26   invoke/vendor/yaml3/composer.py
    25554  03-30-15 15:26   invoke/vendor/yaml3/constructor.py
     3294  03-30-15 15:26   invoke/vendor/yaml3/cyaml.py
     2723  03-30-15 15:26   invoke/vendor/yaml3/dumper.py
    42954  03-30-15 15:26   invoke/vendor/yaml3/emitter.py
     2533  03-30-15 15:26   invoke/vendor/yaml3/error.py
     2445  03-30-15 15:26   invoke/vendor/yaml3/events.py
     1138  03-30-15 15:26   invoke/vendor/yaml3/loader.py
     1440  03-30-15 15:26   invoke/vendor/yaml3/nodes.py
    25495  03-30-15 15:26   invoke/vendor/yaml3/parser.py
     6854  03-30-15 15:26   invoke/vendor/yaml3/reader.py
    13528  03-30-15 15:26   invoke/vendor/yaml3/representer.py
     8820  03-30-15 15:26   invoke/vendor/yaml3/resolver.py
    51879  03-30-15 15:26   invoke/vendor/yaml3/scanner.py
     4165  03-30-15 15:26   invoke/vendor/yaml3/serializer.py
     2573  03-30-15 15:26   invoke/vendor/yaml3/tokens.py
     2051  02-07-16 21:36   invoke-0.12.2.dist-info/DESCRIPTION.rst
       82  02-07-16 21:36   invoke-0.12.2.dist-info/entry_points.txt
     1551  02-07-16 21:36   invoke-0.12.2.dist-info/metadata.json
       47  04-14-15 09:19   invoke-0.12.2.dist-info/pbr.json
        7  02-07-16 21:36   invoke-0.12.2.dist-info/top_level.txt
      110  02-07-16 21:36   invoke-0.12.2.dist-info/WHEEL
     3384  02-07-16 21:36   invoke-0.12.2.dist-info/METADATA
     6003  02-07-16 21:36   invoke-0.12.2.dist-info/RECORD
 --------                   -------
   665490                   71 files

Tar.gz:

drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/
-rw-r--r--  0 jforcier staff     336 Feb  7 20:02 invoke-0.12.2/dev-requirements.txt
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/
-rw-r--r--  0 jforcier staff    1314 Feb  7 20:02 invoke-0.12.2/LICENSE
-rw-r--r--  0 jforcier staff     235 Feb  7 21:11 invoke-0.12.2/MANIFEST.in
-rw-r--r--  0 jforcier staff    3852 Feb  7 21:36 invoke-0.12.2/PKG-INFO
-rw-r--r--  0 jforcier staff    1924 Feb  7 20:02 invoke-0.12.2/README.rst
-rw-r--r--  0 jforcier staff     204 Feb  7 21:36 invoke-0.12.2/setup.cfg
-rw-r--r--  0 jforcier staff    2500 Feb  7 20:51 invoke-0.12.2/setup.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/
-rw-r--r--  0 jforcier staff     421 Feb  7 20:02 invoke-0.12.2/tasks-requirements.txt
-rw-r--r--  0 jforcier staff     514 Feb  7 20:02 invoke-0.12.2/tasks.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/
-rw-r--r--  0 jforcier staff    5870 Feb  7 20:02 invoke-0.12.2/tests/_util.py
-rw-r--r--  0 jforcier staff    4410 Feb  7 20:02 invoke-0.12.2/tests/cli.py
-rw-r--r--  0 jforcier staff   18116 Feb  7 20:02 invoke-0.12.2/tests/collection.py
-rw-r--r--  0 jforcier staff    3803 Feb  7 20:02 invoke-0.12.2/tests/completion.py
-rw-r--r--  0 jforcier staff   20444 Feb  7 20:02 invoke-0.12.2/tests/config.py
-rw-r--r--  0 jforcier staff    1259 Dec 21 16:22 invoke-0.12.2/tests/context.py
-rw-r--r--  0 jforcier staff    9223 Feb  7 20:02 invoke-0.12.2/tests/executor.py
-rw-r--r--  0 jforcier staff    2929 Feb  7 20:02 invoke-0.12.2/tests/init.py
-rw-r--r--  0 jforcier staff    2601 Feb  7 20:02 invoke-0.12.2/tests/loader.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/parser/
-rw-r--r--  0 jforcier staff    1680 Feb  7 20:02 invoke-0.12.2/tests/platform.py
-rw-r--r--  0 jforcier staff   21659 Feb  7 20:02 invoke-0.12.2/tests/program.py
-rw-r--r--  0 jforcier staff   29178 Feb  7 20:02 invoke-0.12.2/tests/runners.py
-rw-r--r--  0 jforcier staff   11204 Feb  7 20:02 invoke-0.12.2/tests/tasks.py
-rw-r--r--  0 jforcier staff    4773 Mar 30  2015 invoke-0.12.2/tests/parser/argument.py
-rw-r--r--  0 jforcier staff   11940 Mar 30  2015 invoke-0.12.2/tests/parser/context.py
-rw-r--r--  0 jforcier staff   14505 Feb  7 20:02 invoke-0.12.2/tests/parser/parser.py
-rw-r--r--  0 jforcier staff      76 Mar 30  2015 invoke-0.12.2/tests/_support/alias_sorting.py
-rw-r--r--  0 jforcier staff     361 Mar 30  2015 invoke-0.12.2/tests/_support/autoprint.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/
-rw-r--r--  0 jforcier staff     407 Dec 11 09:14 invoke-0.12.2/tests/_support/contextualized.py
-rw-r--r--  0 jforcier staff      98 Sep 17 19:17 invoke-0.12.2/tests/_support/debugging.py
-rw-r--r--  0 jforcier staff     730 Mar 30  2015 invoke-0.12.2/tests/_support/decorator.py
-rw-r--r--  0 jforcier staff     113 Jun 23  2013 invoke-0.12.2/tests/_support/decorator_multi_default.py
-rw-r--r--  0 jforcier staff     206 Mar 30  2015 invoke-0.12.2/tests/_support/deeper_ns_list.py
-rw-r--r--  0 jforcier staff     496 Aug 25  2014 invoke-0.12.2/tests/_support/depth_first.py
-rw-r--r--  0 jforcier staff     276 Aug 25  2014 invoke-0.12.2/tests/_support/docstrings.py
-rw-r--r--  0 jforcier staff       7 Aug 25  2014 invoke-0.12.2/tests/_support/empty.py
-rw-r--r--  0 jforcier staff     227 Jun 23  2013 invoke-0.12.2/tests/_support/explicit_root.py
-rw-r--r--  0 jforcier staff     215 Mar 30  2015 invoke-0.12.2/tests/_support/foo.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/ignoreme/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/implicit/
-rw-r--r--  0 jforcier staff     482 May 18  2015 invoke-0.12.2/tests/_support/integration.py
-rw-r--r--  0 jforcier staff     183 Feb  7 20:02 invoke-0.12.2/tests/_support/namespacing.py
-rw-r--r--  0 jforcier staff      44 Mar 30  2015 invoke-0.12.2/tests/_support/oops.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/package/
-rw-r--r--  0 jforcier staff     173 Mar 30  2015 invoke-0.12.2/tests/_support/simple_ns_list.py
-rw-r--r--  0 jforcier staff      84 May  6  2014 invoke-0.12.2/tests/_support/subcollection_task_name.py
-rw-r--r--  0 jforcier staff       0 Jun 23  2013 invoke-0.12.2/tests/_support/package/__init__.py
-rw-r--r--  0 jforcier staff      54 Jun 23  2013 invoke-0.12.2/tests/_support/package/module.py
-rw-r--r--  0 jforcier staff      65 Jul 16  2013 invoke-0.12.2/tests/_support/implicit/tasks.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/ignoreme/ignoremetoo/
-rw-r--r--  0 jforcier staff       0 Aug 25  2014 invoke-0.12.2/tests/_support/ignoreme/ignoremetoo/.no
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/all-three/
-rw-r--r--  0 jforcier staff     172 Apr  1  2015 invoke-0.12.2/tests/_support/configs/collection.py
-rw-r--r--  0 jforcier staff      18 Mar 30  2015 invoke-0.12.2/tests/_support/configs/echo.yaml
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/json/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/json-and-python/
-rw-r--r--  0 jforcier staff      23 Mar 30  2015 invoke-0.12.2/tests/_support/configs/no-dedupe.yaml
-rw-r--r--  0 jforcier staff      19 Mar 30  2015 invoke-0.12.2/tests/_support/configs/no-echo.yaml
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/python/
-rw-r--r--  0 jforcier staff      97 Mar 30  2015 invoke-0.12.2/tests/_support/configs/runtime.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/tests/_support/configs/yaml/
-rw-r--r--  0 jforcier staff     135 Jan 12 18:46 invoke-0.12.2/tests/_support/configs/yaml/explicit.py
-rw-r--r--  0 jforcier staff      15 Mar 30  2015 invoke-0.12.2/tests/_support/configs/yaml/invoke.yaml
-rw-r--r--  0 jforcier staff      97 Mar 30  2015 invoke-0.12.2/tests/_support/configs/yaml/tasks.py
-rw-r--r--  0 jforcier staff      18 Mar 30  2015 invoke-0.12.2/tests/_support/configs/python/invoke.py
-rw-r--r--  0 jforcier staff      46 Mar 30  2015 invoke-0.12.2/tests/_support/configs/json-and-python/invoke.json
-rw-r--r--  0 jforcier staff      44 Mar 30  2015 invoke-0.12.2/tests/_support/configs/json-and-python/invoke.py
-rw-r--r--  0 jforcier staff      19 Mar 30  2015 invoke-0.12.2/tests/_support/configs/json/invoke.json
-rw-r--r--  0 jforcier staff      46 Mar 30  2015 invoke-0.12.2/tests/_support/configs/all-three/invoke.json
-rw-r--r--  0 jforcier staff      44 Mar 30  2015 invoke-0.12.2/tests/_support/configs/all-three/invoke.py
-rw-r--r--  0 jforcier staff      40 Mar 30  2015 invoke-0.12.2/tests/_support/configs/all-three/invoke.yaml
-rw-r--r--  0 jforcier staff     280 Aug 25  2014 invoke-0.12.2/sites/alpha.rst
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/
-rw-r--r--  0 jforcier staff     966 Feb  7 20:02 invoke-0.12.2/sites/shared_conf.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/www/
-rw-r--r--  0 jforcier staff   32768 Feb  7 21:34 invoke-0.12.2/sites/www/.changelog.rst.swp
-rw-r--r--  0 jforcier staff   20985 Feb  7 21:34 invoke-0.12.2/sites/www/changelog.rst
-rw-r--r--  0 jforcier staff     692 Aug 25  2014 invoke-0.12.2/sites/www/conf.py
-rw-r--r--  0 jforcier staff     501 Aug 25  2014 invoke-0.12.2/sites/www/contact.rst
-rw-r--r--  0 jforcier staff    1411 Mar 30  2015 invoke-0.12.2/sites/www/development.rst
-rw-r--r--  0 jforcier staff    6428 Feb  7 20:02 invoke-0.12.2/sites/www/faq.rst
-rw-r--r--  0 jforcier staff     592 Aug 25  2014 invoke-0.12.2/sites/www/index.rst
-rw-r--r--  0 jforcier staff    1171 Feb  7 20:02 invoke-0.12.2/sites/www/installing.rst
-rw-r--r--  0 jforcier staff    1865 Aug 25  2014 invoke-0.12.2/sites/www/prior_art.rst
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/_static/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/api/
-rw-r--r--  0 jforcier staff    4319 Feb  7 20:02 invoke-0.12.2/sites/docs/cli.rst
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/concepts/
-rw-r--r--  0 jforcier staff     419 Aug 25  2014 invoke-0.12.2/sites/docs/conf.py
-rw-r--r--  0 jforcier staff    7434 Feb  7 20:02 invoke-0.12.2/sites/docs/getting_started.rst
-rw-r--r--  0 jforcier staff     954 Feb  7 20:02 invoke-0.12.2/sites/docs/index.rst
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/concepts/cli/
-rw-r--r--  0 jforcier staff     401 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/cli.rst
-rw-r--r--  0 jforcier staff   14768 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/configuration.rst
-rw-r--r--  0 jforcier staff    2594 Apr 29  2015 invoke-0.12.2/sites/docs/concepts/context.rst
-rw-r--r--  0 jforcier staff    4816 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/execution.rst
-rw-r--r--  0 jforcier staff    5920 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/library.rst
-rw-r--r--  0 jforcier staff    2734 Aug 25  2014 invoke-0.12.2/sites/docs/concepts/loading.rst
-rw-r--r--  0 jforcier staff    9390 Jan 12 20:17 invoke-0.12.2/sites/docs/concepts/namespaces.rst
-rw-r--r--  0 jforcier staff    2128 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/responses.rst
-rw-r--r--  0 jforcier staff    7728 Jan 12 19:50 invoke-0.12.2/sites/docs/concepts/cli/background.rst
-rw-r--r--  0 jforcier staff    6167 Feb  7 20:02 invoke-0.12.2/sites/docs/concepts/cli/intro.rst
-rw-r--r--  0 jforcier staff      63 Apr 29  2015 invoke-0.12.2/sites/docs/api/__init__.rst
-rw-r--r--  0 jforcier staff     156 Aug 25  2014 invoke-0.12.2/sites/docs/api/collection.rst
-rw-r--r--  0 jforcier staff      64 Dec 11 09:14 invoke-0.12.2/sites/docs/api/config.rst
-rw-r--r--  0 jforcier staff      68 Aug 25  2014 invoke-0.12.2/sites/docs/api/context.rst
-rw-r--r--  0 jforcier staff     187 Jan 12 18:46 invoke-0.12.2/sites/docs/api/exceptions.rst
-rw-r--r--  0 jforcier staff      80 Aug 25  2014 invoke-0.12.2/sites/docs/api/executor.rst
-rw-r--r--  0 jforcier staff     229 Aug 25  2014 invoke-0.12.2/sites/docs/api/loader.rst
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/sites/docs/api/parser/
-rw-r--r--  0 jforcier staff     151 Aug 25  2014 invoke-0.12.2/sites/docs/api/parser.rst
-rw-r--r--  0 jforcier staff      72 Aug 26  2014 invoke-0.12.2/sites/docs/api/platform.rst
-rw-r--r--  0 jforcier staff      68 Feb  7 20:02 invoke-0.12.2/sites/docs/api/program.rst
-rw-r--r--  0 jforcier staff      96 Dec 11 09:14 invoke-0.12.2/sites/docs/api/runners.rst
-rw-r--r--  0 jforcier staff      60 Aug 25  2014 invoke-0.12.2/sites/docs/api/tasks.rst
-rw-r--r--  0 jforcier staff      56 Aug 25  2014 invoke-0.12.2/sites/docs/api/util.rst
-rw-r--r--  0 jforcier staff      79 Aug 25  2014 invoke-0.12.2/sites/docs/api/parser/argument.rst
-rw-r--r--  0 jforcier staff      75 Aug 25  2014 invoke-0.12.2/sites/docs/api/parser/context.rst
-rw-r--r--  0 jforcier staff      71 Mar 30  2015 invoke-0.12.2/sites/docs/api/parser/parser.rst
-rw-r--r--  0 jforcier staff   15260 Aug 25  2014 invoke-0.12.2/sites/docs/_static/rtd.css
-rw-r--r--  0 jforcier staff       1 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/dependency_links.txt
-rw-r--r--  0 jforcier staff      82 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/entry_points.txt
-rw-r--r--  0 jforcier staff      47 Apr 14  2015 invoke-0.12.2/invoke.egg-info/pbr.json
-rw-r--r--  0 jforcier staff    3852 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/PKG-INFO
-rw-r--r--  0 jforcier staff    4700 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/SOURCES.txt
-rw-r--r--  0 jforcier staff       7 Feb  7 21:36 invoke-0.12.2/invoke.egg-info/top_level.txt
-rw-r--r--  0 jforcier staff    1074 Feb  7 20:02 invoke-0.12.2/invoke/__init__.py
-rw-r--r--  0 jforcier staff      81 Feb  7 21:34 invoke-0.12.2/invoke/_version.py
-rw-r--r--  0 jforcier staff   15319 Jan 12 18:46 invoke-0.12.2/invoke/collection.py
-rw-r--r--  0 jforcier staff    3938 Feb  7 20:02 invoke-0.12.2/invoke/complete.py
-rw-r--r--  0 jforcier staff   21790 Feb  7 20:02 invoke-0.12.2/invoke/config.py
-rw-r--r--  0 jforcier staff    2215 Dec 11 09:14 invoke-0.12.2/invoke/context.py
-rw-r--r--  0 jforcier staff    4013 Sep 17 19:17 invoke-0.12.2/invoke/env.py
-rw-r--r--  0 jforcier staff    5198 Feb  7 20:02 invoke-0.12.2/invoke/exceptions.py
-rw-r--r--  0 jforcier staff    8711 Feb  7 20:02 invoke-0.12.2/invoke/executor.py
-rw-r--r--  0 jforcier staff    3579 Feb  7 20:02 invoke-0.12.2/invoke/loader.py
-rw-r--r--  0 jforcier staff     199 Feb  7 20:02 invoke-0.12.2/invoke/main.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/parser/
-rw-r--r--  0 jforcier staff    5509 Feb  7 20:02 invoke-0.12.2/invoke/platform.py
-rw-r--r--  0 jforcier staff   21175 Feb  7 20:02 invoke-0.12.2/invoke/program.py
-rw-r--r--  0 jforcier staff   36626 Feb  7 20:02 invoke-0.12.2/invoke/runners.py
-rw-r--r--  0 jforcier staff   16588 Feb  7 20:02 invoke-0.12.2/invoke/tasks.py
-rw-r--r--  0 jforcier staff    2844 Feb  7 20:02 invoke-0.12.2/invoke/util.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/vendor/
-rw-r--r--  0 jforcier staff       0 Jun 23  2013 invoke-0.12.2/invoke/vendor/__init__.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/vendor/fluidity/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/vendor/lexicon/
-rw-r--r--  0 jforcier staff   23143 Sep 18 10:27 invoke-0.12.2/invoke/vendor/six.py
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/vendor/yaml2/
drwxr-xr-x  0 jforcier staff       0 Feb  7 21:36 invoke-0.12.2/invoke/vendor/yaml3/
-rw-r--r--  0 jforcier staff    9607 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/__init__.py
-rw-r--r--  0 jforcier staff    4881 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/composer.py
-rw-r--r--  0 jforcier staff   25554 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/constructor.py
-rw-r--r--  0 jforcier staff    3294 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/cyaml.py
-rw-r--r--  0 jforcier staff    2723 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/dumper.py
-rw-r--r--  0 jforcier staff   42954 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/emitter.py
-rw-r--r--  0 jforcier staff    2533 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/error.py
-rw-r--r--  0 jforcier staff    2445 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/events.py
-rw-r--r--  0 jforcier staff    1138 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/loader.py
-rw-r--r--  0 jforcier staff    1440 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/nodes.py
-rw-r--r--  0 jforcier staff   25495 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/parser.py
-rw-r--r--  0 jforcier staff    6854 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/reader.py
-rw-r--r--  0 jforcier staff   13528 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/representer.py
-rw-r--r--  0 jforcier staff    8820 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/resolver.py
-rw-r--r--  0 jforcier staff   51879 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/scanner.py
-rw-r--r--  0 jforcier staff    4165 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/serializer.py
-rw-r--r--  0 jforcier staff    2573 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml3/tokens.py
-rw-r--r--  0 jforcier staff    9776 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/__init__.py
-rw-r--r--  0 jforcier staff    4921 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/composer.py
-rw-r--r--  0 jforcier staff   25145 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/constructor.py
-rw-r--r--  0 jforcier staff    3290 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/cyaml.py
-rw-r--r--  0 jforcier staff    2719 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/dumper.py
-rw-r--r--  0 jforcier staff   43298 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/emitter.py
-rw-r--r--  0 jforcier staff    2559 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/error.py
-rw-r--r--  0 jforcier staff    2445 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/events.py
-rw-r--r--  0 jforcier staff    1132 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/loader.py
-rw-r--r--  0 jforcier staff    1440 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/nodes.py
-rw-r--r--  0 jforcier staff   25542 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/parser.py
-rw-r--r--  0 jforcier staff    6746 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/reader.py
-rw-r--r--  0 jforcier staff   17642 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/representer.py
-rw-r--r--  0 jforcier staff    8972 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/resolver.py
-rw-r--r--  0 jforcier staff   52630 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/scanner.py
-rw-r--r--  0 jforcier staff    4171 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/serializer.py
-rw-r--r--  0 jforcier staff    2573 Mar 30  2015 invoke-0.12.2/invoke/vendor/yaml2/tokens.py
-rw-r--r--  0 jforcier staff    1073 Jul 16  2013 invoke-0.12.2/invoke/vendor/lexicon/__init__.py
-rw-r--r--  0 jforcier staff    3395 Jul 16  2013 invoke-0.12.2/invoke/vendor/lexicon/alias_dict.py
-rw-r--r--  0 jforcier staff     332 Jun 23  2013 invoke-0.12.2/invoke/vendor/lexicon/attribute_dict.py
-rw-r--r--  0 jforcier staff     196 Jun 23  2013 invoke-0.12.2/invoke/vendor/fluidity/__init__.py
-rw-r--r--  0 jforcier staff     135 Jun 23  2013 invoke-0.12.2/invoke/vendor/fluidity/backwardscompat.py
-rw-r--r--  0 jforcier staff    8686 Jun 23  2013 invoke-0.12.2/invoke/vendor/fluidity/machine.py
-rw-r--r--  0 jforcier staff     198 Feb  7 20:02 invoke-0.12.2/invoke/parser/__init__.py
-rw-r--r--  0 jforcier staff    3278 Mar 30  2015 invoke-0.12.2/invoke/parser/argument.py
-rw-r--r--  0 jforcier staff    9145 Feb  7 20:02 invoke-0.12.2/invoke/parser/context.py
-rw-r--r--  0 jforcier staff   13644 Feb  7 20:02 invoke-0.12.2/invoke/parser/parser.py
@denisgarci

Version 0.12.2 is working great for me on Python 2. Thank you!

@noirbizarre

I confirm, 0.12.2 works like a charm ! Thanks !

@ojos
ojos commented Feb 8, 2016

Version 0.12.2 is working. Thank you.

@ojos ojos closed this Feb 8, 2016
@thedrow
Contributor
thedrow commented Feb 8, 2016

It works. Thanks!

@RazerM
RazerM commented Feb 8, 2016

@bitprophet I was surprised to see that building the wheel would include yaml2 and yaml3, so I tried running inv release.build on both Python 2 and Python 3.

In both cases, only the yaml2 or yaml3 version gets bundled bot not both. However, if I do not clean up the build/ dir before changing between my Python 2 / Python 3 virtual environments then the leftover yaml2 from building in the Python 2 env gets included when I build in the Python 3 env.

@bitprophet
Member

@RazerM aha! that would absolutely cause the problem with the 0.12.1 release (specifically, me having nuked the build dir in between build attempts). God bless nondeterministic builds :( </sarcasm> Thanks for the attention to detail ❤️

I've been considering trying to leverage Travis or similar for release builds to avoid these occasional "local workstation state" problems (this is not the first), so I may poke that.

Otherwise I'll update release.publish so it creates & uses a temp virtualenv of the non-active Python version (to publish two non-universal wheels), or similar hacks. (Or, as above, perhaps just undo the exclude change from #255...)

@bitprophet bitprophet added this to the 0.12.3 milestone Feb 8, 2016
@bitprophet bitprophet reopened this Feb 8, 2016
@bitprophet
Member

FTR Travis does offer PyPI deployment (https://docs.travis-ci.com/user/deployment/pypi) but I realized it clashes with the fact that I recently started GPG-signing releases - that part can't be automated.

Was also going to say it's unclear whether it uploads securely, but it does seem to use twine (https://github.com/travis-ci/dpl/blob/9be1699537d5829b43a81103339be32d7936e6dc/lib/dpl/provider/pypi.rb#L97) so if I didn't care about GPG it might actually be pretty sweet. Maybe something for later if I decide GPG signing isn't worth it to enough users.

So for now, going to try extending invocations.packaging to result in a both-yamls wheel instead.

@bitprophet
Member
  • Confirmed that with a clean $CWD/build directory, python setup.py sdist creates a tree in build/lib/invoke that only has vendor/yaml2 under Python 2, and that a subsequent run of the same command under Python 3 ends up layering vendor/yaml3 next to it. So that confirms the theory.
  • Wasn't entirely sure what the relationship was between setup.py build (which offers a -b <dir> option) and setup.py sdist (which does not), then realized, duh, build is only being invoked by bdist_wheel - a vanilla setup.py sdist never even creates a build/ dir at all.
  • Unfortunately, bdist_wheel also doesn't let you control the location of the build directory, though it does let you control the "bdist" directory (which is wholly temporary regardless) and of course the final dist dir.
    • It does offer --skip-build (which does...skip the build step, resulting in a totally useless wheel with no .py files in it) implying that it's indirectly triggering the equivalent of setup.py build - meaning I may be able to just do an explicit build in the same command (tho I dunno if it will then do ANOTHER build on its own?)
    • EDIT: yes, seems like python setup.py build <build-opts including -b> bdist_wheel does have the bdist_wheel step honor the build step's directory option; running python setup.py build -b foobar bdist_wheel shows the wheel using files from foobar, and no build/ directory is ever created.
  • So, I can change things to "cleanly" build in a tempdir (much as I already do for the actual dist files) - that should remove this class of error for the future.
  • Then what remains is how to leverage this...either the explicit "doubled-up" build generating both yaml folders prior to bdist creation, or explicitly non-universal builds created separately. Poking...
@bitprophet
Member

Presently, since we don't have any C extension dependencies, I feel distributing 2x wheels (http://python-packaging-user-guide.readthedocs.org/en/latest/distributing/#pure-python-wheels) is the cleanest approach, hardcoding the 'accidental' doubled-up yaml packages as part of the process seems gross.

Open to arguments though but I'm going to move ahead with adding "execute the build step under Python 3 afterwards" to my process.

@bitprophet bitprophet added a commit to pyinvoke/invocations that referenced this issue Feb 8, 2016
@bitprophet bitprophet Explicitly build in the temp dir as well as creating dists there.
Avoids a class of issues encountered with dirty local build/ dirs

See e.g. pyinvoke/invoke#319
4b1cfaf
@bitprophet
Member

Got that working pretty well locally; it not only expands the tempdir to be used for both build and dist, but cleans the build subdir by default each time (I guess I could also use nested tempdirs or something, but, effort).

Then 2x invocations of build(), one per interpreter, and we have two shiny wheels with the right yaml tree in each.

https://github.com/pyinvoke/invocations/blob/94feb0bc626686b55d13dcbd33c0bb5f7ac235c1/invocations/packaging.py#L282-L374


Having Travis test this out end-to-end is a bit wonky, though. I've fought past a couple minor blockers, and there's at least a few more in my way.

It also means that the build check step runs on every matrix cell; not sure if there's a nice way to say "only run on " without prefixing a ton of lines with extra ugly bash logic-isms. 😕

@bitprophet
Member

Got meself a Travis Docker image so I could poke around. They have nice tidy virtualenvs set up in /home/travis/virtualenv/pythonN.N.N for various values of N, but those values will obviously change over time. Happy to just use /usr/bin/python for building the Python 2 venv under Python 3, for now, I think. Mostly just wanting to make sure it was /usr/bin/python and not something else funky.

Anyway, after the extra handful of expected stupid roadblocks...success?! https://travis-ci.org/pyinvoke/invoke/builds/107947030

Going to re-close this again...

@bitprophet bitprophet closed this Feb 9, 2016
@bitprophet bitprophet added a commit that referenced this issue Feb 9, 2016
@bitprophet bitprophet Changelog re #319 9b22644
@thedrow
Contributor
thedrow commented Feb 9, 2016

👍

@bitprophet bitprophet modified the milestone: 0.15.x Feb 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment