Flask import breaks on OSX

[mattmoor]

You can see it in the Bazel CI job here

[mattmoor]

It looks like this starts working once pip is installed on the system.

[mattmoor]

Ok, I think I understand the error now, but not sure how to fix.

Normal installation of the wheel package installs a Python command for bdist_wheel. Wheel construction seems to rely on the capacity to shell out to:

/usr/bin/python -u -c "..." bdist_wheel -d ...

However, because we don't have pip or wheel installed on the system, this assumption is bad.

I'm not sure how this would work in a virtualenv for instance, but perhaps we could make the context of a {par,py}_binary look like a virtualenv to subprocesses and somehow communicate this kind of tool installation from the whl_library to the py_binary (through py_library hops)?

@duggelz I'd appreciate your thoughts on this (no rush).

[mattmoor]

@duggelz if you are around, I'd appreciate any thoughts you might have in this space.

[mattmoor]

Here's where this is configured in setup.py: https://github.com/pypa/wheel/blob/master/setup.py#L54

[mattmoor]

The wheel's .whl has the following in metadata.json:

    "extensions": {
        "python.commands": {
            "wrap_console": {
                "wheel": "wheel.tool:main"
            }
        },
        "python.details": {
            "contacts": [
                {
                    "email": "dholth@fastmail.fm",
                    "name": "Daniel Holth",
                    "role": "author"
                }
            ],
            "document_names": {
                "description": "DESCRIPTION.rst",
                "license": "LICENSE.txt"
            },
            "project_urls": {
                "Home": "https://bitbucket.org/pypa/wheel/"
            }
        },
        "python.exports": {
            "console_scripts": {
                "wheel": "wheel.tool:main"
            },
            "distutils.commands": {
                "bdist_wheel": "wheel.bdist_wheel:bdist_wheel"
            }
        }
    },

[mattmoor]

Watching this crazy video on how virtualenv works... :)

[mattmoor]

I think this comes down to:

from pkg_resources import iter_entry_points
for entry_point in iter_entry_points(group='distutils.commands', name=None):
    print(entry_point)

Which should show:

...
bdist_wheel = wheel.bdist_wheel:bdist_wheel
...

[mattmoor]

pkg_resources seems to load package information via a convoluted process around sys.path_hooks (docs).

Exploring this in a Bazel context (py_test) I see that sys.path_hooks basically has a single hook: zipimport.zipimporter.

Interestingly pkg_resources.resource_filename works fine in py_test, but I know @duggelz has indicated pkg_resources doesn't like some of the games Bazel / Python plays.

[mattmoor]

Hmm, within the context of the version_test.py, if I add wheel==0.30.0a0 to requirements.txt and if I add:

    ws = pkg_resources.WorkingSet()
    for dist in ws:
      print str(dist)

... to the test, I get the right Distributions of everything...

I'll try putting this into a PAR, and if all else fails having that PAR invoke a little python script.

[mattmoor]

Woof, so this is py_binary vs. par_binary.

If you clone the par-kour branch on my fork:

$ bazel run examples/par:list_resources
...
INFO: Running command line: bazel-bin/examples/par/list_resources
pip 9.0.1          <-- what I bundled
wheel 0.30.0a0     <--
...
$ bazel run examples/par:list_resources.par
...
INFO: Running command line: bazel-bin/examples/par/list_resources.par
...
pip 1.5.4          <-- system installed version
...

[mattmoor]

@duggelz Is this something we could maybe fix in our PAR file's entrypoint?

[mattmoor]

It looks like the logic in py_binarys entrypoint basically takes the set of things for which we had importpath=".", builds a new PYTHONPATH environment variable, and then execs the original main script.

[mattmoor]

FYI, this matches @duggelz investigation here

[duggelz]

After various futile attempts, I think the only reasonable way to make this work is to have the .par file unpack itself at runtime.

[mattmoor]

@duggelz What's left, this seems to work for me in the rules_python OSX Bazel CI leg.

[mattmoor]

I haven't tried Flask, but I was hitting this on the extras PR, and no more.

[duggelz]

The only thing left is to add a good e2e test where we explicit create virtualenvs with various problems and run bazel inside them. But we effectively already get that by running on different OS's :)

[thundergolfer]

Will close this, as the last comment was over 3 years ago and in the interim time the pip_install repo rule has been introduced which can allow users to import Flask on OSX.