Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flask doesn't work with nix-shell #42924

Closed
AntonyBrand opened this issue Jul 3, 2018 · 19 comments

Comments

@AntonyBrand
Copy link

commented Jul 3, 2018

Issue description

When creating a project, flask messed up the $PATH horribly. Something to do with setuptools

Steps to reproduce

  1. Create default.nix:
with import<nixpkgs> {};
stdenv.mkDerivation rec {
  name = "env";
  env = buildEnv { name = name; paths = buildInputs; };
  buildInputs = [
  python3
  python36Packages.flask
  ];
}
  1. Run nix-shell, see:
[nix-shell:~/repos/the_glastonbury_map]$ ./run_server 
 * Serving Flask app "application.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
  File "/nix/store/jx467x1rpga9ic0fjp5qygd1ja3rsnli-python3.6-Flask-1.0.2/bin/flask", line 2
    export PATH='/nix/store/89ammdpygm0cx2y4184x4kfvsncc0dsr-python3-3.6.5/bin:/nix/store/jx467x1rpga9ic0fjp5qygd1ja3rsnli-python3.6-Flask-1.0.2/bin:/nix/store/amah49jgii6lbvq3igmncjmll34qm71i-python3.6-setuptools-39.2.0/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax

(note that run_server comes from @rebeccaskinner demo repo which accompanies her blog post

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the
results.

[nix-shell:~/repos/the_glastonbury_map]$ nix-shell -p nix-info --run "nix-info -m"
these paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
  /nix/store/5ljwbw3adks6cwxbc9mvvn2k0zb7b1d2-DarwinTools-1
  /nix/store/fvr8sb8ad6yjramxn4nqlyijmbhx3rk2-nix-info
copying path '/nix/store/5ljwbw3adks6cwxbc9mvvn2k0zb7b1d2-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/fvr8sb8ad6yjramxn4nqlyijmbhx3rk2-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-darwin"`
 - host os: `Darwin 15.4.0, macOS 10.11.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0.4`
 - channels(antonybrand): `"unstable-18.09pre144939.14a9ca27e69"`
 - channels(root): `"nixpkgs-18.09pre144939.14a9ca27e69"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`
@AntonyBrand

This comment has been minimized.

Copy link
Author

commented Jul 3, 2018

More help to reproduce:

AntonyBrand/py-todo@ae6b9c6

Contains a very basic flask app in src/application.py 👍

#!/usr/bin/env python

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, middle bench!!'

./run_map should work (./run_server does!) but it fails with the error above:

[nix-shell:~/repos/py-todo]$ ./run_map 
 * Serving Flask app "src/application.py" (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
  File "/nix/store/jx467x1rpga9ic0fjp5qygd1ja3rsnli-python3.6-Flask-1.0.2/bin/flask", line 2
    export PATH='/nix/store/89ammdpygm0cx2y4184x4kfvsncc0dsr-python3-3.6.5/bin:/nix/store/jx467x1rpga9ic0fjp5qygd1ja3rsnli-python3.6-Flask-1.0.2/bin:/nix/store/amah49jgii6lbvq3igmncjmll34qm71i-python3.6-setuptools-39.2.0/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax
@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 3, 2018

You should probably use python36.withPackages: https://nixos.org/nixpkgs/manual/#using-python

@AntonyBrand

This comment has been minimized.

Copy link
Author

commented Jul 3, 2018

@dotlambda Is that in addition to using python3 or instead of?

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 3, 2018

Instead of both, python and flask.

gavinrogers pushed a commit to gavinrogers/theglastonburymap that referenced this issue Jul 3, 2018
Gavin Rogers
Issue: gh issue NixOS/nixpkgs#42924
So this seems to not work when ```FLASK_DEBUG=1```
Issue reported at NixOS/nixpkgs#42924
@gavinrogers

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2018

I've reproduced this working without FLASK_ENV=1, but when I add it in, I get the error. I've got it a branch to demonstrate

If you check out that branch, nix-shell followed by ./run_works works fine. Using ./run_broken doesn't: seems like flask debug is making this go horribly wrong?

@gavinrogers

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2018

@dotlambda The manual is vague, how would you use withPackages in this situation? A comment with pseudo-code would probably suffice. Thanks!

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 3, 2018

Just put (python3.withPackages (ps: with ps; [ flask ])) into buildInputs.

gavinrogers pushed a commit to gavinrogers/theglastonburymap that referenced this issue Jul 3, 2018
Gavin Rogers
Issue NixOS/nixpkgs#42924 but with python36.withPackages
Changed the nix expression to use python36.withPackages.
Still getting bizzare PATH error when using FLASK_DEBUG=1
gavinrogers pushed a commit to gavinrogers/theglastonburymap that referenced this issue Jul 3, 2018
Gavin Rogers
Issue: NixOS/nixpkgs#42924 put python36.withPackages into buildInputs
Still not working. using ```FLASK_DEBUG=1``` breaks things
@gavinrogers

This comment has been minimized.

Copy link
Contributor

commented Jul 3, 2018

Ok I've tried it several ways, doesn't seem to improve anything.

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 3, 2018

After looking closer, it seems like Flask is trying to run its executable as a Python script (Restarting with stat). Therefore we get a SyntaxError.
This is caused by Werkzeug doing some shady stuff for determining what to execute when reloading: https://github.com/pallets/werkzeug/blob/9cdcb93e3aa31f101bdf9477e54c48c40023b110/werkzeug/_reloader.py#L59-L73
I think this should be fixed upstream. Maybe you can create a stripped-down test case for them: When not on Nixpkgs, just create a flask wrapper manually, add that to your PATH and try to run an app that causes reloading.

Anyway, in this specific case, it will work if you add

postPatch = ''
  substituteInPlace werkzeug/_reloader.py \
    --replace "rv = [sys.executable]" "return sys.argv"
'';
doCheck = false;

to the expression for Werkzeug.

@gavinrogers

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2018

@dotlambda Before I open an issue with upstream, does it make sense to find out why the PATH seems mangled in the error message? The part that doesn't look right to me is bin'${PATH:+':'}$PATH at the end.

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 4, 2018

There is no syntax error: this is perfectly valid bash. But Werkzeug is trying to execute it as Python.

@bjornfor

This comment has been minimized.

Copy link
Contributor

commented Jul 15, 2018

I think this should be fixed upstream.

Isn't this issue self-inflicted by nixpkgs wrapping python programs as bash scripts?

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 16, 2018

Isn't this issue self-inflicted by nixpkgs wrapping python programs as bash scripts?

Yes, but I think there's no way around that if we don't want Python libraries to be installed globally.

@bjornfor

This comment has been minimized.

Copy link
Contributor

commented Jul 16, 2018

The is a possible long-term solution: make wrappers in the language of the wrapped program.

@kidd

This comment has been minimized.

Copy link

commented Jul 26, 2018

Regarding #42924 (comment) , how exactly should I add that postPatch phase for werkzeug in the default.nix? I'm still very new writing nix expressions :(. Any chance at completing this default.nix from the OP?

with import<nixpkgs> {};
stdenv.mkDerivation rec {
  name = "env";
  env = buildEnv { name = name; paths = buildInputs; };
  buildInputs = [
  python3
  python36Packages.flask
  ];
}

Thanks!

@CMCDragonkai

This comment has been minimized.

Copy link
Member

commented Jul 27, 2018

Has this been reported upstream?

@dotlambda

This comment has been minimized.

Copy link
Member

commented Jul 27, 2018

@kidd That shell.nix should use python36.withPackages. You can use packageOverrides to override the werkzeug package: https://nixos.org/nixpkgs/manual/#building-packages-and-applications

@carlosdagos

This comment has been minimized.

Copy link
Member

commented Jul 31, 2018

@kidd, I think that what @dotlambda is saying is to try something like this in your default.nix:

with import <nixpkgs> {};

let myWerkzeug = python3Packages.werkzeug.overrideAttrs (oldAttrs: rec {
  postPatch = ''
  substituteInPlace werkzeug/_reloader.py \
    --replace "rv = [sys.executable]" "return sys.argv"
  '';
  doCheck = false;
}); in stdenv.mkDerivation rec {
  name = "env";
  env = buildEnv { name = name; paths = buildInputs; };
  buildInputs = [
    python3
    python3Packages.flask
    myWerkzeug
  ];
}

I'm using overrideAttrs because I think that's a shorter version. Also not using python3.withPackages. Hope it's useful 😄

@dotlambda

This comment has been minimized.

Copy link
Member

commented Aug 3, 2018

@gavinrogers Did you open an upstream issue? It's been a month. Even if not, I'll have to close this because it's actually an upstream bug.

@dotlambda dotlambda closed this Aug 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.