buildPythonPackage vs buildPythonApplication

[qknight]

Issue description (short)

when we build taiga-back with buildPythonApplication we can run the django application using ./manage.py but not using WSGI mode as the propagatedBuildInputs aren't inherited into the penv used by systemd. when we use buildPythonPackage they are inherited and it works.

FIX: our fix was to use buildPythonPackage in the end but we lost 2 days on debugging this!

REQUEST: could we add a assert or WARNING somewhere in the python abstraction to inform users of that fact? OR could we also inherit the propagatedBuildInputs from the buildPythonApplication?

i'm not very familiar with the python implementation in nixpkgs but this should probably be addressed as others will also run into this making nix look bad.

Issue description

for our WSGI application taiga.wsgi we need to create a proper environment with the PYTHONPATH set since we can't use the usual wrapper.

so our code looks like this:

  python = pkgs.python3;
  penv = with pkgs.python3Packages; with myPythonPackages;  python.buildEnv.override {
    extraLibs = [
      taiga-back
      pkgs.python3Packages.gunicorn
      pkgs.python3Packages.gevent
    ];
  };

...

    systemd.services.taiga-back = rec {
     description = "${config.uniqueName} main service (taigaio, django)";

      wantedBy      = [ "multi-user.target" ];
      after         = [ "network.target" ];

      environment = {
        PYTHONPATH = "${taigaBackConfigPkg}:${penv}/${python.sitePackages}/";
      };

      serviceConfig = {
        User = mkUniqueUser "taigaio";
        Group = mkUniqueGroup "taigaio";
        WorkingDirectory = "${config.stateDir}/www";
        #FIXME check if we can use that
        PrivateTmp = false;
        
        ExecStart = ''
          ${pkgs.python3Packages.gunicorn}/bin/gunicorn taiga.wsgi \
            -k gevent \
            -u ${mkUniqueUser "taigaio"} \
            -g ${mkUniqueGroup "taigaio"} \
            --name gunicorn-taiga \
            --pythonpath=${environment.PYTHONPATH} \
            --log-level ${if config.enableDebug then "debug" else "info"} \
            --workers ${toString config.wsgiWorkers} \
            --pid ${config.stateDir}/www/gunicorn-taiga.pid \
            --bind unix:${config.runtimeDir}/socket
        '';
        Restart = "always";
        PermissionsStartOnly = true;
        #PrivateDevices = true;
        TimeoutSec = 300; # initial ./manage.py migrate can take a while
      };
    };

taiga-back.nix

here is the buildPythonPackage vs. buildPythonApplication issue:

python3Packages.buildPythonPackage rec {
  pname = "taiga-back";
  version = "3.3.13";
  #version = "3.3.14";
  name = "${pname}-${version}";

  src = fetchFromGitHub {
    owner = "taigaio";
    repo = "taiga-back";
    rev = "${version}";
    sha256 = "1cy5ak7mw9ia9b0d1lil0s6ck87kinrmicsw61m6vpkal1slnnf7";  # 3.3.13
    #sha256 = "1csampq2g84za3fxnlfr9sffq304mz5mj9000dyk4y0720w9v1d1"; # 3.3.14
  };

  propagatedBuildInputs = with python3Packages; with myPythonPackages; [
    pkgs.gettext
    amqp
    asana
    bleach
    cairosvg
    celery
    cryptography
    cssutils
    dateutil
    diff-match-patch
    django
    (django-ipware.override { django = django; })
    (django-jinja.override { django = django; })
    (django-pglocks.override { django = django; })
    django-picklefield
    (django-sampledatahelper.override { django = django; })
    (django-sites.override { django = django; })
    (django-sr.override { django = django; })
    (djmail.override { django = django; })
    (easy-thumbnails.override { django = django; })
    fn
    gunicorn
    html5lib
    jinja2
    lxml
    markdown
    netaddr
    pillow
    premailer
    psd-tools
    psycopg2
    pygments
    pyjwkest
    pyjwt
    python_magic

Technical

nixos-version 
18.03.d6c6c7f-nixcloud_9a0eb10 (Impala)

[FRidh]

REQUEST: could we add a assert or WARNING somewhere in the python abstraction to inform users of that fact? OR could we also inherit the propagatedBuildInputs from the buildPythonApplication?

It's a pity time is wasted on this, but the answer is no, and no.

This difference with it's behavior in python.buildEnv is actually the only functional difference between the two functions, and it exists for a clear reason: a Python env should not get the site-packages from an application that just happens to be written in Python. In case of an application the fact it is written in Python is an implementation detail.

This is important so we can have e.g. a Python 3 environment that calls a Python 2 application. Note some additional issues may exist with PYTHONPATH, but that needs to be solved elsewhere.

What is needed is a line in the manual explaining this behavior.

[FRidh]

Documented at b7e0c40.

[vincentbernat]

A user may want to run a WSGI app through gunicorn. In this case, it makes sense to include the application into a buildEnv:

  issoEnv = pkgs.python3.buildEnv.override {
      extraLibs = [
        pkgs.isso
        pkgs.python3Packages.gunicorn
        pkgs.python3Packages.gevent
      ];
  };

Then, we could invoke ${issoEnv}/bin/gunicorn isso.run to run Isso through Gunicorn. Is there a better way for that? If not, would it makes sense to makes Isso a module instead of an application (and I suppose this could impact other WSGI applications).