Credstash not runnable as an application

[ivanbrennan]

Issue description

After installing credstash with nix-env, running it raises a SyntaxError. It looks like Python is attempting to import a module from one of the wrapper bash scripts.

The bin directory has shell scripts credstash and credstash.py, each of which execs actual python code held in .credstash-wrapped and .credstash.py-wrapped. In .credstash-wrapped, it tries to load a Python module from the credstash.py shell wrapper, causing a SyntaxError.

$ nix-env --install --attr nixos.python27Packages.credstash
$ credstash setup
Traceback (most recent call last):
  File "/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin/.credstash-wrapped", line 8, in <module>
    from credstash import main
  File "/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin/credstash.py", line 2
    export PATH='/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/bin:/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin:/nix/store/m4w6rq47p317klmfiixh844jrj6l6wla-python2.7-setuptools-39.2.0/bin:/nix/store/1ahzxgr72vf1za50m34rc3m3vx79n9jq-python2.7-jmespath-0.9.0/bin:/nix/store/ps9mlc8zrd5w10im7pwd6vlfrzxbzbqv-python2.7-docutils-0.14/bin'${PATH:+':'}$PATH
              ^
SyntaxError: invalid syntax

bin/.credstash-wrapped

#!/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/bin/python2.7

# -*- coding: utf-8 -*-
import sys;import site;import functools;sys.argv[0] = '/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin/credstash';functools.reduce(lambda k, p: site.addsitedir(p, k), ['/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/lib/python2.7/site-packages','/nix/store/4lf8as80xw6x6wpbp26grmj5dxvjfd6d-python2.7-idna-2.7/lib/python2.7/site-packages','/nix/store/m4w6rq47p317klmfiixh844jrj6l6wla-python2.7-setuptools-39.2.0/lib/python2.7/site-packages','/nix/store/3js1s9557gx8xhd1a50zyv5zwkzr2kcf-python2.7-asn1crypto-0.24.0/lib/python2.7/site-packages','/nix/store/qm1ajzfan3fzy39yl1wyx7shs693rj3f-python2.7-packaging-17.1/lib/python2.7/site-packages','/nix/store/kimmvi74dnyjbl8i8xwf60z29szmxmh2-python2.7-pyparsing-2.2.0/lib/python2.7/site-packages','/nix/store/v9fvkmqs3j0dw43bhbjmakapc74jdjny-python2.7-six-1.11.0/lib/python2.7/site-packages','/nix/store/ikzm75zqzjvzkfqjsgny638ga1zr8x3y-python2.7-enum34-1.1.6/lib/python2.7/site-packages','/nix/store/9xghys844gq1lvz091m4fn2ds620k2wc-python2.7-ipaddress-1.0.18/lib/python2.7/site-packages','/nix/store/i726wa1h1831xn7i5xvdirjqmwr0mihn-python2.7-pycparser-2.18/lib/python2.7/site-packages','/nix/store/6ph0pq9inpbjm09ad8yrzh3ghar89x56-python2.7-cffi-1.11.5/lib/python2.7/site-packages','/nix/store/655sdqhcc0h3367585qfw9n55b7186jz-python2.7-cryptography-2.2.2/lib/python2.7/site-packages','/nix/store/65gj2v27ry7pkykhlhnpb9r2mqgnw5sd-python2.7-boto3-1.7.48/lib/python2.7/site-packages','/nix/store/mhlgk1jws791kx3hw7gdjhcv1f0ki08h-python2.7-botocore-1.10.48/lib/python2.7/site-packages','/nix/store/g9l0dlgz006fpdvjlnizn5qxr3g574d0-python2.7-python-dateutil-2.7.3/lib/python2.7/site-packages','/nix/store/x24bh0z3wmjpj71bnffldxk3q4qjy8nj-python2.7-setuptools_scm-2.1.0/lib/python2.7/site-packages','/nix/store/1ahzxgr72vf1za50m34rc3m3vx79n9jq-python2.7-jmespath-0.9.0/lib/python2.7/site-packages','/nix/store/lr51nbn3qgwm7aq2pq4x3b63drk9j3ry-python2.7-ply-3.11/lib/python2.7/site-packages','/nix/store/ps9mlc8zrd5w10im7pwd6vlfrzxbzbqv-python2.7-docutils-0.14/lib/python2.7/site-packages','/nix/store/bwl99y7vhh77g695ni9i3yljznvbq9w6-python2.7-ordereddict-1.1/lib/python2.7/site-packages','/nix/store/jjibsz0ypvza72m13rl6k2jkp02i2fmk-python2.7-simplejson-3.15.0/lib/python2.7/site-packages','/nix/store/z9s4p6x8r0r2b3djpyf464fybbmpn2b9-python2.7-s3transfer-0.1.13/lib/python2.7/site-packages','/nix/store/9850dqiinkks72vyb3ysab8wgnj4wa7z-python2.7-futures-3.2.0/lib/python2.7/site-packages','/nix/store/a4samxpliaszak5hs25jad47vqlhharx-python2.7-PyYAML-3.12/lib/python2.7/site-packages'], site._init_pathinfo());
import re
import sys

from credstash import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

bin/credstash.py

#! /nix/store/8zkg9ac4s4alzyf4a8kfrig1j73z66dw-bash-4.4-p23/bin/bash -e
export PATH='/nix/store/d3aa78jf7pm9lk6ad67x4f21a068dxn6-python-2.7.15/bin:/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin:/nix/store/m4w6rq47p317klmfiixh844jrj6l6wla-python2.7-setuptools-39.2.0/bin:/nix/store/1ahzxgr72vf1za50m34rc3m3vx79n9jq-python2.7-jmespath-0.9.0/bin:/nix/store/ps9mlc8zrd5w10im7pwd6vlfrzxbzbqv-python2.7-docutils-0.14/bin'${PATH:+':'}$PATH
export PYTHONNOUSERSITE='true'
exec -a "$0" "/nix/store/rd4l1qkpx4qfzamh08a701nxr34kgjzr-python2.7-credstash-1.14.0/bin/.credstash.py-wrapped"  "${extraFlagsArray[@]}" "$@"

I've repro'd with the Python 3 version as well.

Steps to reproduce

nix-env --install --attr nixos.python27Packages.credstash
credstash setup

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.54, NixOS, 18.09pre145679.dae9cf6106d (Jellyfish)
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.0.4
• channels(root): "nixos-18.09pre145679.dae9cf6106d"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

I've also repro'd on Ubuntu 18.04

[peterhoeg]

A quick thought - try building it with dontWrapPythonPrograms = true; and see if that does the trick.

[ivanbrennan]

@peterhoeg It looks like setting dontWrapPythonPrograms = true; breaks dependency lookups. I set up an overlay to try it:

self: super: {
  pythonX = self.python.override {
    packageOverrides = self': super': {
      credstash = super'.credstash.overridePythonAttrs(old: rec {
        dontWrapPythonPrograms = true;
      });
    };
  };
}

and installed:

$ nix-env --install --attr nixos.pythonX.pkgs.credstash

But now I get:

$ credstash setup
Traceback (most recent call last):
  File "/home/ivan/.nix-profile/bin/credstash", line 7, in <module>
    from credstash import main
  File "/nix/store/r4xwdkm0077c52dvqkrsrvshjl0kxq9g-python2.7-credstash-1.14.0/bin/credstash.py", line 26, in <module>
    import boto3
ImportError: No module named boto3

[peterhoeg]

Offhand no idea, sorry. Maybe @FRidh can chip in?

[FRidh]

The .py is a library and should be moved to a site packages folder.

[ivanbrennan]

Does that involve changing the credstash derivation or the way I'm using it? I'm not all that familiar with managing Python packages.

[sean-bennett112]

@ivanbrennan

It looks like I'm able to solve this with the following changes:

• set dontWrapPythonPrograms = true;
• replace buildPythonPackage with buildPythonApplication

Here's a minimal nix shell with that working, can you say whether it works for you?

shell.nix snippet:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

let
  my_credstash = callPackage ./credstash.nix {
    inherit python3;
  };
in
stdenv.mkDerivation {
  name = "credstash-test";
  buildInputs = my_credstash;
}

credstash.nix:

{ stdenv, python3 }:

with python3.pkgs;

buildPythonApplication rec {
  pname = "credstash";
  version = "1.15.0";

  src = fetchPypi {
    inherit pname version;
    sha256 = "814560f99ae2409e2c6d906d878f9dadada5d1d0a950aafb6b2c0d535291bdfb";
  };

  nativeBuildInputs = [ nose ];

  propagatedBuildInputs = [ cryptography boto3 pyyaml docutils ];

  # No tests in archive
  doCheck = false;

  # Wrapping credstash causes errors
  dontWrapPythonPrograms = true;

  meta = with stdenv.lib; {
    description = "A utility for managing secrets in the cloud using AWS KMS and DynamoDB";
    homepage = https://github.com/LuminalOSS/credstash;
    license = licenses.asl20;
  };
}

One question about this is how to upstream. I don't know enough about the Python nix machinery to know whether switching buildPythonPackage with buildPythonApplication will break things, since credstash is both a python module as well as a CLI application.

[ivanbrennan]

@sean-bennett112 Thanks for the example, it works!. Looking at the docs again I'm starting to understand the situation, and was able to write a shell.nix that references the original credstash derivation rather than re-writing it:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

let
  credstash = python3Packages.toPythonApplication (
    python3Packages.credstash.overridePythonAttrs (old: rec {
      dontWrapPythonPrograms = true;
    })
  );

in mkShell { buildInputs = [ credstash ]; }

and it still works:

$ nix-shell --pure --command 'credstash get my.secret.key'
supersecretvalue

All Python packages reside in pkgs/top-level/python-packages.nix and all applications elsewhere. In case a package is used as both a library and an application, then the package should be in pkgs/top-level/python-packages.nix since only those packages are made available for all interpreter versions.

A distinction is made between applications and libraries, however, sometimes a package is used as both. In this case the package is added as a library to python-packages.nix and as an application to all-packages.nix. To reduce duplication the toPythonApplication can be used to convert a library to an application.
The Nix expression shall use buildPythonPackage and be called from python-packages.nix. A reference shall be created from all-packages.nix to the attribute in python-packages.nix, and the toPythonApplication shall be applied to the reference:

We can see this by searching <nixpkgs> for assignments to a youtube-dl attribute:

$ cd $(grep -oP 'nixpkgs=\K[^:]+' <<< "$NIX_PATH")

$ grep --include='*.nix' --recursive 'youtube-dl = '
pkgs/top-level/python-packages.nix:  youtube-dl = callPackage ../tools/misc/youtube-dl {};
pkgs/top-level/all-packages.nix:  youtube-dl = with python3Packages; toPythonApplication youtube-dl;

And this corresponds to youtube-dl being available as both a library and an application:

$ nix-env -f '<nixpkgs>' --query --available --attr python3.pkgs.youtube-dl
python3.7-youtube-dl-2018.11.18

$ nix-env -f '<nixpkgs>' --query --available --attr youtube-dl
youtube-dl-2018.11.18

In contrast, credstash has only been provided as a library:

$ grep --include='*.nix' --recursive 'credstash = '
pkgs/top-level/python-packages.nix:  credstash = callPackage ../development/python-modules/credstash { };

$ nix-env -f '<nixpkgs>' --query --available --attr python3.pkgs.credstash
python3.7-credstash-1.15.0

$ nix-env -f '<nixpkgs>' --query --available --attr credstash
error: attribute 'credstash' in selection path 'credstash' not found

So maybe the fix is to change credstash/default.nix to include:

  dontWrapPythonPrograms = true;

and add a line to all-packages.nix:

  credstash = with python3Packages; toPythonApplication credstash;

I haven't been able to get this working in an overlay, however.

# credstash-overlay-1.nix
self: super: {
  credstash = super.python3Packages.toPythonApplication (
    super.python3Packages.credstash.overridePythonAttrs (old: rec {
      dontWrapPythonPrograms = true;
    })
  );
}

$ credstash get my.secret.key
Traceback (most recent call last):
  File "/run/current-system/sw/bin/credstash", line 7, in <module>
    from credstash import main
  File "/nix/store/8rmldlvlv1z1xl7w02dy7f5qhkzdrg8z-python3.7-credstash-1.15.0/bin/credstash.py", line 26, in <module>
    import boto3
ModuleNotFoundError: No module named 'boto3'

I've tried all combinations of self and super, as well as the more intricate approach mentioned here.

[ivanbrennan]

Well I think I figured out what was going on. I put up PR #51807 describing the issue and the fix in more detail. Maybe there's a cleaner solution, but I guess that's what the PR process is for 😁

In the meantime, here's the solution rolled into an overlay:

self: super: {
  credstash = with super.python3Packages; toPythonApplication (
    credstash.overridePythonAttrs (old: rec {
      postInstall = "rm $out/bin/credstash.py";
    })
  );
}