Private package fails with `error: option --home not recognized`

[bryantbiggs]

I have a private repo on Github with a python package located under a subdirectory sdk/ that I can pip install locally with:

pip install -e git+ssh://git@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

Trying to add this to the requirments.txt, it fails with a permission denied - even with adding:

custom:
  pythonRequirements:
    dockerizePip: true
    dockerSsh: true

So I tried using a Git token to get around SSH issue for now; again it works locally with:

pip install -e git+ssh://<access-token>@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

However, now I am getting the error error: option --home not recognized

Output from SLS_DEBUG=* sls deploy --stage=dev:

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generated requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/requirements.txt in /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/B2/Documents/xxx/aws/utils/cloudwatch_events/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.6
Obtaining sdk from git+https://<access-token>@github.com/<org>/<repo>.git@master#egg=sdk&subdirectory=sdk (from -r /var/task/requirements.txt (line 1))
  Cloning https://<access-token>@github.com/<org>/<repo>.git to ./src/sdk
Installing collected packages: sdk
  Running setup.py develop for sdk
    Complete output from command /var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --home not recognized

    ----------------------------------------
Command "/var/lang/bin/python3.6 -c "import setuptools, tokenize;__file__='/var/task/src/sdk/sdk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-zkul1ujz" failed with error code 1 in /var/task/src/sdk/sdk

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:247:11)
    at installRequirementsIfNeeded (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:423:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/B2/Documents/xxx/aws/utils/cloudwatch_events/node_modules/serverless-python-requirements/lib/pip.js:502:29)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.11.4
     Serverless Version:     1.31.0

For reference, my setup.py:

#!/usr/bin/env python
"""
    Setup
    -----
    Enables the `setuptools` to build and install the library
"""

from setuptools import setup

packages = ['sdk']

requires = [
    'boto3~=1.9.2',
    'botocore~=1.12.2',
    'dnspython~=1.15.0',
    'email_validator~=1.0.3',
    'idna~=2.6',
    'requests~=2.19.1'
]
test_requirements = [
    'freezegun~=0.3.10',
    'pytest~=3.5.0',
    'pytest-cov'
]


with open('README.md', 'r', 'utf-8') as rfile:
    readme = rfile.read()

setup(
    name='sdk',
    version='0.1.0',
    description='Internal software development kit',
    long_description=readme,
    long_description_content_type='text/markdown',
    author='xxx team',
    author_email='admin@xxx.com',
    packages=packages,
    package_dir={'sdk': 'sdk'},
    python_requires=">=3.6",
    install_requires=requires,
    classifiers=[
        'Development Status :: 3 - Alpha',
        'Intended Audience :: Developers',
        'Natural Language :: English',
        'Topic :: Software Development :: Libraries',
        'Programming Language :: Python',
        'Programming Language :: Python :: 3.6'
    ],
    tests_require=test_requirements
)

serverless: 1.31.1
serverless-python-requirements: 4.2.1

Any thoughts?

[AndrewFarley]

What specifically did you add to your requirements.txt file? And if that repo is public, can you just share the actual repo url instead of hiding it so we can debug further?

[dschep]

What @AndrewFarley said. Do you have the -e in your requirements.txt? That almost definitely is an issue with the changes you made recently Andrew.

[bryantbiggs]

@dschep - yes, I have:

-e git+https://<access-token>@github.com/<org>/<repo>.git@<branch>#"egg=sdk&subdirectory=sdk"

in my requirements.txt file (thats all I have in it for now while troubleshooting)

@AndrewFarley - unfortunately, it is not a public repo so I am unable to share the url. However, I have also tried replacing my private repo with the requests library in a very similar form under my requirements.txt file and it still gives an error. try using this url in place of my private url:

-e git+https://github.com/requests/requests.git@master#"egg=requests"

This works to install locally, but not within the serverless project with serverless-python-requirements

[AndrewFarley]

Hey guys, want a good laugh, this appears to be a pip/python bug. :P Check it out...

bash-4.2# python2.7 -m pip install --upgrade -t /var/task/ -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Collecting numpy (from measurements->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/c9/16/1134977cc35d2f72dbe80efa75a8e989ac21289f8e7e2c9005444cd17cd5/numpy-1.15.1-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
    Complete output from command /usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help
    
    error: option --home not recognized
    
    ----------------------------------------
  Rolling back uninstall of Measurements
Command "/usr/bin/python2.7 -c "import setuptools, tokenize;__file__='/var/task/src/measurements/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" develop --no-deps --home=/tmp/pip-target-WMZs4g" failed with error code 1 in /var/task/src/measurements/

But if I remove the -t /var/task option...

bash-4.2# python2.7 -m pip install --upgrade -r requirements.txt 
Obtaining measurements from git+https://github.com/jladan/package_demo.git#egg=measurements (from -r requirements.txt (line 3))
  Updating ./src/measurements clone
Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib64/python2.7/site-packages (from measurements->-r requirements.txt (line 3)) (1.15.1)
Installing collected packages: measurements
  Found existing installation: Measurements 0.5
    Uninstalling Measurements-0.5:
      Successfully uninstalled Measurements-0.5
  Running setup.py develop for measurements
Successfully installed measurements

Thought it'd make you laugh a little like it did me. I did some extended testing, the -t /var/task/ is unnecessary... so I've removed it in an upcoming MR. I'm doing a little more testing (running all tests) to make sure it's fine and then I'll push it and MR it. So, fix incoming...

[bryantbiggs]

ah damn, good catch @AndrewFarley - looks like its been a well-documented bug in pip that they have been trying to reconcile pypa/pip#4390

I really appreciate the assistance and the package guys!

[AndrewFarley]

@bryantbiggs As mentioned here: pypa/pip#562 (comment) please try to remove the -e before your requirements. Seems to be a long standing pip/py bug. :P Confirmed it fixes it for me. Anyways, for something like this plugin, -e makes no sense because you wouldn't "edit" your files in a readonly environment (lambda). :P @dschep after author confirmed, I'd say safe to close this bug. Although, if you want, I/we can add a check to check for -e's with egg's and warn the user or something, eh? idk

[dschep]

Agreed, -e doesn't make much sense in a deployed package. A warning would be a good idea... or.... is there any reason the plugin couldn't just strip the -e?

[AndrewFarley]

Oh, it absolutely could! :) I tried removing /var/task and it doesn't work, I tried chdir-ing into the /var/task folder before running the command and it mostly worked but had some bugs. So, I think auto-stripping the -e makes a lot of sense. I'll MR that (instead) then, eh? The only problem might be that @dschep is that -e puts stuff in ./src/ and if people are expecting things there (which they shouldn't, but hey) then it will break stuff.

Sooo... maybe just a warning is best?

[bryantbiggs]

So I tried removing -e and no luck. To make it easier to communicate, I made a repro here https://github.com/bryantbiggs/repro-serverless-requirements that you can test out. Its very similar to the setup I am working with - I have my serverless stacks in one sub-directory and instead of symlinking a bunch of internal files to use in my stacks, i'd rather make an internal package and just pip install that so that it keeps my requirements the same, etc. Anyways, as you'll see in there - you can pip install the sdk locally, but the serverless-python-requirements doesn't seem to be able to pick it.

[AndrewFarley]

Alright thanks for that I’ll play with it a bit tomorrow and see if I can come up with a nice workaround.

By the way do you know if it worked on an older version of this plugin? Eg 4.1.0?

[bryantbiggs]

Thanks @AndrewFarley - I just tried v4.1.0 and v4.0.0 and neither of those worked. Different error message though:

Error --------------------------------------------------

  Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-8zynn1hw/sdk"


     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-8zynn1hw/sdk"

    at installRequirements (/Users/B2/Downloads/repro-serverless-requirements/serverless/node_modules/serverless-python-requirements/lib/pip.js:114:11)
    at ServerlessPythonRequirements.installAllRequirements (/Users/B2/Downloads/repro-serverless-requirements/serverless/node_modules/serverless-python-requirements/lib/pip.js:202:5)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.spawn (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:408:17)
    at Deploy.BbPromise.bind.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:123:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:113:10)
    at BbPromise.reduce (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:55)
From previous event:
    at PluginManager.invoke (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:390:22)
    at PluginManager.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/classes/PluginManager.js:421:17)
    at variables.populateService.then.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:810:20)
    at tryOnImmediate (timers.js:768:5)
    at processImmediate [as _immediateCallback] (timers.js:745:5)
From previous event:
    at Serverless.run (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/Users/B2/.nvm/versions/node/v8.11.4/lib/node_modules/serverless/bin/serverless:43:50)
    at <anonymous>

[AndrewFarley]

We just have better debugging in the new version. Okay well at least the new stuff didn’t break this for you. That is good to hear. So I’m heading out but I’ll try to manage a fix for this tomorrow if I can

[dschep]

Yeah, now that you mention it @bryantbiggs I think this has never worked. See: #36

[blbradley]

Hey, I ran into this. Thanks for reporting.