Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble running pipenv in CI #290

Closed
lburdzy opened this Issue Mar 25, 2017 · 13 comments

Comments

Projects
None yet
4 participants
@lburdzy
Copy link

lburdzy commented Mar 25, 2017

I've been trying to get pipenv to work with gitlab CI, but i'm getting errors. Here are last lines of output:

To activate this project's virtualenv, run the following:
 $ pipenv shell
$ pipenv run --no-interactive python manage.py migrate
ERROR: Job failed: exit code 1

And here is my .gitlab.ci.yaml

# This file is a template, and might need editing before it works on your project.
# Official framework image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python
image: python:latest

# Pick zero or more services to be used on all builds.
# Only needed when using a docker container to run your tests in.
# Check out: http://docs.gitlab.com/ce/ci/docker/using_docker_images.html#what-is-service
services:
  - postgres:latest

variables:
  POSTGRES_DB: db
  POSTGRES_USER: user
  POSTGRES_PASSWORD: password

# This folder is cached between builds
# http://docs.gitlab.com/ce/ci/yaml/README.html#cache
cache:
  paths:
  - ~/.cache/pip/

# This is a basic example for a gem or script which doesn't use
# services such as redis or postgres
before_script:
  - python -V                                   # Print out python version for debugging
  # Uncomment next line if your Django app needs a JS runtime:
  # - apt-get update -q && apt-get install nodejs -yqq
  - pip install pipenv
  - pipenv install --dev

test:
  variables:
    DATABASE_URL: "postgresql://postgres:postgres@postgres:5432/$POSTGRES_DB"
  script:
  - pipenv run python manage.py migrate
  - pipenv run python manage.py test
  - pipenv run coverage run -m pytest
  - pipenv run coverage run report

Any idea why it fails? I did also try to run pipenv shell, but that failed too.

@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Mar 25, 2017

Hey @lburdzy, I'm not immediately sure what's going on here since the CI logs are pretty tight lipped.

My first question is where --no-interactive is coming from. It doesn't look like that's in the command you're running in .gitlab.yml under the script section.

Also, have you tried running this command locally and does it work? If not we may be able to do some debugging there.

@lburdzy

This comment has been minimized.

Copy link
Author

lburdzy commented Mar 25, 2017

Oh, sorry, I mixed up the config version. I tried running it both with and without --no-interactive, got the same result.
The command works locally, both on my environment and in a local docker image (I can even run pipenv shell there).

@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Mar 25, 2017

So I think we'll need to do some debugging directly on Gitlab then. Do you happen to have a project I can clone and run some tests on? If not if you can create one, that would be incredibly helpful.

What we're looking for here is some kind of debug output for why this is failing. pexpect is spawning a child process that runs your supplied command. It's stored as the variable c in def run(...) in pipenv/cli.py. If you want to try forking pipenv and installing with pip from your fork you can probably test this yourself too. I believe c.read() should give you what you want, but I haven't dug too deeply into pexpect yet and won't have a chance to until this evening.

@lburdzy

This comment has been minimized.

Copy link
Author

lburdzy commented Mar 27, 2017

I've created an empty test project, but I'm getting weird results. It seems that pipenv run can successfully execute, but it fails at running coverage. On my original project it does not get through first command.
Here is the project: https://gitlab.com/lburdzy/pipenv_test_project/
And here is CI result: https://gitlab.com/lburdzy/pipenv_test_project/builds/13090712

@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Mar 28, 2017

Thanks for putting this together @lburdzy! So the fact we're getting an error at a different point makes the debugging process here a bit harder.

My guess on the failure at pipenv run coverage run -m pytest is that it's an issue in line with #249. coverage is spawning it's own subprocess to run pytest after we've spawned a subprocess to run coverage which seems to break itself out of the virtualenv. In order to prevent this, you'll want to try something like pipenv shell -c "coverage run -m pytest".

I'm going to try to set aside some time this morning to test this, but if you have a chance to get to it, let me know the results. Thanks!

@kienpham2000

This comment has been minimized.

Copy link

kienpham2000 commented Apr 4, 2017

I'm currently running into similar issue with Gitlab CI. When I tried:

$ pipenv shell -c "python test.py"
Windows is not currently supported.
ERROR: Job failed: exit code 1

If I activate the shell with: $ pipenv shell, I get this:

  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/cli.py", line 975, in shell
    c.interact()
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pexpect/pty_spawn.py", line 740, in interact
    mode = tty.tcgetattr(self.STDIN_FILENO)
termios.error: (25, 'Inappropriate ioctl for device')
ERROR: Job failed: exit code 1
@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Apr 4, 2017

@kienpham2000 thanks for looking into this. I patched that error message yesterday and it looks like the docker instance on GitlabCI isn't going to support pipenv shell.

I'm currently piecing together what all is wrong inside of run and it looks to be a combination of missing dependencies on @lburdzy's project and likely a hiccup somewhere in our vendored dependencies. I'm still on the hunt for what exactly is causing the failure. I'll add more soon.

@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Apr 6, 2017

OK @lburdzy, I've finally gotten to a place that we can start working forward. First off, thanks for setting up that gitlab repo, it was incredibly valuable for debugging.

So the short of this is running pytest without a specified directory will aggregate all files starting with test and run them. This is grabbing all of the pipenv tests and running them in an unexpected environment which is causing them to fail. I'm assuming you're not trying to get coverage reports for pipenv in your project, so specifying the exact directory of your tests would be were I'd start. Why this is happening specifically on Gitlab, I'm not entirely sure.

The other issue is you are running pytest without ever installing it (at least in the example project). If you have your dev dependencies installed things should run smoothly.

Let me know if that helps put you on the right track. I'm going to look into getting some better error logging in this case.

@piotr-szpetkowski

This comment has been minimized.

Copy link

piotr-szpetkowski commented Apr 6, 2017

I'm also affected by this issue. It seems like stdout works correctly at travis (as we can see right here https://travis-ci.org/kennethreitz/pipenv/jobs/218528105), however it seems like all of stdout printed by pipenv is getting ignored by gitlab-ci. I'm pretty sure it still works correctly, because if I somehow manage to fix all of the issues CI passes successfuly, so it's probably about capturing stdout / stderr?

@piotr-szpetkowski

This comment has been minimized.

Copy link

piotr-szpetkowski commented Apr 25, 2017

I've found a workaround for the issue - not sure if it's a proper way, however it's only thing that at the moment works for me. The solution is to put following commands into CI job:

- export VENV_HOME_DIR=$(pipenv --venv)
- source $VENV_HOME_DIR/bin/activate
- py.test --capture=no --ds=config.settings.test --flake8 .
@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Apr 25, 2017

@piotr-szpetkowski Thanks for checking in and leaving your solution! #320 may make this work slightly better, but it still needs to be tested it in docker to confirm.

@piotr-szpetkowski

This comment has been minimized.

Copy link

piotr-szpetkowski commented Apr 26, 2017

@nateprewitt #320 solved the issue for me without using workaround, so now I can simply use:

- pipenv run py.test --capture=no --ds=config.settings.test --flake8 .

For me issue is fixed.

@nateprewitt

This comment has been minimized.

Copy link
Member

nateprewitt commented Apr 26, 2017

Great to hear @piotr-szpetkowski, I'm closing things out then. Thanks everyone for your input and help!

sdementen added a commit to sdementen/piecash that referenced this issue Mar 5, 2018

sdementen added a commit to sdementen/piecash that referenced this issue Mar 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.