New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add virtualenv support - Feature/697 #930

Merged
merged 4 commits into from Jun 19, 2015

Conversation

Projects
None yet
5 participants
@djhaskin987
Contributor

djhaskin987 commented May 29, 2015

This PR is for #697 .

My shot at a virtualenv input plugin, after taking a look at @evilsocket 's code. Thanks for letting me stand on your shoulders, @evilsocket !

Features

  1. Virtualenv named based on the pip package it's built around
  2. You can specify a particular version of a pip package to build a virtualenv around e.g. psutil==2.2.1
  3. No surprises. I took the script from here --> https://github.com/brutasse/graphite-api/blob/master/fpm/build-deb.sh and put it straight into FPM as much as possible
  4. Version resolution: The version of the "chief" package is used if no other version is specified
  5. Optionally add extra files, like the script above does when it adds files that go under /etc
  6. By default, prefixes virtualenv package names with virtualenv, but this can be overridden
  7. Option exists to change the index url which pip uses to build the virtualenv
  8. Tried as much as possible to conform to how other FPM plugins are coded

Caveats

  1. You cannot currently use '>=' or '<=' in the package specification, only '==' currently exists as an option.
  2. Places virtualenvs in an install location specified by the --virtualenv-install-location. Couldn't use prefix, as this changes both where the virtualenv and the extra files go. If I specified --prefix foo and --virtualenv-install-location /bar, the virtual env for psutil would be installed under /foo/bar/psutil
  3. You get bad results if you use a relative path for the value of --virtualenv-install-location.
  4. No specs yet (they will be forthcoming) but I have manually tested everything here and it works 馃憤

Example

A default case:


~/Workspace/compute/src/github.com/djhaskin987/fpm $ bin/fpm -s virtualenv -t deb --virtualenv-other-files-dir pkg/ example
Debian packaging tools generally labels all files in /etc as config files, as mandated by policy, so fpm defaults to this behavior for deb packages. You can disable this default behavior with --deb-no-default-config-files flag {:level=>:warn}
Created package {:path=>"virtualenv-example_1.0_amd64.deb"}

A rather involved example:

~/Workspace/compute/src/github.com/djhaskin987/fpm $ bin/fpm -f -n barg -v 3.0.0 --prefix /boogar --virtualenv-install-location /junie --no-virtualenv-fix-name -s virtualenv -t rpm --virtualenv-other-files-dir pkg/ psutil==2.2.1
virtualenv_folder: /junie/psutil
no value for epoch is set, defaulting to nil {:level=>:warn}
Force flag given. Overwriting package at barg-3.0.0-1.x86_64.rpm {:level=>:warn}
no value for epoch is set, defaulting to nil {:level=>:warn}
Created package {:path=>"barg-3.0.0-1.x86_64.rpm"}
@jordansissel

This comment has been minimized.

Owner

jordansissel commented May 29, 2015

I'll see about testing this soon. Thank you and @evilsocket for working on this! :)

@untergeek

This comment has been minimized.

untergeek commented May 29, 2015

I volunteer to use https://github.com/elastic/curator as a test env.

@evilsocket

This comment has been minimized.

evilsocket commented May 29, 2015

@jordansissel @djhaskin987 it's been my pleasure :)

@jordansissel

This comment has been minimized.

Owner

jordansissel commented Jun 15, 2015

In testing this, I found a few things:

  • utime: Operation not permitted @ utime_internal

This may only affect OSX, not sure yet. Here's the stack trace (trimmed for brevity)

/Users/jls/projects/fpm/lib/fpm/util.rb:170:in `utime': Operation not permitted @ utime_internal - /var/folders/4r/4fpyt_2d1qvfrgb2fnngrqm40000gn/T/package-virtualenv-staging20150615-48864-vpnc11/./usr/share/python/curator/.Python (Errno::EPERM)
        from /Users/jls/projects/fpm/lib/fpm/util.rb:170:in `copy_metadata'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:203:in `copy'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:148:in `block in clone'
...
        from /Users/jls/.rvm/rubies/ruby-2.1.6/lib/ruby/2.1.0/find.rb:42:in `find'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:146:in `clone'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:71:in `block in input'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:69:in `chdir'
        from /Users/jls/projects/fpm/lib/fpm/package/dir.rb:69:in `input'
        from /Users/jls/projects/fpm/lib/fpm/package/virtualenv.rb:120:in `input'
...

Works on Ubuntu 12.04, though. :)
This seems unrelated to this patch, probably, but still a bug to fix.


Testing the resulting package:

% head /usr/share/python/elasticsearch-curator/bin/curator
#!/usr/share/python/elasticsearch-curator/bin/python

% /usr/share/python/elasticsearch-curator/bin/curator --version
curator, version 3.1.0

So far so good!

@jordansissel

This comment has been minimized.

Owner

jordansissel commented Jun 15, 2015

Installing the .deb on a system without any python installation:

root@3984584eefbf:/work# dpkg -i virtualenv-elasticsearch-curator_1.0_amd64.deb
Selecting previously unselected package virtualenv-elasticsearch-curator.
(Reading database ... 11528 files and directories currently installed.)
Preparing to unpack virtualenv-elasticsearch-curator_1.0_amd64.deb ...
Unpacking virtualenv-elasticsearch-curator (1.0) ...
Setting up virtualenv-elasticsearch-curator (1.0) ...
root@3984584eefbf:/work# /usr/share/python/elasticsearch-curator/bin/curator
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
ImportError: No module named site

It can't find the site module, but it's in the .deb: ./usr/share/python/elasticsearch-curator/lib/python2.7/site.py

On systems with python available without virtualenv, the system's site.py will be used when the virtualenv one cannot be found. I think this is a bug. @djhaskin987 - thoughts? Is this expected? (I don't know much about virtualenv)

@djhaskin987

This comment has been minimized.

Contributor

djhaskin987 commented Jun 19, 2015

@jordansissel I believe virtualenv works either by creating a hard link between or copying python outright from the system. It therefore assumes that python is on the system. The ideal thing would be guaranteeing that python is on the parent system by adding a dependency, but it would be a nightmare deciding what python is called by the target system's package manager. I think the only thing feasible is to print a warning about this. Do you think that would suit?

@djhaskin987

This comment has been minimized.

Contributor

djhaskin987 commented Jun 19, 2015

@jordansissel as to the bug, it appears this line is the "culprit":

    120     dir.input(".")

Which means that this happens during cloning the directory.

@jordansissel

This comment has been minimized.

Owner

jordansissel commented Jun 19, 2015

锘縄 believe virtualenv works either by creating a hard link between or copying python outright from the system. It therefore assumes that python is on the system.

Yeah I noticed that it copies (or hardlinks, I didnt' check) the python binary but creates symlinks for things like site.py and other similar things.

Going forward, it's probably OK to have this as a known limitation. Unfortunately, it doesn't solve @untergeek's needs (self-contained package), yet, but I think this works well enough to move forward with merging.

@jordansissel

This comment has been minimized.

Owner

jordansissel commented Jun 19, 2015

@djhaskin987 agreed that the bug isn't in your code (dir copying something funky happens on OSX)

jordansissel added a commit that referenced this pull request Jun 19, 2015

@jordansissel jordansissel merged commit 86795c4 into jordansissel:master Jun 19, 2015

@jordansissel

This comment has been minimized.

Owner

jordansissel commented Jun 19, 2015

Huge thanks to @evilsocket, @djhaskin987, and everyone else for making virtualenv support happen :)

@djhaskin987 djhaskin987 deleted the djhaskin987:feature/697 branch Jun 19, 2015

@djhaskin987

This comment has been minimized.

Contributor

djhaskin987 commented Jun 19, 2015

Yay! thx :) 馃憤

@jordansissel jordansissel changed the title from Feature/697 to Add virtualenv support - Feature/697 Jul 21, 2015

@nffdiogosilva

This comment has been minimized.

nffdiogosilva commented Jun 6, 2016

Hello there,

I'm having problems understanding correctly how the command works to generate a package with a specific virtualenv and its requirements.

How can I create a package for a virtualenv with multiple requirements?

Thanks in advance,
Nuno

EDIT: I created a issue related with this question #1129

jordansissel added a commit that referenced this pull request Jun 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment