Skip to content

/ saltfs

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standardize python setup, ensure homu gets python3 #148

Merged
merged 3 commits into from Feb 11, 2016
+86 −62
Merged

Standardize python setup, ensure homu gets python3 #148

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e2a8efc
Standardize python setup, ensure homu gets python3
aneeshusa Nov 3, 2015
b4e4b52
Install python, pip, virtualenv on OS X
aneeshusa Nov 4, 2015
a3baeaa
Restart services on package change
aneeshusa Nov 5, 2015
File filter...
Filter file types
Jump to…
Jump to file
Failed to load files.

Always

Just for now

Next

Standardize python setup, ensure homu gets python3 

- Trusty comes with python3, pip and virtualenv by default, but write
states defensively to install these dependencies as required and require
the pip or virtualenv states as used by other states.
- Homu requires a python3 toolchain, use virtualenv-3.4 and pass python:
  python3 to get the correct setup (including pip3 inside the venv).
- Update the style guide example to use pkg.installed instead of
  pip.installed to avoid muddling it with a require: pip.
Fixes #142.
  • Loading branch information
@aneeshusa
aneeshusa committed Feb 11, 2016
commit e2a8efc5eac2b0fe80e43dc75d795f3e6dfc20e2
12 STYLE_GUIDE.md
View file
Open in desktop
@@ -19,16 +19,16 @@ regardless of the number of packages.
*Unsafe*:

```salt
buildbot-dependencies:
pip.installed:
- name: buildbot == 0.8.12
essential-dependencies:
pkg.installed:
- name: cowsay

This comment has been minimized.

Sign in to view
Copy link
@Manishearth

Manishearth Feb 11, 2016

Member 
 _____________________________________ 
< a very essential dependency, indeed >
 ------------------------------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
```

*Better*:

```salt
buildbot-dependencies:
pip.installed:
essential-dependencies:
pkg.installed:
- pkgs:
- buildbot == 0.8.12
- cowsay
```
5 android-dependencies.sls
View file
Open in desktop
@@ -23,7 +23,10 @@ android-dependencies:
- libgl1-mesa-dev
- refresh: True
pip.installed:
- name: s3cmd
- pkgs:
- s3cmd
- require:
- pkg: pip

android-sdk:
archive.extracted:
38 buildbot/master.sls
View file
Open in desktop
@@ -1,20 +1,19 @@
buildbot:
buildbot-master:
pip.installed:
- pkgs:
- buildbot == 0.8.12
- service_identity == 14.0.0

txgithub:
pip.installed

boto:
pip.installed

buildbot-master:
- txgithub == 15.0.0
- boto == 2.38.0
- require:
- pkg: pip
service.running:
- enable: True
- require:
- pip: buildbot
- pip: buildbot-master
- watch:
- file: /home/servo/buildbot/master
- file: /etc/init/buildbot-master.conf

/home/servo/buildbot/master:
file.recurse:
@@ -24,30 +23,20 @@ buildbot-master:
- group: servo
- dir_mode: 755
- file_mode: 644
- watch_in:
- service: buildbot-master

/etc/init/buildbot-master.conf:
file.managed:
- source: salt://buildbot/buildbot-master.conf
- user: root
- group: root
- mode: 644
- watch_in:
- service: buildbot-master

buildbot-github-listener:
service.running:
- enable: True

/usr/local/bin/github_buildbot.py:
file.managed:
- source: salt://buildbot/github_buildbot.py
- user: root
- group: root
- mode: 755
- watch_in:
- service: buildbot-github-listener

/etc/init/buildbot-github-listener.conf:
file.managed:
@@ -56,5 +45,10 @@ buildbot-github-listener:
- user: root
- group: root
- mode: 644
- watch_in:
- service: buildbot-github-listener

buildbot-github-listener:
service.running:
- enable: True
- watch:
- file: /usr/local/bin/github_buildbot.py
- file: /etc/init/buildbot-github-listener.conf
4 buildbot/slave.sls
View file
Open in desktop
@@ -1,9 +1,11 @@
{% from 'common/map.jinja' import config as common with context %}
buildbot-slave.pip:
buildbot-slave-dependencies:
pip.installed:
- pkgs:
- buildbot-slave == 0.8.12
- require:
- pkg: pip
{{ common.servo_home }}/buildbot/slave:
file.recurse:
31 common/init.sls
View file
Open in desktop
@@ -1,11 +1,38 @@
{% from tpldir ~ '/map.jinja' import config with context %}
{% if grains['kernel'] != 'Darwin' %}
python-pkgs:
# Ubuntu has python2 as default python
python2:
pkg.installed:
- pkgs:
- python-pip
- python
- python-dev
python3:
pkg.installed:
- pkgs:
- python3
# Ensure pip is default by purging pip3
pip:
pkg.installed:
- pkgs:
- python-pip
- reload_modules: True
pip3:
pkg.purged:
- pkgs:
- python3-pip
# Virtualenv package creates virtualenv and virtualenv-3.4 executables

This comment has been minimized.

Sign in to view
Copy link
@Manishearth

Manishearth Nov 4, 2015

Member

In my experience I needed to pip3 install virtualenv before I got virtualenv3. Should we make two keys here, one with bin_env: pip and one with bin_env: pip3?

This comment has been minimized.

Sign in to view
Copy link
@aneeshusa

aneeshusa Nov 4, 2015

Author Member

I just tested this in a fresh vagrant box. It started out without pip or virtualenv (any version). I ran

  1. sudo apt-get install python-pip
  2. sudo pip install virtualenv
    This created /usr/local/bin/virtualenv and /usr/local/bin/virtualenv-3.4 executables, the latter of which works fine for the homu venv - I peeked inside the vagrant box and the homu venv had python3 and pip3.

bin_env is used when you want to run pip inside a virtualenv, such as in homu/init.sls, so I'm not sure what you are proposing with the two keys. Could you please clarify?

This comment has been minimized.

Sign in to view
Copy link
@Manishearth

Manishearth Nov 4, 2015

Member

Are the default python/pip/virtualenv the 2.7 ones?

bin_env is for both selecting a virtualenv or selecting a custom binary to run. It lets you say "Install this pip package, but use the pip3.4 binary". https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.pip.html

This comment has been minimized.

Sign in to view
Copy link
@aneeshusa

aneeshusa Nov 4, 2015

Author Member

Yes, on Ubuntu the defaults are all 2.7. Ubuntu is not planning to change this until PEP 394 says otherwise.

bin_env: Right, I'm using this but in a different way. The approach I'm taking is to use virtualenv.managed and pass in python3 in homu/init.sls; salt automatically will install pip3 in the virtualenv (pip3 is bundled with python starting with python3.4). Then, I use pip.installed and set bin_env to the virtualenv location for homu's dependencies, which means it ends up using pip3 under the hood.

Also, for future reference you'll want to use the state documentation, not the module docs :)

This comment has been minimized.

Sign in to view
Copy link
@Manishearth

Manishearth Nov 4, 2015

Member

Okay, sounds good. So when you say python: python3 in the homu setup below, does that mean that it will also use virtualenv3? Because homu doesn't even install properly with an older virtualenv version, it needs to use pip3, python3, and virtualenv3. (Not sure if virtualenv2 even lets you create a python3 env, but it's good to be certain)
virtualenv:
pip.installed:
- pkgs:
- virtualenv
- require:
- pkg: pip
- pkg: pip3
{% endif %}
servo:
46 homu/init.sls
View file
Open in desktop
@@ -1,44 +1,42 @@
https://github.com/servo/homu:
homu:
git.latest:
- name: https://github.com/servo/homu
- rev: e07f74e7a3a185768e71639d6a328ef8ea234f92
- target: /home/servo/homu
- user: servo
- require_in:
- pip: install_homu

/home/servo/homu/cfg.toml:
file.managed:
- source: salt://homu/cfg.toml
- template: jinja
- user: servo
- group: servo
- mode: 644
- watch_in:
- service: homu

/home/servo/homu/_venv:
virtualenv.managed:
- name: /home/servo/homu/_venv
- venv_bin: virtualenv-3.4
- python: python3

This comment has been minimized.

Sign in to view
Copy link
@aneeshusa

aneeshusa Nov 4, 2015

Author Member

@Manishearth the venv_bin option is what tells salt to use virtualenv-3.4, and the python option tells virtualenv to install python3 (and pip3) inside the virtualenv.

This comment has been minimized.

Sign in to view
Copy link
@Manishearth

Manishearth Nov 4, 2015

Member

Oh, right, I didn't notice that, sorry 😄
- system_site_packages: False
- require_in:
- pip: install_homu

install_homu:
- require:
- pkg: python3
- pip: virtualenv
pip.installed:
- bin_env: /home/servo/homu/_venv
- editable: /home/servo/homu

homu:
- require:
- git: homu
- virtualenv: /home/servo/homu/_venv
service.running:
- enable: True
- require:
- pip: install_homu
- pip: homu
- watch:
- file: /home/servo/homu/cfg.toml
- file: /etc/init/homu.conf

/home/servo/homu/cfg.toml:
file.managed:
- source: salt://homu/cfg.toml
- template: jinja
- user: servo
- group: servo
- mode: 644

/etc/init/homu.conf:
file.managed:
- source: salt://homu/homu.conf
- user: root
- group: root
- mode: 644
- watch_in:
- service: homu
4 servo-dependencies.sls
View file
Open in desktop
@@ -28,8 +28,10 @@ servo-dependencies:
{% endif %}
pip.installed:
- pkgs:
- virtualenv
- ghp-import
- require:
- pkg: pip
- pip: virtualenv
{% if grains['kernel'] == 'Darwin' %}
# Workaround for https://github.com/saltstack/salt/issues/26414
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.