adds script for setting up dev env in ubuntu #393

Oishika-Pradhan · 2018-05-18T16:49:53Z

Description

A bash script that sets up the dev env and hosts the application for (new) developers on ubuntu and osx.

Fixes #384

Type of Change:

Code

Code/Quality Assurance Only

This change requires a documentation update (software upgrade on readme file)
New feature (non-breaking change which adds functionality pre-approved by mentors)

How Has This Been Tested?

@Fenn-CS can you please check whether the script runs on ubuntu?
I have tested out the osx script

Checklist:

My PR follows the style guidelines of this project
I have commented my code or provided relevant documentation, particularly in hard-to-understand areas
I have made corresponding changes to the documentation

Code/Quality Assurance Only

My changes generate no new warnings

coveralls · 2018-05-18T16:53:50Z

Coverage remained the same at 98.655% when pulling 6af8da9 on Oishika-Pradhan:develop into 2943851 on systers:develop.

Fenn-CS

LGTM

Fenn-CS · 2018-05-19T19:33:24Z

@nida @prachi1210 ps review this

tonythomas01

-1

tonythomas01 · 2018-05-20T18:15:13Z

scripts/setup_env_osx.sh

+pip3 install virtualenv
+
+# setting up virtualenv and installing requirements
+virtualenv -p $(which python3) venv


this can lead to trouble. there are considerable differences between python 3.4 and python 3.5 etc. The only possible good solution seems to use https://github.com/pyenv/pyenv and point to a fixed python version. Like say python 3.6.5

tonythomas01 · 2018-05-20T18:16:46Z

scripts/setup_env_ubuntu.sh

@@ -0,0 +1,69 @@
+#!/bin/bash


please rename the command to say setup_env_linux_debian.sh or setup_env_linux_.sh. Ubuntu is a child derivative.

tonythomas01 · 2018-05-20T18:17:37Z

scripts/setup_env_ubuntu.sh

+        fi
+    done
+
+for ppkg in "$PYTHON_PREREQ[@]"


why not just running a pip install -r requirements/dev.txt or something like that ?

tonythomas01 · 2018-05-20T18:18:09Z

scripts/setup_env_ubuntu.sh

+
+if [ ${#MISSING[@]} -ne 0 ]; then
+    echo "Following required packages are missing, please install them first."
+    echo ${MISSING[*]}


ideally, your script should apt-get install all-those-packages-your-app-need.

tonythomas01 · 2018-05-20T18:19:06Z

scripts/setup_env_ubuntu.sh

+export SECRET_KEY=foobarbaz
+python ../systers_portal/manage.py migrate
+python ../systers_portal/manage.py cities_light
+python ../systers_portal.manage.py createsuperuser


../systers_portal/manage.py

Fenn-CS · 2018-05-20T21:30:34Z

@Oishika-Pradhan @tonythomas01 has made some change request.... They will make the script robust.

Oishika-Pradhan · 2018-05-21T18:07:30Z

I'll make the necessary changes @Fenn-CS @tonythomas01

Oishika-Pradhan · 2018-05-22T05:27:30Z

I have made the changes. Please look into it @tonythomas01 @Fenn-CS

Fenn-CS · 2018-05-22T21:51:42Z

scripts/setup_env_linux_debian.sh

+sed -i.bak "s/'PASSWORD': '',/'PASSWORD': '$password',/" ../systers_portal/systers_portal/settings/dev.py
+
+# host the server for the first time
+export SECRET_KEY=foobarbaz


What if you export the key in typical string format? export SECRET_KEY='foobarbaz' What you did works anyways... @tonythomas01 can review this one more time and if its ok to him then we are good.

I think this is fine as long as nobody is going to use this on production. 👍

Fenn-CS · 2018-05-22T22:56:59Z

scripts/setup_env_linux_debian.sh

+
+virtualenv -p $(which python3.6) venv
+source venv/bin/activate
+pip install -r requirements/dev.txt


This script is inside the script directory this is meant to be run in the project root you want to change that to pip install -r ../requirements/dev.txt

Fenn-CS · 2018-05-22T23:07:58Z

scripts/setup_env_linux_debian.sh

+apt-get install python3.6 python-pip
+pip install virtualenv
+
+virtualenv -p $(which python3.6) venv


I think you should just create the virtual environment without specifying the python version virtaulenv will pick the latests or you can ask user to specif path to desired python version which should be 3.6. Because this construct leads to problems on ubuntu

virtualenv doesn't pick the latest version, it picks the default one. And the application is based on python3.6.
What we can do is ask for the desired path for the python3.6 binary to the user.

@Oishika-Pradhan agreed on that. There is a problem tbh - apt-get install python3.6 might not work on some architectures (might even break some depending on how messed up their original install is). This is why I always recommend people to use a python version manager, like pyenv. Let me paste how this part should look like (copying from my own personal project)

Install pyenv using following scripts: https://github.com/pyenv/pyenv-installer. The installation requires you to add some things to your .bashrc or .zshrc file. In my case, it is: $ curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash ... export PATH="/home/user/.pyenv/bin:$PATH" eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" ... and source the rc file using source ~/.bashrc on macos: ... brew install pyenv brew install pyenv-virtualenv ... and add the following to your ~/.bash_profile (and after that do source ~/.bash_profile): ... eval "$(pyenv init -)" eval "$(pyenv virtualenv-init -)" ... Now install the right python version using ... $ pyenv install 3.6.4 ... Hint: If it shows something like: WARNING: The Python sqlite3 extension was not compiled. Missing the SQLite3 lib?, you can install it following something like: https://gist.github.com/antivanov/01ed4eac2d7486a170be598b5a0a4ac7. See that we are not using SQLite for local testing anymore, so you can actually ignore this. create a virtual environment for the project: ... pyenv virtualenv 3.6.4 yourvenvname-p3 ... activate the virtual environment for the repository (this will create a .python-version file, which is ignored via .gitignore): ... pyenv local yourvenvname-p3 ... - this would automatically activate the env as soon as you CD into the dir install dependencies: ... pip install -r requirements-dev.txt ...

Fenn-CS · 2018-05-22T23:09:38Z

scripts/setup_env_linux_debian.sh

+
+# create the database using psql and the user postgres
+read -p "Enter username: " username
+read -s -p "Enter password: " password


What if a confirmation password is added and the two compared to prevent mistakes, because the password is not displayed.

tonythomas01

-1 python3.6 will not work in <=16.04 ubuntu (and possibly other archs as well).

tonythomas01 · 2018-05-25T15:18:41Z

scripts/setup_env_linux_debian.sh

+fi
+
+# install system-wide dependencies 
+apt-get install python3.6 python-pip


this fails in Ubuntu 14.04 and 16.04

tonythomas01 · 2018-05-25T15:19:24Z

scripts/setup_env_linux_debian.sh

+apt-get install python3.6 python-pip
+pip install virtualenv
+
+read -p "Python3.6 executable path: " pythonpath


Question: is there a requirement for portal app to use a certain python 3.6 version ? If there is, I would just recommend using pyenv and save the day. I had pasted the instructions somewhere as I remember.

Oishika-Pradhan · 2018-05-31T05:56:51Z

@tonythomas01 I have updated the scripts to use pyenv and pyenv-virtualenv. Please review.

tonythomas01

nice looking very good but some minor nits, but I just approve :)

tonythomas01 · 2018-05-31T07:53:00Z

scripts/setup_env_linux_debian.sh

+fi
+
+# install system-wide dependencies 
+apt-get install python3 python-pip


so when you are using pyenv, apt-get install python3 is not required. You can kill it. also, install curl and git as well.

Oishika-Pradhan · 2018-05-31T10:46:39Z

Done @tonythomas01, thanks for reviewing.

Fenn-CS · 2018-06-01T14:24:53Z

@Oishika-Pradhan looks good at the level of executable commands and functionality however the script fails at user/db creation. The four fails match with all the four psql postgres -c " "
` commands

psql: FATAL:  role "root" does not exist
psql: FATAL:  role "root" does not exist
psql: FATAL:  role "root" does not exist
psql: FATAL:  role "root" does not exist
Traceback (most recent call last):
  File "../systers_portal/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "../systers_portal/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "../systers_portal/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management
Traceback (most recent call last):
  File "../systers_portal/manage.py", line 8, in <module>
    from django.core.management import execute_from_command_line
ImportError: No module named django.core.management

I believe in ubuntu we have to supply a password to use the postgres user, sure you have to include the -u flag or something please look at : https://stackoverflow.com/questions/16973018/createuser-could-not-connect-to-database-postgres-fatal-role-tom-does-not-e/16974197#16974197

Oishika-Pradhan · 2018-06-02T11:46:20Z

@Fenn-CS I had to run psql from the user postgres. I have fixed it.

nida

Looks good.

tonythomas01 · 2018-06-03T09:10:57Z

scripts/setup_env_linux_debian.sh

+
+# install system-wide dependencies 
+apt-get install git curl python-pip
+curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash


you have to add postgresql installation scripts. also, probably

sudo apt-get install python-pip python-dev libpq-dev postgresql postgresql-contrib git curl

?

updates the dev env setup (unix) section, now asks users to use the scripts for osx and debian

Fenn-CS

The my installation was successful. But the script at the level of running migrations because of SECRET_KEY... issues please adjust the PR according the comments. Also the script must be run inside the script folder others it would since the relative paths would break. But you should bear in mind that is possible for someone to run the script out of the scripts folder...That means after your final changes you need to update the documentation (setup guide)

Fenn-CS · 2018-06-12T00:22:26Z

scripts/devenv-debian

+        sed -i.bak "s/'NAME': 'systersdb',/'NAME': '$db',/" ../systers_portal/systers_portal/settings/dev.py
+        sed -i.bak "s/'USER': '',/'USER': '$username',/" ../systers_portal/systers_portal/settings/dev.py
+        sed -i.bak "s/'PASSWORD': '',/'PASSWORD': '$password',/" ../systers_portal/systers_portal/settings/dev.py
+


the secret key is ideally export immediately after the virtual environment is created.

Fenn-CS · 2018-06-12T00:23:45Z

scripts/devenv-debian

+
+    elif [ $1 == 'run' ]; then
+        # host the development server
+        export SECRET_KEY='foobarbaz'


Please move export SECRET_KEY=foobar to the before all python ... instructions

@Fenn-CS when a developer would run ./devenv-debian run just to host the server, the env variable SECRET_KEY must be initialized, and the other python ... commands don't need it anyways.

But setup is failing on ubuntu... Complaining of SECRET_KEY not found in path... Anything that would access the manage.py file will need that e.g python ../systers_portal/manage.py migrate you can't run that without setting the SECRET_KEY first....

I have added export SECRET_KEY='foobarbaz' before all the python commads, please check.

- argument 'run' hosts the server - argument 'setup' sets up the devenv - changed the names of the scripts for easier usability - added argument error handling

Fenn-CS · 2018-06-13T14:09:24Z

I just hope the OSX script has been tested properly... As I have concentrated on the debian version which Looks good.

tonythomas01 · 2018-06-13T14:22:23Z

This has matured enough. Anything that shows up later can be caught and fixed later I guess. Also, with python and bash - its not easy to write a 100% always working solution.

theyashshahs · 2020-06-11T16:10:48Z

@Oishika-Pradhan resolve the merge conflicts :)

theyashshahs

remove the merge conflicts

theyashshahs · 2020-07-04T14:38:35Z

@Oishika-Pradhan change the PR title to follow the commit guidelines and resolve the merge conflicts :)

theyashshahs · 2020-07-13T18:18:01Z

ping @Oishika-Pradhan

SanketDG · 2020-08-10T08:05:25Z

@anitab-org/portal-maintainers @anitab-org/portal-managers Do we really need this?

IMO having manual instructions + Docker is enough right now, also we are playing with the idea of pre-made environments.

sakshi1499 · 2020-08-10T11:42:06Z

@SanketDG I totally agree! Yes in the last session we did discuss about CodeSandbox and other options. Closing this PR.

Oishika-Pradhan force-pushed the develop branch 4 times, most recently from 0c9ffa9 to 1b9f97d Compare May 19, 2018 09:50

Fenn-CS self-requested a review May 19, 2018 19:30

Fenn-CS previously approved these changes May 19, 2018

View reviewed changes

tonythomas01 suggested changes May 20, 2018

View reviewed changes

Oishika-Pradhan dismissed Fenn-CS’s stale review via 13f0df7 May 22, 2018 05:23

Oishika-Pradhan force-pushed the develop branch 3 times, most recently from 66286fa to 7978945 Compare May 22, 2018 05:25

Fenn-CS suggested changes May 22, 2018

View reviewed changes

Oishika-Pradhan force-pushed the develop branch from 7978945 to 974e6ec Compare May 23, 2018 15:36

tonythomas01 suggested changes May 25, 2018

View reviewed changes

Oishika-Pradhan force-pushed the develop branch 2 times, most recently from b9910f0 to 8a6dbc0 Compare May 31, 2018 05:55

tonythomas01 previously approved these changes May 31, 2018

View reviewed changes

Oishika-Pradhan dismissed tonythomas01’s stale review via 92ed03a May 31, 2018 10:16

Oishika-Pradhan force-pushed the develop branch from 8a6dbc0 to 92ed03a Compare May 31, 2018 10:16

Oishika-Pradhan force-pushed the develop branch from 0381060 to a2cbf90 Compare June 2, 2018 11:43

Oishika-Pradhan force-pushed the develop branch from a2cbf90 to 4dadca7 Compare June 2, 2018 11:48

nida previously approved these changes Jun 2, 2018

View reviewed changes

tonythomas01 reviewed Jun 3, 2018

View reviewed changes

Oishika-Pradhan added 3 commits June 11, 2018 19:42

adds script for setting up dev env in ubuntu and osx

1175018

updates the README

7fda32e

updates the dev env setup (unix) section, now asks users to use the scripts for osx and debian

Updated the dev env setup scripts and README accordingly

93e8122

Oishika-Pradhan dismissed nida’s stale review via 93e8122 June 11, 2018 14:13

Oishika-Pradhan force-pushed the develop branch from 4dadca7 to 93e8122 Compare June 11, 2018 14:13

Fenn-CS reviewed Jun 12, 2018

View reviewed changes

Oishika-Pradhan force-pushed the develop branch from 8709bf9 to 6af8da9 Compare June 13, 2018 12:57

Updated the scripts (added arguments)

6af8da9

- argument 'run' hosts the server - argument 'setup' sets up the devenv - changed the names of the scripts for easier usability - added argument error handling

Fenn-CS approved these changes Jun 13, 2018

View reviewed changes

tonythomas01 approved these changes Jun 13, 2018

View reviewed changes

nida approved these changes Jun 18, 2018

View reviewed changes

theyashshahs reviewed Jun 11, 2020

View reviewed changes

sakshi1499 closed this Aug 10, 2020

adds script for setting up dev env in ubuntu #393

adds script for setting up dev env in ubuntu #393

Conversation

Oishika-Pradhan commented May 18, 2018 • edited

Description

Type of Change:

How Has This Been Tested?

Checklist:

coveralls commented May 18, 2018 • edited

Fenn-CS left a comment

Choose a reason for hiding this comment

Fenn-CS commented May 19, 2018

tonythomas01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fenn-CS commented May 20, 2018

Oishika-Pradhan commented May 21, 2018

Oishika-Pradhan commented May 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tonythomas01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oishika-Pradhan commented May 31, 2018

tonythomas01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oishika-Pradhan commented May 31, 2018

Fenn-CS commented Jun 1, 2018

Oishika-Pradhan commented Jun 2, 2018

nida left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fenn-CS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Oishika-Pradhan Jun 12, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fenn-CS commented Jun 13, 2018

tonythomas01 commented Jun 13, 2018 • edited

theyashshahs commented Jun 11, 2020

theyashshahs left a comment

Choose a reason for hiding this comment

theyashshahs commented Jul 4, 2020 • edited

theyashshahs commented Jul 13, 2020

SanketDG commented Aug 10, 2020

sakshi1499 commented Aug 10, 2020

Oishika-Pradhan commented May 18, 2018 •

edited

coveralls commented May 18, 2018 •

edited

Oishika-Pradhan Jun 12, 2018 •

edited

tonythomas01 commented Jun 13, 2018 •

edited

theyashshahs commented Jul 4, 2020 •

edited