Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Db connection issue #100

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -100,3 +100,6 @@ webapp/static/output.js

# Users info
users.json

# Flask Session
flask_session/
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,6 @@ numpy
scipy
scikit-learn
flask_sqlalchemy
flask-session
flask-login
gunicorn
14 changes: 9 additions & 5 deletions tests/test_db_select.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
from tests.utils import rig_test_client
from webapp import app
import json
import flask

def test_db_list():
response = app.test_client().get("/db_list")
assert response.status_code == 200


def test_db_choose():
response = app.test_client().get("/db_choose/cmpd")
assert response.status_code == 200
response_data = json.loads(response.get_data().decode('utf-8'))
assert response_data['result'] == app.config['DB_NAME']

with rig_test_client({}) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
response = test_app.get("/db_choose/cmpd")
assert response.status_code == 200
response_data = json.loads(response.get_data().decode('utf-8'))
assert response_data['result'] == session['engine']
8 changes: 6 additions & 2 deletions tests/test_evaluations_feature_distribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,19 +66,23 @@

def test_get_feature_dist_test():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/feature_dist_test/arrests_id_p1d_arrestscrimetype_assault_avg/2014-04-24'
response = test_app.get(url)
assert response.status_code == 200
response_data = json.loads(response.get_data().decode('utf-8'))
expected = load_json_example('/evaluations/1/feature_dist_test')
assert response.status_code == 200
assert expected == response_data

def test_get_feature_dist_train():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what exactly is happening in these cases where you only set the session engine in the context-managed block? usually a context manager is doing something in __enter__ and something else in __exit__. does this affect the remainder of the test? is it different from:

session = test_app.session_transaction()
session['engine'] = 'cmpd'

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will give me this error if I don't use the context-managed block with session.
TypeError: '_GeneratorContextManager' object does not support item assignment

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense 😄 I would've guessed that it had an interface that wasn't based around the context manager, but I'm not sure it matters, anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, in fact, I would guess that it's writing the session to the backend upon __exit__ 😉

url = '/evaluations/1/feature_dist_train/arrests_id_p1d_arrestscrimetype_assault_avg'
response = test_app.get(url)
assert response.status_code == 200
response_data = json.loads(response.get_data().decode('utf-8'))
expected = load_json_example('/evaluations/1/feature_dist_train')
assert response.status_code == 200
assert expected == response_data

2 changes: 2 additions & 0 deletions tests/test_evaluations_feature_importance.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

def test_feature_importance():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/feature_importance/10'
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_individual_importance.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@

def test_individual_importances():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we just want to add an optional engine to rig_test_client and have it handle this?

session['engine'] = 'cmpd'
url = '/evaluations/1/individual_feature_importance/1234/2014-12-01'
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_metricsovertime.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@

def test_metric_overtime():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/1/metric_overtime'
response = test_app.post(
route,
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_model_comments.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@

def test_model_comments():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/model_comments'
response = test_app.post(
route,
Expand Down
4 changes: 4 additions & 0 deletions tests/test_evaluations_model_groups.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ def test_model_groups_1():
'evaluations': evaluations_data
}
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/search_model_groups/all'
response = test_app.post(
route,
Expand Down Expand Up @@ -104,6 +106,8 @@ def test_model_groups_2():
'evaluations': evaluations_data
}
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/search_model_groups/all'
response = test_app.post(
route,
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_modelprediction.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@

def test_model_prediction():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/model_result/{}'.format(EVALUATION_START_TIME)
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_precisionrecallthreshold.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@

def test_precision_recall_threshold():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/threshold_precision_recall/2015-01-01'
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_response_dist.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@

def test_response_dist():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/response_dist/{}'.format(EVALUATION_START_TIME)
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_roc.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@

def test_roc():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/roc/{}'.format(EVALUATION_START_TIME)
response = test_app.get(url)
assert response.status_code == 200
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_searchmodels.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@
@pytest.mark.skip(reason="no way of currently testing this")
def test_search_models():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/search_models'
response = test_app.post(
route,
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_searchmodels_overtime.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,8 @@ def test_convert():
@pytest.mark.skip(reason="no way of currently testing this")
def test_search_models_over_time():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
route = '/evaluations/search_models_over_time'
response = test_app.post(
route,
Expand Down
2 changes: 2 additions & 0 deletions tests/test_evaluations_simpleprecisionrecall.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@

def test_simple_precision_recall():
with rig_test_client(data) as test_app:
with test_app.session_transaction() as session:
session['engine'] = 'cmpd'
url = '/evaluations/1/simple_precision_recall/{}'.format(EVALUATION_START_TIME)
response = test_app.get(url)
assert response.status_code == 200
Expand Down
6 changes: 5 additions & 1 deletion tests/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import testing.postgresql
from sqlalchemy import create_engine
import json

from flask import session

def load_json_example(route):
filepath = 'sample_json' + route + '.json'
Expand Down Expand Up @@ -139,4 +139,8 @@ def rig_test_client(data):
engine = create_engine(dburl)
setup_data(engine, data)
app.config['SQLALCHEMY_DATABASE_URI'] = dburl
app.config['SECRET_KEY'] = 'testit!'
app.config['SQLALCHEMY_BINDS'] = {
'cmpd': dburl,
}
yield app.test_client()
14 changes: 13 additions & 1 deletion webapp/__init__.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,24 @@
from flask import Flask
from flask_session import Session
from webapp.flask_util_js import FlaskUtilJs
from flask_sqlalchemy import SQLAlchemy
import flask_login
import json
from config import db_dict
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is me getting out of scope, but I'm seeing a lot of code in our projects to map environment and/or file configuration variables to database URIs. The standard I've used in the past is simply to pass database URIs around, in a single setting variable. For frameworks which don't have native support for this URI, there are often simple libraries to add support; and, it's cool that Flask-SQLAlchemy supports these natively.

DATABASE_URI=postgresql://USER@HOST:PORT/DATABASE?PARAMS

Copy link
Collaborator Author

@tweddielin tweddielin Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I absolutely agree with you that the config thing is kinda messy or redundant. I don't like that personally too. Maybe we should change it in other PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍


app = Flask(__name__,instance_relative_config=True)
app = Flask(__name__, instance_relative_config=True)
app.secret_key = "tyra american top models"
app.config['SQLALCHEMY_DATABASE_URI'] = db_dict['cmpd']['url']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you intend to keep this setting? It looks like you've just switched defaulting to SFPD with CMPD.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh and in case this setting is required, that is that Flask requires a default database connection, that's OK, of course; though, I might suggest we make it something that doesn't actually resolve, to avoid unintentionally defaulting.

Moreover, we might want a database user/schema for Tyra, e.g. for session data, which this could (at some point) point to.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(bump)


app.config['SQLALCHEMY_BINDS'] = {key: db_dict[key]['url'] for key in db_dict.keys()}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing, but if you like 😸

{key: value['url'] for (key, value) in db_dict.items()}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point.



db = SQLAlchemy(app)

app.config['SESSION_TYPE'] = 'filesystem'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where do we currently deploy Tyra? Just want to check that it has a filesystem appropriate for writing session data.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's on an ec2 machine.

sess = Session(app)
sess.init_app(app)

fujs = FlaskUtilJs(app)
login_manager = flask_login.LoginManager()
login_manager.init_app(app)
Expand Down
Loading