README.md

CommonClarity

####Built by Katie Schick

####The Project

CommonClarity transforms standardized test data into a visual, interactive reporting dashboard for teachers, to help inform and target instruction.

Data-driven instruction has grown in popularity with the rise of standardized testing. In many districts around the country, students take standardized tests nearly every other month. These tests generate a huge amount of data, which teachers are expected to utilize to inform instruction; however, they lack the time and the tools to make sense of it.

CommonClarity makes it easy for teachers to upload their data and instantly generate a dashboard of reports, showing their students' strengths and weaknesses according to Common Core State Standards. Data are grouped in useful ways to expose patterns and track progress, so that teachers can better help students improve.

####Table of Contents

• Technologies Used
• How It Works
• Product Details
  • The Database
  • The RESTful API
    • Optimizing Queries
  • AngularJS and D3
• Product Screenshots
  • Student Set-up
  • Easy Data Import
  • Reporting Dashboard
• [Try It Yourself!] (#try-it-yourself)

####Technologies Used

CommonClarity was written using AngularJS, D3, JavaScript, Python, Flask, HTML5, CSS3/Sass, jQuery, BeautifulSoup, SQLAlchemy, and Postgresql.

Click here for a full dashboard view.

####How It Works

Teachers can quickly set up their classes of students in the system and easily import the CSV files of their test data. Those files are then parsed by matching student names and Common Core State Standards to the database and entering scores for each student according to each standard.

Reports are generated by aggregating the data across all classes, by individual class, and by individual student and generating an interactive dashboard. This dashboard shows teachers the following information:

• 1. The top Common Core State Standards students are struggling with, and which students have not met each of those standards
• 2. The students who are struggling the most and the percentage of standards each of those students has not met
• 3. A high-level overview of student performance on the most recent test
• 4. A detailed view of student performance, broken out by all tests and by Common Core State Standard
• 5. Student performance on the most recent test, compared to the rest of the school and the district overall
• 6. A list of the standards tested on the most recent test
• 7. An individual class's performance on the most recent test, broken out by student
• 8. Student improvement on the most recent test from the test prior

####Product Details

#####The Database

The Postgres database uses SQLAlchemy as its ORM and contains tables for users, cohorts, student cohorts, tests, Common Core State Standards, scores, and normed scores.

To seed the Common Core State Standards table, I scraped the standards from the CCSS website using Beautiful Soup and wrote the data into a CSV file. The data were cleaned using the script parse.py and seeded into the database table using the script standards_seed.py.

The script scores_seed.py was used to seed the tests and scores tables before the test upload function was built.

#####The RESTful API

The module api.py contains the RESTful API for CommonClarity. Each function is called by a Flask route in app.py and, when called, queries the server, manipulates and aggregates the data, and converts the response into JSON to display on the front-end.

######Optimizing Queries

Upon reviewing my RESTful API, I noticed some inefficiencies in my queries: namely, that I wasn't using the backref relationships in my database as effectively as I could, and I had quite a lot of nested for-loops. After targeting those issues, I was able to significantly improve performance time for the queries that were taking a long time:

Query                           Pre-optimization  Post-optimization   Improvement
All cohorts by standard         1912ms            133ms               93% faster
Top "falling behind" standards  1940ms            370ms               81% faster
All single cohort data          3659ms            852ms               77% faster

#####AngularJS and D3

CommonClarity is built as a single-page webapp using AngularJS for the front end. Flask is used to serve the initial template (index.html) and the other pages are in partials. The routes and controllers are configured in app.js. The controllers in controllers.js make calls to RESTful API endpoints and Angular directives are used to display the JSON.

The reports are drawn using D3 and a custom Angular directive (directives.js). Each directive contains a render and a watch function: to watch for changes to the data and re-render the graph when changes are detected. Each graph is scaled according to the amount of data returned and features legends and tooltips.

####Product Screenshots

Click here to see a screenshot of the full dashboard, and scroll down for more product screenshots.

#####Student Set-up

Quickly set up your classes of students, either by uploading a CSV file or by manually entering each name.

#####Easy Data Import

Easily import your standardized test data into the system.

#####Reporting Dashboard

Once your data is imported, the reporting dashboard will automatically update with your new data.

Click here for a full dashboard view.

Use the drill-down menus to view all your classes at once, one class individually, or one particular student.

Target an individual student to see how that student is performing.

See how students are performing, according to specific Common Core State Standards.

Get a visual overview of how all the students in a class are performing.

####Try It Yourself!

#####Environment

1. Clone the repository:

$ git clone https://github.com/kkschick/CommonClarity.git

2. Create and activate a virtual environment in the same directory:

$ pip install virtualenv
$ virtualenv env
$ . env/bin/activate 

3. Install the required packages using pip:

(env)$ pip install -r requirements.txt

#####Database

1. To run the Postgres server, download and run postgres.app, and follow the instructions to set up Postgres on your machine—instructions for Mac.

2. Create the database in PostgreSQL:

$ psql
# CREATE DATABASE clarity;

3. And then, in your virtual environment, create the database using Python:

(env)$ python -i model.py
(env)$ create_db()

4. Still in your virtual environment, create the database tables and seed the standards table:

(env)$ python model.py
(env)$ python standards_seed.py

5. Run the app:

(env)$ python app.py

6. Point your browser to:

http://localhost:5000/