This project is a library consisting of a command line interface and a client for interacting with Coursera's research exports. Up to date documentation of the data provided by Coursera for research purposes is available in the Partner Resource Center , Coursera Data Exports Guide.
To install this package, execute:
pip install courseraresearchexports
pip is a python package manager.
If you do not have
pip installed on your machine, please follow the
installation instructions for your platform.
virtualenv venv source venv/bin/activate pip install courseraresearchexports
containers subcommand requires
docker to already be installed
on your machine. Please see the docker installation instructions for platform
To enable tab autocomplete, please install argcomplete using
pip install autocomplete and execute
activate-global-python-argcomplete. Open a new shell and
press tab for autocomplete functionality.
See the argcomplete documentation for more details.
Authorize your application using courseraoauth2client:
courseraoauth2client config authorize --app manage_research_exports
To use the
containers functionality, a docker instance must be running.
Please see the docker getting started guide
for installation instructions for your platform.
If you have a previously installed version of courseracourseexports, execute:
pip install courseraresearchexports --upgrade
This will upgrade your installation to the newest version.
Command Line Interface
The project includes a command line tool. Run:
for a complete list of features, flags, and documentation. Similarly,
documentation for the subcommands listed below is also available (e.g. for
jobs) by running:
courseraresearchexports jobs -h
Submit a research export request or retrieve the status of pending and completed export jobs.
Creates an data export job request and return the export request id. To create a data export requests for all available tables for a course:
courseraresearchexports jobs request tables --course_slug $COURSE_SLUG \ --purpose "testing data export"
$COURSE_SLUG with your course slug (The course slug is the part after
/learn in the url. For
the slug is machine-learning).
You may also use --course_id if you know your course id. This is also necessary for non publically available courses.
If a more limited set of data is required, you can specify which schemas are included with the export. (e.g. for the demographics tables):
courseraresearchexports jobs request tables --course_slug $COURSE_SLUG \ --schemas demographics --purpose "testing data export"
For more information on the available tables/schemas, please refer to the Coursera Data Exports Guide.
If you are a data coordinator, you can request that user ids are linked between domains of the data export:
courseraresearchexports jobs request tables --course_slug $COURSE_SLUG \ --purpose "testing data export" --user_id_hashing linked
Data coordinators can also request clickstream exports:
courseraresearchexports jobs request clickstream --course_slug $COURSE_SLUG \ --interval 2016-09-01 2016-09-02 --purpose "testing data export"
By default, clickstream exports will cache results for days already exported. To ignore the cache and request exports for the entire date range, pass in the flag
We have 2 rate limits for creating jobs: up to 15 jobs per hour per user, and for each scope (course/specialization/group), one request per hour.
Lists the details and status of all data export requests that you have made:
courseraresearchexports jobs get_all
Retrieve the details and status of an export request:
courseraresearchexports jobs get $EXPORT_REQUEST_ID
Download a completed table or clickstream to your local destination:
courseraresearchexports jobs download $EXPORT_REQUEST_ID
Due to the size of clickstream exports, we persist download links for completed clickstream export requests on Amazon S3. The clickstream data for each day is saved into a separate file and download links to these files can be retrieved by running:
courseraresearchexports jobs clickstream_download_links --course_slug $COURSE_SLUG
Creates a docker container using the postgres image and loads export data
into a postgres database on the container. To create a docker container
from an export, first
request an export using the
jobs command. Then,
$EXPORT_REQUEST_ID, create a docker container with:
courseraresearchexports containers create --export_request_id $EXPORT_REQUEST_ID
This will download the data export and load all the data into the database running on the container. This may take some time depending on the size of your export. To create a docker container with an already downloaded export (please decompress the archive first):
courseraresearchexports containers create --export_data_folder /path/to/data_export/
After creation use the
list command to check the status of the
container and view the container name, database name, address and port to
connect to the database. Use the db connect $CONTAINER_NAME command to open
a psql shell.
Lists the details of all the containers created by
courseraresearchexports containers list
Start a container:
courseraresearchexports containers start $CONTAINER_NAME
Stop a container:
courseraresearchexports containers stop $CONTAINER_NAME
Remove a container:
courseraresearchexports containers remove $CONTAINER_NAME
Open a shell to a postgres database:
courseraresearchexports db connect $CONTAINER_NAME
Create a view in the postgres database. We are planning to include commonly used denormalized views as part of this project. To create one of these views (i.e. for the demographic_survey view):
courseraresearchexports db create_view $CONTAINER_NAME --view_name demographic_survey
If you have your own sql script that you'd like to create as a view run:
courseraresearchexports db create_view $CONTAINER_NAME --sql_file /path/to/sql/file/new_view.sql
This will create a view using the name of the file as the name of the view, in this case "new_view".
Note: as user_id columns vary with partner and user id hashing, please refer to the exports guide for SQL formatting guidelines.
Export a table or view to a csv file. For example, if the demographic_survey was created in the above section, use this commmand to create a csv:
courseraresearchexports db unload_to_csv $CONTAINER_NAME --relation demographic_survey --dest /path/to/dest/
List all the tables present inside a dockerized database:
courseraresearchexports db list_tables $CONTAINER_NAME
List all the views present inside a dockerized database:
courseraresearchexports db list_views $CONTAINER_NAME
Using courseraresearchexports on a machine without a browser
Sometimes, a browser is not available, making the oauth flow not possible. Commonly, this occurs when users want to automate the data export process by using an external machine.
To get around this, you may generate the access token initially on a machine with browser access [e.g your laptop]. The access token is serialized in your local file system at ~/.coursera/manage_research_exports_oauth2_cache.pickle.
Requests after the first can use the refresh token flow, which does not require a browser. By copying the initial pickled access token to a remote machine, that machine can continue to request updated data.
Bugs / Issues / Feature Requests
Please us the github issue tracker to document any bugs or other issues you encounter while using this tool.
Developing / Contributing
We recommend developing
courseraresearchexports within a python
To get your environment set up properly, do the following:
virtualenv venv source venv/bin/activate python setup.py develop pip install -r test_requirements.txt
To run tests, simply run:
Code should conform to pep8 style requirements. To check, simply run:
pep8 courseraresearchexports tests