Skip to content

Latest commit

 

History

History
18 lines (12 loc) · 1.11 KB

DATA_README.md

File metadata and controls

18 lines (12 loc) · 1.11 KB
Raw

Steps to do a data scrape for a semester.

  1. Add the new term to Term.java, change the current Term to it, set the date on DataUtils.java, and update the term param in MitCatalogScraper. Also create a corresponding folder under data/.

  2. Run MitCatalogScraper. Revise that it's working correctly. You should have 4 files after the scrape. TODO(ipince): document scraper.

  3. ssh into the server, mysqldump the classes, books, and classbooks tables. scp them over to your local machine. ssh -p 226 rodrigo@bookspicker.com mysqldump -u bp -p bp_production books classes classbooks > bp_prod_books_classes_classbooks_.sql (go back to local shell) scp -P 226 rodrigo@bookspicker.com:~/bp_prod_books_classes_classbooks_.sql bp_prod_books_classes_classbooks_.sql mysql -u bp -p < bp_prod_books_classes_classbooks_.sql

  4. Run MitCatalogImporter. This will populate your local db with the new classes and books.

  5. Run BP locally and make sure the db is being accessed correctly.

  6. mysqldump the local db's classes, books, and classbooks tables. scp the file back to the server and load them into the db.