Archive scraper

[phi-line]

New Functionality:

• Advanced Scraper that gets all data back until 2010 (100k+ courses, 64 quarters, 2 campuses)
• Selenium cookie scraper that allows for authenticated requests to MyPortal

New Modules:

• scrape_advanced.py Archival scraper
• selenium_login.py Logs you into MyPortal and gathers cookie

To run this PR:
Set a couple environment variables either in Pycharm or in your bash_rc. They should contain your MyPortal login details.

MP_USER="INSERT CWID"
MP_PASS="INSERT PASSWORD"

pip install pipenv

pipenv install

pipenv run scrape_advanced.py

Watch as the magic happens (takes ~30 mins)

[phi-line]

Fixes #33

[TryExceptElse]

Looks good overall, but there are a few things that could probably be improved before this is pulled.

[TryExceptElse]

Are we sure we want to exclude all the template and example .html files?

[TryExceptElse]

This function might be broken up into smaller functions.

[phi-line]

This has been addressed with commit 363b07b

[TryExceptElse]

This can be condensed using a single itertools.product call.

[TryExceptElse]

If this is used in scrape_advanced.py, it can be declared and used there to reduce clutter in the setup.py module. The same applies to any other constants used only within a single module.

[TryExceptElse]

Catching all AttributeErrors produced in such a large are of code could easily swallow many errors that we do not want to be caught, and can be very hard to debug. Is there a way to avoid excepting these from such a large area, or entirely?

[DanielNghiem]

After a 'pipenv install` and running scrape_advanced.py, I get the following:

Traceback (most recent call last):
  File "scrape_advanced.py", line 12, in <module>
    from selenium_login import scrape_cookies, kill_driver
  File "/Users/danielnghiem/Projects/OwlAPI/selenium_login.py", line 13, in <module>
    driver = webdriver.Chrome(chrome_options=chrome_options)
  File "/Users/danielnghiem/.local/share/virtualenvs/OwlAPI-TkGlEiua/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 68, in __init__
    self.service.start()
  File "/Users/danielnghiem/.local/share/virtualenvs/OwlAPI-TkGlEiua/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 83, in start
    os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

It looks like I need to download chromedriver and add it to my PATH.
@phi-line : Can you add the setup instructions?

[phi-line]

@DanielNghiem I have addressed your comment in this wiki page. Please feel free to add anything that you think is incorrect about the setup instructions I have provided.

[phi-line]

@TryExceptElse please merge this PR when you get a chance so I can move on with the new data format (all CRN in one table). I'd like to separate out these two PRs instead of lumping everything into one.