Scraps the Cornell rosters to analyze and find interesting maps and insights so as not to constantly check back and forth during the pre-enrollments.
Parse the roster pages subject by subject for each semester.
- Grab the list of semesters that uses this new version of the roster (from fall 2014 to today)
- For each semester, grab the list of course codes (subjects) that are offered that semester
- Click into the link of each subject code and Cornell roster will show all the courses that have that subject code.
- Generate the CSV file that captures all the relevant information for each of the section offerings. They include lecture/discussion, professor, date time location, and more (see actual implementation).
- The main part: Create a Beautiful Soup parser that filters relevant HTML tag and its attributes (known from playing with the inspect tool on chrome).
- Cache the HTML information into a relevant folder while traversing through the semesters and subjects in that semester. This way, I do not have to request again.
- While traversing, add a check on whether I already have these information cached. If so, grab the HTML text instead of sending another request. If not, then send the request but also write the HTML into a file so later when we want to resume we can simply skip the long waiting part.
A lookup system not based on semesters that we originally parsed the roster from, but with professors.
This allows us to check the past offerings from the professor and with the help of Cornell Median Grades, we can better plan the courses to work around and get good GPAs.
Create a class that checks all the csv files in that directory and finds a list of lecture type sections that corresponds to that professor. Return that list.
- Helper function that finds all the distinct professors from the section
- For each professor, run that class and with the List of CSV rows to grab the relevant datas, save these rows into a file (in the folder called professors).