Skip to content
This repository has been archived by the owner on Jan 23, 2019. It is now read-only.

Identify endpoints / Pages for data sources #2

Open
envp opened this issue Feb 8, 2017 · 3 comments
Open

Identify endpoints / Pages for data sources #2

envp opened this issue Feb 8, 2017 · 3 comments
Assignees
Projects

Comments

@envp
Copy link
Contributor

envp commented Feb 8, 2017

#We need to identify API endpoints for various pieces of information to completely build the dependencies for all of the courses.

Status:

Purpose Endpoint Docs/Wiki Comments
Schedule https://one.uf.edu/apix/soc/schedule/?term= $TERM &category= $CATEGORY &lastrow=$LASTROW TODO There are more parameters check the wiki for info.
Description https://one.uf.edu/apix/soc/cdesc/ $COURSE_CODE TODO This is the only query parameter, however we may need to build a crude parser for the response containing pre-reqs/coreqs.
Dependencies TODO TODO Course description offers some rough unstructured info, but can we do better?
Evaluations TODO TODO Evaluate using the evaluations page for this info

Assignments:

Dependencies - @pikachuster

Updates

Schedule - Scrapy project
Dependencies / Descriptions - Currently being added to the scrapy based code.
Evaluations - Probably going to use scrapy / selenium here whichever comes through first reliably

@envp envp created this issue from a note in Tasks (Data Collection) Feb 8, 2017
@ghost
Copy link

ghost commented Feb 9, 2017

Regarding evaluations, probably should start looking at https://evaluations.ufl.edu/results/. Tried looking at the page source of a single evaluation, seems some Javascript magic is going on usually towards the end? CRTL+F for "api" didn't seem to bring up anything relevant, except that they seem to be using JQuery.

Saw something interesting, <script id="evaluationTemplate" type="text/x-jquery-tmpl"> on line 165 of my sample page, https://evaluations.ufl.edu/results/instructor.aspx?ik=-957811086. Also of interest is <script type="text/javascript"> on line 276 and on...

Those two scripts, combined, seem to be pulling data from somewhere and displaying it on screen.

@ghost
Copy link

ghost commented Feb 9, 2017

Using Firefox Inspector on a single table cell, I'm able to see the raw data between the table tags. But just where and how is the data even coming in?

@envp
Copy link
Contributor Author

envp commented Feb 10, 2017

Nice find! It seems to be a jQuery template, which gets filled using the JSON of the response. I need to compare that with the response json fully to be sure if that is all it is doing, since it sends only a single request to the API.

Will edit with confirmation

@envp envp assigned ghost Feb 15, 2017
@envp envp self-assigned this Feb 27, 2017
@mirdaki mirdaki added the data label Mar 22, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Tasks
Data Collection
Development

No branches or pull requests

2 participants