Data model #60

TryExceptElse · 2018-06-11T07:59:12Z

New Functionality

Data from various quarters can be retrieved.
User input is now validated before use, this improves user feedback and allows additional feedback in the future.

Improvements:

Use of a single data model class and helpers allows re-use of data access logic, and improves Extensibility.

New Modules:

model.py and contained classes now handle data access.
access.py handles access of model and data retrieval methods for view.
filter.py handles filtering of data.
input.py handles input validation.
schema.py contains input schema.

There are a number of things that still need improvement before this PR is merged, and so I'm creating this PR with the intent of getting additional eyes to find issues rather than with the intent to immediately bring it into master.

# Conflicts: # owl/model.py

…sts.

phi-line · 2018-06-13T18:53:43Z

Yeah we will have to coordinate how to handle the two datascrapers in unison. One will take care of handling live incoming data, and the other will handle archival data.

TryExceptElse · 2018-06-13T21:09:05Z

Is there a reason the advanced scraper can't be used for all quarters, possibly with an optional argument to get either all or recent data? It might be easier to standardize our stored data in this way.

# Conflicts: # owl/schema.py # tests/model/test_input.py

phi-line · 2018-06-14T06:40:06Z

Is there a reason the advanced scraper can't be used for all quarters, possibly with an optional argument to get either all or recent data? It might be easier to standardize our stored data in this way.

@TryExceptElse

I think this could be an option if the Selenium Webdriver is portable enough to work on the server. It is a possibility however. It also depends how frequently they update the advanced route. I can do a spike and see if we can nullify the need for the basic scraper. I do agree that it would make the data a lot more manageable in our DB.

phi-line · 2018-06-16T17:43:14Z

Please add documentation to the wiki in regards with how to use the new data model.

TryExceptElse · 2018-06-17T17:44:36Z

To update on where things are right now: model.py needs a substantial refactor to use the new db organization, and it may be some time before that is completed. I'll try to document as things get done.

phi-line · 2018-06-17T19:45:38Z

That's fine. I'm working on a PR to provide the data in a new format for the model. It will be served with my current open PR.

fractalbach · 2018-06-18T12:12:15Z

There are a number of things that still need improvement before this PR is merged, and so I'm creating this PR with the intent of getting additional eyes to find issues rather than with the intent to immediately bring it into master

👀

fractalbach · 2018-06-18T14:41:05Z

https://developers.google.com/protocol-buffers/

TryExceptElse · 2018-06-18T16:18:27Z

@fractalbach
I'm currently using Flask-Inputs with JsonSchema to validate inputs

phi-line · 2020-08-10T19:39:59Z

Closing this, as issues are being addressed with current PRs

FYI @madhavarshney @TryExceptElse

madhavarshney · 2020-08-10T19:41:38Z

I thought I'd leave it open until all of these changes are addressed, but that works as well 😄.

TryExceptElse added 30 commits June 1, 2018 16:44

Fixed get_key docstring.

3d90b93

Minor cleanup in server.py

e3d141c

Began implementing DataModel as a distinct class.

3ea2e63

Added initial tests for DataModel and reorganized test data.

4604b5d

Began implementing Departments + assorted fixes.

87301d5

Fixed owl infestation.

6928cb5

Added SectionQuarterView and other fixes.

3d329df

Implemented section properties and associated test methods.

537e5e5

Updated gitignore to ignore generated test files.

bde3669

Added more documentation to model.py

9a2a529

Fixed odd occasional test issue by using absolute import.

d7d64de

Minor debug message improvements, and minor additions.

652f550

Merge remote-tracking branch 'origin/data_model' into data_model

67d9412

# Conflicts: # owl/model.py

Changed import order in model.py to be pep-8 compliant.

ddbe327

Mending more minor model mistakes.

f688570

Began implementing request.py; handles validation + feedback of reque…

d8c97ea

…sts.

Implemented Request class and associated classes and tests.

b195404

Added more request tests + made additional bugfixes.

f733a5d

Added docstrings to Issue methods.

33ab187

Added custom validators for Request.Fields

42001fc

Minor request.py changes.

ace3730

Requests now can store sub-requests.

7a52913

Added some more annotations to help pycharm's type annotation detection.

46df6ef

Condensed generate_test_data; slightly more readable now?

aa3ce74

Merge remote-tracking branch 'origin/data_model' into data_model

388fdcd

Merge remote-tracking branch 'origin/data_model' into data_model

0c1681d

Modified ClassDuration to store maya interval.

2a195a0

Implemented input classes and schema.

b7347a1

Implemented section filter.

3525416

Implemented data model Accessor class.

679d49f

Added instructor and conflict filters to SectionFilter

3d790b9

TryExceptElse added 2 commits June 13, 2018 12:10

Added input tests and fixed schema + removed placeholders.

d1917c8

completed unfinished get_urls method + supporting methods.

dcfea37

TryExceptElse added 7 commits June 13, 2018 15:30

Refactored model tests to use data in temporary directories.

aa373d1

Merge branch 'new_filters' into data_model

3714f29

# Conflicts: # owl/schema.py # tests/model/test_input.py

Refactored Accessor tests to use temporary data directories.

3748667

Began implementing quarter method caching.

8ba767d

Added documentation + minor fixes.

9de217b

Renamed NotRequest to PseudoRequest

6ecc9c6

Added intersection tests for ClassDuration

f322503

phi-line mentioned this pull request Jun 14, 2018

Spike to nullify the need for '5 minute' #62

Closed

TryExceptElse added 5 commits June 14, 2018 13:55

Added serialization module and quarter duration property.

9e8a6c2

Improved deserialization hook.

386de17

Added documentation for encoder default method.

38a46fe

Test speed of and ability to cache custom serializable objects.

9cb81a0

Added documentation for _get_type_key in serial.py

0602e48

madhavarshney added the wontfix This will not be worked on label Jul 28, 2020

madhavarshney marked this pull request as draft August 10, 2020 06:00

phi-line closed this Aug 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Data model #60

Data model #60

Uh oh!

TryExceptElse commented Jun 11, 2018

Uh oh!

phi-line commented Jun 13, 2018

Uh oh!

TryExceptElse commented Jun 13, 2018

Uh oh!

phi-line commented Jun 14, 2018

Uh oh!

phi-line commented Jun 16, 2018

Uh oh!

TryExceptElse commented Jun 17, 2018

Uh oh!

phi-line commented Jun 17, 2018

Uh oh!

fractalbach commented Jun 18, 2018

Uh oh!

fractalbach commented Jun 18, 2018

Uh oh!

TryExceptElse commented Jun 18, 2018

Uh oh!

phi-line commented Aug 10, 2020

Uh oh!

madhavarshney commented Aug 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Data model #60

Data model #60

Uh oh!

Conversation

TryExceptElse commented Jun 11, 2018

Uh oh!

phi-line commented Jun 13, 2018

Uh oh!

TryExceptElse commented Jun 13, 2018

Uh oh!

phi-line commented Jun 14, 2018

Uh oh!

phi-line commented Jun 16, 2018

Uh oh!

TryExceptElse commented Jun 17, 2018

Uh oh!

phi-line commented Jun 17, 2018

Uh oh!

fractalbach commented Jun 18, 2018

Uh oh!

fractalbach commented Jun 18, 2018

Uh oh!

TryExceptElse commented Jun 18, 2018

Uh oh!

phi-line commented Aug 10, 2020

Uh oh!

madhavarshney commented Aug 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants