Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[0.1.dev21] Beter data loading utilities #74

Merged
merged 20 commits into from
Dec 11, 2018
Merged

[0.1.dev21] Beter data loading utilities #74

merged 20 commits into from
Dec 11, 2018

Conversation

smmaurer
Copy link
Member

@smmaurer smmaurer commented Dec 11, 2018

This PR updates the utility that templates use to load estimation or simulation data from Orca.

Functionality

It removes TemplateStep._get_df() and replaces it with a new function utils.get_data(). See snapshot here: utils.py#L127-L202

  • draws data from one or more Orca tables
  • can apply row filters
  • can drop non-relevant columns

Automatically detects columns included in the model expression and filters. Ignores any columns not found in the source tables, to support use cases where data is assembled separately for choosers and alternatives and then merged together -- the model expression in that case would include terms from both sets of tables.

Discussion

This should fix the problem that @mxndrwgrdnr raised in this comment on issue #67, where incidental overlapping columns in the choosers and alternatives table were preventing a MergedChoiceTable from being built properly. After this update, incidental columns would be dropped.

This should also improve performance -- reducing memory usage and possibly improving speed a bit.

Other changes

  • templates that used TemplateStep._get_df() have been updated to use utils.get_data() instead

  • templates using the older TemplateStep._get_data() method are unchanged; that migration will be easier as part of the switch to settings objects (see issue Settings objects for groups of common parameters #54)

  • implemented the run_all() method for the Segmented MNL template

  • updates to unit tests

Versioning

  • updates the version number to 0.1.dev21

Copy link
Member

@mxndrwgrdnr mxndrwgrdnr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, Sam. Thanks for getting this done so quickly.

@coveralls
Copy link

coveralls commented Dec 11, 2018

Pull Request Test Coverage Report for Build 79

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 37 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.4%) to 89.817%

Files with Coverage Reduction New Missed Lines %
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/large_multinomial_logit.py 4 94.97%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/segmented_large_multinomial_logit.py 5 91.89%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/utils.py 11 83.7%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/shared.py 17 71.95%
Totals Coverage Status
Change from base Build 70: 0.4%
Covered Lines: 979
Relevant Lines: 1090

💛 - Coveralls

@coveralls
Copy link

coveralls commented Dec 11, 2018

Pull Request Test Coverage Report for Build 81

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 37 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.4%) to 89.817%

Files with Coverage Reduction New Missed Lines %
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/large_multinomial_logit.py 4 94.97%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/segmented_large_multinomial_logit.py 5 91.89%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/utils.py 11 83.7%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/shared.py 17 71.95%
Totals Coverage Status
Change from base Build 70: 0.4%
Covered Lines: 979
Relevant Lines: 1090

💛 - Coveralls

1 similar comment
@coveralls
Copy link

Pull Request Test Coverage Report for Build 81

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 37 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.4%) to 89.817%

Files with Coverage Reduction New Missed Lines %
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/large_multinomial_logit.py 4 94.97%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/segmented_large_multinomial_logit.py 5 91.89%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/utils.py 11 83.7%
/home/travis/build/UDST/urbansim_templates/urbansim_templates/models/shared.py 17 71.95%
Totals Coverage Status
Change from base Build 70: 0.4%
Covered Lines: 979
Relevant Lines: 1090

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants