Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up TSV fetching #543

Merged
merged 2 commits into from
Feb 10, 2021
Merged

Speed up TSV fetching #543

merged 2 commits into from
Feb 10, 2021

Conversation

shorowit
Copy link
Contributor

@shorowit shorowit commented Feb 9, 2021

Pull Request Description

A bug in the buildstock.rb get_measure_args_from_option_names() method was causing the entirety of every TSV to be processed even when the option(s) of interest had already been found. As the number and length of TSVs has grown, so has this bottleneck.

The bug causes both the BuildExistingModel and ApplyUpgrade measures to be far slower than they should be.

Time spent inside the method when running the first building w/ an upgrade from test_measures_osw.rb:

  • develop branch: 62.7 seconds => 6.7 seconds
  • restructure-v3 branch: 38.5 seconds => 3.4 seconds

Checklist

Not all may apply:

  • Unit tests have been added or updated
  • All rake tasks have been run, and pass
  • Documentation has been modified appropriately
  • Any new options are added to project_testing
  • project_testing runs without any failures
  • No unexpected regression test changes
  • All tests are passing
  • The changelog has been updated appropriately
  • This branch is up-to-date with develop

For more information on how to perform these checklist items, see the documentation's Advanced Tutorial.

…d was causing the entire TSV to be processed even when the option(s) of interest had already been found.
Copy link
Contributor

@joseph-robertson joseph-robertson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@shorowit shorowit merged commit eb403f6 into develop Feb 10, 2021
@shorowit shorowit deleted the speed_up_tsv_fetching branch February 10, 2021 01:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants