Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speedups #688

Merged
merged 2 commits into from
Feb 8, 2022
Merged

Speedups #688

merged 2 commits into from
Feb 8, 2022

Conversation

sparkiegeek
Copy link
Contributor

Done

Improved performance of parsing MAAS document index from ~1.5s to 0.9s

  • Keep soup around after parsing index, use for preamble
  • [URL Mapping] Don't use CSS selectors for well structured data
  • [URL Mapping] Don't do big string manipulation on soup

QA

  • Run the python tests
  • Verify index document loads, nav is correct, urls are mapped, redirects are correct

Issue / Card

#559

Screenshots

[if relevant, include a screenshot]

@webteam-app
Copy link

Demo starting at https://maas-io-688.demos.haus

@sparkiegeek sparkiegeek force-pushed the speedups branch 2 times, most recently from 939c019 to 3a689cb Compare February 7, 2022 15:17
@codecov
Copy link

codecov bot commented Feb 7, 2022 •
edited
Loading

Codecov Report

Merging #688 (1ddeb07) into main (1b789a9) will increase coverage by 23.83%.
The diff coverage is 92.20%.

❗ Current head 1ddeb07 differs from pull request most recent head 1dd2764. Consider uploading reports for the commit 1dd2764 to get more accurate results

Impacted Files Coverage Δ
webapp/doc_parser.py 66.37% <87.17%> (+50.51%) ⬆️
tests/test_doc_parser.py 97.36% <97.36%> (ø)

@sparkiegeek sparkiegeek added python Pull requests that update Python code Review: Code needed Review: QA needed labels Feb 7, 2022
@sparkiegeek
More improvements to performance of FastDocsParser
1dd2764 
* Keep soup around after parsing index, use for preamble
* [URL Mapping] Don't use CSS selectors for well structured data
* [URL Mapping] Don't do big string manipulation on soup
jkfran
jkfran approved these changes Feb 8, 2022
View reviewed changes
Copy link
Contributor

@jkfran jkfran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. It's a shame that we can't use for Maas the latest version of our parser that all the other websites are using. I am sure these improvements can be included there.

@sparkiegeek sparkiegeek merged commit b10e81b into canonical:main Feb 8, 2022
@sparkiegeek sparkiegeek deleted the speedups branch February 8, 2022 10:51
@jkfran jkfran mentioned this pull request Feb 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkfran jkfran jkfran approved these changes

Assignees

@jkfran jkfran

Labels
python Pull requests that update Python code Review: Code +1 Review: QA +1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@sparkiegeek @webteam-app @jkfran