Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mutualise HTML Crawler for better performances #36

Merged
merged 9 commits into from
Jun 7, 2021
Merged

Mutualise HTML Crawler for better performances #36

merged 9 commits into from
Jun 7, 2021

Conversation

Tom32i
Copy link
Collaborator

@Tom32i Tom32i commented Sep 21, 2020 •
edited
Loading

Before
Capture d’écran 2021-05-03 à 12 16 58

After
Capture d’écran 2021-05-03 à 12 17 45

Refacto plus clean :

  • 1 interface pour gérer la génération de crawler
  • 1 implementation "naive" qui fait comme avant : génère le crawler puis dump le HTML.
  • 1 implementation "optimisée" : génère le crawler et l'utilise pour tout les processors, dump le HTML uniquement à la fin.

L'implementation optimisée n'est qu'une option activable.

@Tom32i Tom32i self-assigned this Sep 21, 2020
@Tom32i Tom32i force-pushed the html-processor branch 2 times, most recently from eb86d7f to ad5e8d2 Compare September 21, 2020 09:53
Base automatically changed from assets-management to master September 21, 2020 09:53
@Tom32i Tom32i changed the title Mutualise HHTML Crawler for better performances Mutualise HTML Crawler for better performances Sep 21, 2020
@Tom32i Tom32i force-pushed the html-processor branch 2 times, most recently from ce29c4f to f255a01 Compare May 3, 2021 09:49
@Tom32i Tom32i added the enhancement New feature or request label May 3, 2021
@Tom32i Tom32i marked this pull request as ready for review May 3, 2021 09:54
@Tom32i Tom32i mentioned this pull request May 3, 2021
Merged
ogizanagi
ogizanagi reviewed May 6, 2021
View reviewed changes
Copy link
Member

@ogizanagi ogizanagi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for the record, tested as well on my-side on the elao_ website:

Before

Capture d’écran 2021-05-06 à 15 42 48

After

Capture d’écran 2021-05-06 à 15 46 17

src/Service/SharedHtmlCrawlerManager.php Outdated Show resolved Hide resolved
ogizanagi
ogizanagi reviewed Jun 7, 2021
View reviewed changes
src/Behaviour/HtmlCrawlerManagerInterface.php Outdated Show resolved Hide resolved
src/Behaviour/ProcessorInterface.php Show resolved Hide resolved
src/DependencyInjection/Configuration.php Show resolved Hide resolved
src/Service/NaiveHtmlCrawlerManager.php Outdated Show resolved Hide resolved
src/Service/NaiveHtmlCrawlerManager.php Outdated Show resolved Hide resolved
src/Service/SharedHtmlCrawlerManager.php Outdated Show resolved Hide resolved
src/Service/SharedHtmlCrawlerManager.php Outdated Show resolved Hide resolved
src/Service/NaiveHtmlCrawlerManager.php Outdated Show resolved Hide resolved
src/Processor/TableOfContentProcessor.php Outdated Show resolved Hide resolved
src/Behaviour/HtmlCrawlerManagerInterface.php Show resolved Hide resolved
Tom32i and others added 4 commits June 7, 2021 10:43
@Tom32i @ogizanagi
Refactoring from @ogizanagi
efe6db9 
Thanks!

Co-authored-by: Maxime Steinhausser <ogizanagi@users.noreply.github.com>
Fixed lint
3e152c2
Removed force option
74a064a
Updated documentation about processors
0dcfd72
@Tom32i Tom32i merged commit 9b81e96 into master Jun 7, 2021
@Tom32i Tom32i deleted the html-processor branch June 7, 2021 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogizanagi ogizanagi ogizanagi approved these changes

Assignees

@Tom32i Tom32i

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Tom32i @ogizanagi