Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactoring scan pipeline for oversized lines #564

Merged
merged 19 commits into from
Jun 17, 2024
Merged

Refactoring scan pipeline for oversized lines #564

merged 19 commits into from
Jun 17, 2024

Conversation

babenek
Copy link
Contributor

@babenek babenek commented Jun 14, 2024 •
edited
Loading

Description

Please include a summary of the change and which is fixed.

  • Yield target line with chunks when it is over maximal line length
  • Use subtext near value for ML features to line search. Otherwise it takes long time during pre-processing
  • Use compare method for candidates to skip possible duplicates in chunks overlaps

PS: benchmark performance may fail due last release has build issse #565

How has this been tested?

Please describe the tests that you ran to verify your changes.

  • UnitTest
  • Benchmark
  • long line file scan ~20Mb in ~40sec

@babenek babenek changed the title refactoring of long line scan Refactoring scan pipeline for oversized lines Jun 17, 2024
@babenek babenek marked this pull request as ready for review June 17, 2024 07:45
@babenek babenek requested a review from a team as a code owner June 17, 2024 07:45
Yullia
Yullia approved these changes Jun 17, 2024
View reviewed changes
Copy link
Contributor

@Yullia Yullia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@babenek babenek merged commit 606a28d into Samsung:main Jun 17, 2024
25 of 28 checks passed
@babenek babenek deleted the auxiliary branch June 17, 2024 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xDizzix xDizzix xDizzix approved these changes

@Yullia Yullia Yullia approved these changes

@iuriimet iuriimet Awaiting requested review from iuriimet

@csh519 csh519 Awaiting requested review from csh519

@kmnls kmnls Awaiting requested review from kmnls

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@babenek @xDizzix @Yullia