Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up loading of comparison page #1616

Closed
1 of 2 tasks
Kr0nox opened this issue Feb 27, 2024 · 1 comment
Closed
1 of 2 tasks

Speed up loading of comparison page #1616

Kr0nox opened this issue Feb 27, 2024 · 1 comment
Assignees
Labels
enhancement Issue/PR that involves features, improvements and other changes minor Minor issue/feature/contribution/change report-viewer PR / Issue deals (partly) with the report viewer and thus involves web-dev technologies
Milestone

Comments

@Kr0nox
Copy link
Member

Kr0nox commented Feb 27, 2024

Currently the submissionFileIndex is just one big file containing information about all files of all submissions. This file can be pretty large for some datasets. Since the entire file has to be read and parsed when opening a comparison this can lead to slowdowns.
There are two possbile improvements for this:

  • Only write submissions into the index that are part of an exported comparison. Depending on the dataset and the amount of exported comparisons this can reduce the size of the index file significantly. This change can be conisdered non breaking, since it would be backwards compatibile, allthough technically there is less information in the report than previously.
  • Write an index file per submission. This would take the load from reading one big file to only reading two short files. This would change the report thus could be considered breaking, but it could be implemented backwards compatible, by doing one thing when the big index file is present and another when the short ones are
@tsaglam tsaglam added enhancement Issue/PR that involves features, improvements and other changes minor Minor issue/feature/contribution/change report-viewer PR / Issue deals (partly) with the report viewer and thus involves web-dev technologies labels Feb 28, 2024
@tsaglam tsaglam added this to the 5.1.0 milestone Mar 27, 2024
@Kr0nox Kr0nox self-assigned this Mar 29, 2024
@Kr0nox Kr0nox modified the milestones: 5.1.0, 6.0.0 Apr 2, 2024
@Kr0nox
Copy link
Member Author

Kr0nox commented Jul 27, 2024

Write an index file per submission. This would take the load from reading one big file to only reading two short files. This would change the report thus could be considered breaking, but it could be implemented backwards compatible, by doing one thing when the big index file is present and another when the short ones are

This was implemented in https://github.com/jplag/JPlag/tree/faster-submission-index. There is no measurable time gain measurable

@Kr0nox Kr0nox closed this as completed Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Issue/PR that involves features, improvements and other changes minor Minor issue/feature/contribution/change report-viewer PR / Issue deals (partly) with the report viewer and thus involves web-dev technologies
Projects
None yet
Development

No branches or pull requests

2 participants