Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add redo snapshotter. #1301

Merged
merged 5 commits into from Jun 6, 2020

Conversation

tejal29
Copy link
Member

@tejal29 tejal29 commented Jun 6, 2020

The Redo snapshot mode, use the following heuristics to calculate if a file has changed

  1. File Mod Time
  2. File Size
  3. File Mode
  4. File Group Id, UserId

This is inspired from https://apenwarr.ca/log/20181113
Redo snapshotter has proved to increase the performance for large file systems.
The performance could be upto 56%
See more at graph
As the number of files increase, time spent in hashing increases

@googlebot googlebot added the cla: yes CLA signed by all commit authors label Jun 6, 2020
@tejal29
Copy link
Member Author

tejal29 commented Jun 6, 2020

Performance Improvements


Number of Files | Total Build Time Full Snapshot - current | Total Build Time Redo Snapshot | % decrease
-- | -- | -- | --
10000 | 13.834973 | 14.722817 | -6.42%
50000 | 47.926338 | 45.414848 | 5.24%
100000 | 96.440554 | 91.042636 | 5.60%
200000 | 187.346273 | 179.147903 | 4.38%
300000 | 341.233622 | 263.565265 | 22.76%
500000 | 1404.410121 | 562.507168 | 59.95%
700000 | 1772.379601 | 763.790115 | 56.91%

@tejal29 tejal29 merged commit 1033ad7 into GoogleContainerTools:master Jun 6, 2020
@tejal29 tejal29 added the area/performance issues related to kaniko performance enhancement label Aug 22, 2020
@tejal29 tejal29 deleted the addRedoSnapshotter branch April 26, 2021 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/performance issues related to kaniko performance enhancement cla: yes CLA signed by all commit authors
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants