Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-Allocating Frozen Tier ShardBytes Can be very Slow on Windows #78834

Open
original-brownbear opened this issue Oct 7, 2021 · 1 comment
Assignees
Labels
>bug :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team

Comments

@original-brownbear
Copy link
Member

There are two issues with pre-allocating the frozen tier shared bytes on windows:

  1. On Windows we do not use a FS native way like fallocate on Linux to pre-allocate the large shared cache file. This can take a significant amount of time for larger files on slower disks. We should look into a native way to speed the process up and can probably also speed it up directly by increasing the write chunks size we use when allocating the file form the current 8k.
  2. On node restart we will try to pre-allocate the file and fail on the attempt to do it via a native call. As a result we delete only to then write it again from scratch, thus running into the first problem again on node restarts.
@original-brownbear original-brownbear added >bug :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs labels Oct 7, 2021
@original-brownbear original-brownbear self-assigned this Oct 7, 2021
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Oct 7, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team
Projects
None yet
Development

No branches or pull requests

2 participants