vmbackupmanager: incompatibility issues with local FS on windows #4704
Comments
|
Hi @tekert It includes a fix for paths join which creates a path with the following path |
|
Yes, it there a way to execute the backup without waiting for the next hour? |
|
@tekert You can add |
|
Seems it did something, but crashes after a succesfull backup. |
|
@tekert Thank you for the feedback! |
|
No problem zek, wish i could help with the code. Ran the new exe two times, one with a current backup of the last run, and another restarting victoria metrics, deleting all backups and any snapshot leftover, clean state, same error log. Could the data from that file be corrupted for the backup but somehow causing no troubles for victoriametrics? |
|
Another run again results in another error log: |
|
@tekert Thanks a lot for your testing. It seems to me like it will be better to set up a proper testing environment for those fixes and verify it works after fixing all the issues. |
zekker6
changed the title
|
I could send you the data in private, is just test data from the last month, its only 142mb. |
|
@tekert Hello! May I ask you to test this build and see if that works for you? We've tested this internally and it seems to be working well, wanted to confirm that it works in your case as well. |
|
@zekker6 Executing same command as before: ✔️ Shutting down the service (Control+C) and restarting it also works correctly. ✔️ Seems everything is working. |
|
Oh, i noticed snapshots are not being deleted after the backup. It's that intended? |
|
Snapshots should be renamed to something containing Windows does not allow deleting files in case any other process is using it. And this is frequently the case with snapshots since snapshots in VM are based on creating symlinks, so the file can still be in use. |
|
Ouch, false alarm, an additional snapshot from some testing remained on the system and mistaked it for a backup, doing delete_all and restarting the previous test confirms |
|
Unfortunately the only option to remove those and be sure that this will work correctly is to periodically restart the VM. |
|
I was thinking, since the write interval from memory can be configured, maybe handles can be restarted internally and resume writes after? |
|
Apart writing data from memory storage can also perform:
So the problem with deleting even symlinked files is that this file can be opened for reading at query execution path.
This seems like we need to stop all DB operations anyway, very similar to what happens on graceful shutdown during restart. That's why we decided not to implement this as a built-in functionality to keep code base simpler. |
|
I understand, i see, thanks for the explanations zek, I'll try to schedule a restart because last time i checked the data folder increased 4x in size for the last month. |
|
@tekert Lets leave this issue open until fixes which are included into builds I've provided will be merged into upstream version and released. We'll close this issue once those fixes will be released. |
zekker6
added a commit
that referenced
this issue
Aug 1, 2023
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
zekker6
added a commit
that referenced
this issue
Aug 1, 2023
Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
zekker6
added a commit
that referenced
this issue
Aug 1, 2023
|
@zekker6 I was wondering if the handles for those hardlinked files could be opened with |
Indeed, such API exists with new version of Windows. But it extremely unreliable. It has few exclusions, one of them - You cannot delete mapped files. It makes this API unusable for VictoriaMetrics, since it mostly open files with MMAP. I suppose, mapping propagated to the hard-linked files. |
f41gh7
pushed a commit
that referenced
this issue
Aug 10, 2023
valyala
pushed a commit
that referenced
this issue
Aug 11, 2023
* app/vmbackupmanager/storage: fix path join for windows See: #4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/backup: fixes for windows support - close dir before running os.RemoveAll. Windows FS does not allow to delete directory before all handles will be closed. - add path "normalization" for local FS to use the same format of paths for both *unix and Windows See #4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
valyala
pushed a commit
that referenced
this issue
Aug 11, 2023
valyala
pushed a commit
that referenced
this issue
Aug 11, 2023
* app/vmbackupmanager/storage: fix path join for windows See: #4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> * lib/backup: fixes for windows support - close dir before running os.RemoveAll. Windows FS does not allow to delete directory before all handles will be closed. - add path "normalization" for local FS to use the same format of paths for both *unix and Windows See #4704 Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com> --------- Signed-off-by: Zakhar Bessarab <z.bessarab@victoriametrics.com>
|
FYI, vmbackupmanager should properly make backups to local filesystem on Windows starting from v1.93.0. Closing the issue as fixed then. |
AndrewChubatiuk
pushed a commit
to AndrewChubatiuk/VictoriaMetrics
that referenced
this issue
Oct 3, 2023
…eparators at Part.Path Previously Part.Path could contain `\` directory separators on Windows OS, which could result in incorrect filepaths generation when making backups at object storage. Updates VictoriaMetrics#4704 This is a follow-up for f2df8ad
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Can't create backups using vmbackupmanager.
vmbackup works correctly.
vmbackupmanager produces an error the moment a backup is scheduled to run.
To Reproduce
Executing this command:
vmbackupmanager-windows-amd64-prod.exe -eula -dst="fs://E:\Backups\victoria-metrics-backups" -storageDataPath=%VICTORIAMETRICS%\victoria-metrics-data -snapshot.createURL=http://localhost:8428/snapshot/create -maxBytesPerSecond=10485760 -keepLastDaily=2 -keepLastWeekly=1 -keepLastMonthly=3Version
vmbackupmanager-20230602-121930-tags-v1.91.2-enterprise-0-g9c0ee3336
Logs
Screenshots
No response
Used command-line flags
-eula -dst="fs://E:\Backups\victoria-metrics-backups" -storageDataPath=%VICTORIAMETRICS%\victoria-metrics-data -snapshot.createURL=http://localhost:8428/snapshot/create -maxBytesPerSecond=10485760 -keepLastDaily=2 -keepLastWeekly=1 -keepLastMonthly=3Additional information
Windows Server 2019
as Admin.
The text was updated successfully, but these errors were encountered: