Filer.backup does not work as expected

[Stranger10110]

I am trying to test a backup to local disk (both on Windows and Linux) of files that are stored with Filer (MySQL), but it doesn't seem to work properly - it just doesn't copy all the files, only specific files and folders, not even full sizes. Maybe it's just getting stuck for whatever reason?
I also tried to change the directory on Filer in replication.toml, but it does no effect.
It also makes two folders of different dates, but I've configured it to "is_incremental = false".

System Setup
Weed cluster is started with Docker compose (Docker.zip; it contains filer.toml as well).

OS version:

• Windows 10 Pro 20H2 19042.985
• Linux mx 4.19.0-6-amd64 Thank you #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

Weed version:

• version 30GB 2.49 42fb03a windows amd64
• version 30GB 2.49 42fb03a linux amd64

FIler.backup start command: .\weed.exe filer.backup -filerProxy
Replication.toml: replication(.toml).txt

Expected behavior
A full copy of files stored with Filer on a local disk.

Screenshots
Filer

Windows backup

Linux backup
Same picture as on Windows.

Additional context
File.backup output from Windows: win filer.backup log.txt.
File.backup output from Linux: linux filer.backup log.txt.

[kmlebedev]

@Stranger10110 Check the filer.replicate parameter

[source.filer]
  enabled = true
  grpcAddress = "192.168.0.244:18080"
  directory = "/buckets"

[sink.local]
  enabled = true
  directory = "/data"

[Stranger10110]

@kmlebedev My settings are:
[source.filer]
enabled = true
grpcAddress = "localhost:18888" # 192.168.0.244 == localhost
directory = "/"

[sink.local]
enabled = true
directory = "./backup" # local folder to download files to
is_incremental = false

I don't understand, what's wrong with them? I've tried to do a backup on Linux VM with 192.168.0.244 ip address parameter there (IP of a host machine), but it doesn't work either.

Why have you set such port 18080? Filer is on 8888, so it should be 18888, isn't it?

[chrislusf]

Thanks for the details! Could you please add -v=4 to the command, i.e., weed -v=4 filer.backup ..., and show the logs?

[Stranger10110]

Here it is: win filer.backup log 2.txt

I've changed folder structure (by uploading 2 new folders ("base_1" and "base_2") and changing on file) since last launch. There are some changes (it downloaded these 2 new folders and changed file, also made more "date" folders), but it's still does not work as expected. I think it ignores "is_incremental = false" as well.

(I don't see missing files in v=4 logs either)

[chrislusf]

I added some fix for the is_incremental. Please use this to run https://github.com/chrislusf/seaweedfs/releases/dev

[Stranger10110]

Yes, it's not incremental now, but still missing some of the files.

When I've run it first time with a new version files were same as before. But I've deleted them, run again, cancel it in the middle of the process (with CTRL+C) and every time now it's got "stuck" again and miss more of the files.

I think it downloads to the time when I hit CTRL+C and no more, but I'm not 100% sure. Maybe a bug with restarting backup from the last point of progress?

[chrislusf]

I fixed some obvious problems with the local sink, and updated the dev release. Please help to check again. If still problem, please attach filer.backup logs.

[Stranger10110]

Unfortunately, almost no changes for me.

First try : downloaded those 2 "new" folders only (in folder test2; there is no 'Meta_test2' folder anymore too).

win_filer.backup_log_3.txt

Second try: uploaded a new file (verthash.dat), retried to make a backup, result - those 2 folders and new file only.

win_filer.backup_log_4.txt

So it downloads files from some point of time and nothing before that, as it literally says it in the log:
"I0527 03:08:13 16648 filer_backup.go:99] resuming from 2021-05-26 03:16:46.1899912 +0300 MSK"

It needs a way to make it start from the start, I think, like '-startFromBeginning" parameter maybe

[chrislusf]

See "weed filer.backup -h"

[Stranger10110]

Oh, you mean -timeAgo, my bad, I've overlooked it.

I've set it to 9999999h, it now downloads all of the files, besides "base_1/*.*" files and "base_2" folder, and it seems it starts downloading over in the infinite loop (don't know from which point exactly though, but not from all the start I think) after getting an error (look at the last line).

I can't capture all of the logs since there are too many of them and I can't make Windows capture log to a file. These logs on the screenshot I can capture because backup pauses at this moment for a few seconds

[Stranger10110]

Interestingly enough, but it have run successfully on WSL 2 (Ubuntu)

Still failing on Windows after a second try

[chrislusf]

what is the replication.toml file for windows?

[Stranger10110]

Same as before. It was also used for WSL 2
replication.toml.txt

[chrislusf]

it showed some "D:/tests/..." folder in the log. Not sure how that happened.

[Stranger10110]

You mean there? Yes, it's in the error message, there are

• "backup/test2/base_1" (normal backup relative folder path) +
• "D:/tests/Lab_1" (real local folder path, from where those files were uploaded) +
• "Metr_1.odt" (filename)

That's a really strange error, I don't understand it either...

[chrislusf]

can you use weed shell to display the file tree via fs.tree test2?

[Stranger10110]

PS D:\Seaweed> .\weed shell
master: localhost:9333 filer: localhost:8888
> fs.tree test2
test2
├──Office 2007.dat
├──Platform_designer_lab
│ ├──Platform_designer_lab.docx
│ ├──Platform_designer_lab.pdf
│ ├──Platform_designer_lab_2.docx
│ ├──Platform_designer_lab_dirty.docx
│ ├──Скрины
│ │ ├──IMG_20201110_220652.jpg
│ │ ├──opcodes — копия.PNG
│ │ ├──original_connections.PNG
│ │ ├──original_connections_2.PNG
│ │ ├──reg_memory.PNG
│ │ ├──regs.PNG
│ │ ├──rng — копия.PNG
│ │ ├──rng.PNG
│ │ ├──rng2 — копия.PNG
│ │ └──rng2.PNG
│ ├──заметки.txt
│ └──ссылки.txt
├──Platform_designer_lab (1).docx
├──Platform_designer_lab.docx
├──Platform_designer_lab.pdf
├──SOME01.wbfs
├──base_1
│ ├──Metr_1.odt
│ ├──Metr_1.pdf
│ ├──Test
│ │ └──измерения.ods
│ └──измерения.ods
├──base_2
│ ├──blank_praktika_uchebnaya_novyy.pdf
│ ├──blanky.odt
│ ├──blanky.pdf
│ ├──praktika.odt
│ ├──praktika.pdf
│ └──uchebnaya_otchet.pdf
├──server.docx
└──verthash.dat
6 directories, 33 files

[chrislusf]

ok. not too many files. Can you run this weed filer.meta.tail -timeAgo=9999h and share the output?

[Stranger10110]

Here it is: meta.txt

P.S.: I do just some tests right now, so I don't care about these files, if you want me to do something with them to investigate bugs, just let me know

[chrislusf]

The metadata logs showed the source filer had an upload to /test2/base_1 with the file name as D:/tests/Lab_1/Metr_1.pdf at epoch time 1621980155. Usually the file name will be just the name itself. How did that windows path come in? I tried to upload from windows and could not reproduce.

[Stranger10110]

I think that it was a bug in my Python client

But then why did backup on WSL 2 finished successfully? Is it because D:/tests/Lab_1/Metr_1.pdf is a correct path on Linux?

Interestingly enough, but it have run successfully on WSL 2 (Ubuntu)

[chrislusf]

share your python code that caused the problem? I want to see how to prevent the bug in the future.

[Stranger10110]

I think I've already fixed that bug.
But basically I'm uploading files by a PUT method to http://*host*/*api_path*/*filer_path_to_the_file*, where *host* is my own reverse proxy server written in Golang, so I think I have accidentally sent the wrong *filer_path_to_the_file* (with D:/tests/Lab_1/ prefix)

[chrislusf]

Thanks! Added a fix for windows.