myloader stuck while restore

[aloks2019]

I see that myloader took 4 hrs before starting first database creation while restore.
I started myloader and got following line in log:
**Message: 02:25:24.566: Server version reported as 5.7.12-log
<after this line myloader took 4hrs before starting first database created>

What it does at this stage?
How to tune this stage.
I had 20 millions files in backup directory.
Database size : 180gb
Total Hours: 9
Waited after 1st line :4 hrs
Actual restoration: 5 hrs

[davidducos]

Hi @aloks2019, thank you for reporting this bug! I found where is the problem and I will be fixing it soon as this is major issue.
The problem is related to the use of a new lock, as we are storing the files in a list and that was being done sequentially as, which means, that if you have millions of files it will take a lot of time to complete. I will be adding a new queue to process that in parallel and performing the sort in O(n). I will keep you posted.

[aloks2019]

Thanks @davidducos for working on this.when are you planning to push it to master as we have another large restore next week?

[davidducos]

hi @aloks2019,
I'm already working on branch https://github.com/mydumper/mydumper/tree/fix_719 if you want to test, it will be really useful for me.

[davidducos]

BTW, @aloks2019 if you use a streamed backup, you will avoid this kind of issues

[aloks2019]

Can you explain more about this? Thanks Alok

On Thu, Jun 16, 2022 at 10:32 AM David Ducos ***@***.***> wrote: BTW, @aloks2019 <https://github.com/aloks2019> if you use a streamed backup, you will avoid this kind of issues — Reply to this email directly, view it on GitHub <#719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALITW47UAHRHSCCNOX33QQDVPM3HRANCNFSM5YMEY5HQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thanks, Alok Srivastava

[davidducos]

You are executing mydumper that saves all the files in the directory that you are using as parameter. But if you use mydumper --stream > backup.sql , you will get a single file. Then you can do cat backup.sql | myloader --stream to load the backup.
When you are using --stream on myloader, the files are going to be executed as soon as they are allowed to do it.
More info in: https://www.percona.com/blog/mydumper-stream-implementation/

[aloks2019]

You are working on fix_719 branch. How can I install mydumper with that fix on rhel7? Thanks Alok

On Thu, Jun 16, 2022 at 11:53 AM David Ducos ***@***.***> wrote: You are executing mydumper that saves all the files in the directory that you are using as parameter. But if you use mydumper --stream > backup.sql , you will get a single file. Then you can do cat backup.sql | myloader --stream to load the backup. When you are using --stream on myloader, the files are going to be executed as soon as they are allowed to do it. More info in: https://www.percona.com/blog/mydumper-stream-implementation/ — Reply to this email directly, view it on GitHub <#719 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALITW4ZZZ2YXJQAZD6QVP2LVPNEYZANCNFSM5YMEY5HQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Thanks, Alok Srivastava

[davidducos]

You will need to compile it yourself:

git clone https://github.com/mydumper/mydumper.git
cd mydumper
git checkout -t origin/fix_719
cmake .
make