ability to set "packet size" in myloader

[ttauzo]

Since insert statements can be easily cut into smaller ones or grouped into bigger ones it would be great to provide such functionality in myloader. It would help loading into dbs with different max_allowed_packet setting without the need to dump the same data multiple times (with different settings).

[davidducos]

With --rows you can reduce the insert size.
And with --defaults-extra-file you can set the file content:

[myloader_session_variables]
max_allowed_packet = 1000000000

Reference: https://github.com/mydumper/mydumper#defaults-file

[ttauzo]

But i understand that rows you can limit how many "values" parts will be in insert statement. Depending on row length it can vary in terms of packet size. And myloader_session_variables just sets what maximum packet db will accept.
I was thinking about "on the fly" spliting / grouping inserts based on size (not rows count) directly in myloader to "to simulate" --statement-size (that you can set in mydumper) but while loading with myloader

[davidducos]

and WHY do you need this? how is this going to improve performance?

[ttauzo]

It wont influence performance but will make dumps more flexible. eg. I can export from my shared hosting dumps with big max_allowed_packet to improve performance in case i need migrate the data but i cannot use this dump to recover these data to original db (after failure) as i cannot change max_allowed_packet to bigger one (used to export). It is an edge situation but still sth i have to deal with.

[davidducos]

why do you say that you cannot change the max_allowed_packet as it is a session variable?

[ttauzo]

It is a shared hosting. For session i get info that it is read-only. For GLOBAL i get info that i need super-user pon., 18 wrz 2023, 21:17 użytkownik David Ducos ***@***.***> napisał:

…

why do you say that you cannot change the max_allowed_packet as it is a session variable? — Reply to this email directly, view it on GitHub <#1303 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFKO7AC2GM7IM54XMUOVDOLX3CM3XANCNFSM6AAAAAA44652QQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ttauzo]

Session variable is read only " max_allowed_packet on the command line or in an option file. The session value of this variable is read only " https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html

[davidducos]

You can use on mydumper:

  -s, --statement-size             Attempted size of INSERT statement in bytes, default 1000000