Out Of Memory killer triggered

[Glucy-2]

Describe the bug
Out Of Memory killer triggered when I am trying to transfer a 1.59GB .sqlite file
screenshot of VNC viewer:

Expected behaviour
It works

Actual result
screenshot of VNC viewer:

> ./.local/bin/sqlite3mysql -f ./ledger.sqlite -d ledger -u * -h *.*.*.* --mysql-password ****************
2022-03-22 20:42:10 INFO     Transferring table players
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:00<00:00, 74.73it/s]
2022-03-22 20:42:11 INFO     Adding unique index to column "player_id" in table players
2022-03-22 20:42:11 INFO     Adding unique index to column "player_id" in table players
2022-03-22 20:42:11 INFO     Transferring table ActionIdentifiers
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 47.51it/s]
2022-03-22 20:42:11 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-22 20:42:11 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-22 20:42:11 INFO     Transferring table worlds
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 33.94it/s]
2022-03-22 20:42:11 INFO     Adding unique index to column "identifier" in table worlds
2022-03-22 20:42:11 INFO     Adding unique index to column "identifier" in table worlds
2022-03-22 20:42:12 INFO     Transferring table ObjectIdentifiers
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1314/1314 [00:16<00:00, 80.31it/s]
2022-03-22 20:42:28 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-22 20:42:28 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-22 20:42:28 INFO     Transferring table sources
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [00:00<00:00, 73.59it/s]
2022-03-22 20:42:29 INFO     Adding unique index to column "name" in table sources
2022-03-22 20:42:29 INFO     Adding unique index to column "name" in table sources
2022-03-22 20:42:30 INFO     Transferring table actions
 19%|████████████████████████▍                                                                                                     | 483474/2494012 [1:41:22<7:33:52, 73.83it/s]
[1]    1017098 killed     ./.local/bin/sqlite3mysql -f ./ledger.sqlite -d ledger -u * -h

System Information
debian 11 which has 8 GB of memory

$ sqlite3mysql --version

| software               | version                                                     |
|------------------------|-------------------------------------------------------------|
| sqlite3-to-mysql       | 1.4.12                                                      |
|                        |                                                             |
| Operating System       | Linux 5.10.0-13-amd64                                       |
| Python                 | CPython 3.9.2                                               |
| MySQL                  | mysql  Ver 8.0.26 for Linux on x86_64 (Source distribution) |
| SQLite                 | 3.34.1                                                      |
|                        |                                                             |
| click                  | 8.0.4                                                       |
| mysql-connector-python | 8.0.28                                                      |
| pytimeparse            | 1.1.8                                                       |
| simplejson             | 3.17.6                                                      |
| six                    | 1.16.0                                                      |
| tabulate               | 0.8.9                                                       |
| tqdm                   | 4.63.0                                                      |

[techouse]

Hi and thanks for the report, however, I'm not sure how to address this.

Did you check which process used how much memory?

What other processes were you running?

How much did MySQL (the usual suspect) consume?

Have you tried running it with Python 3.10?

Have you tried running it on a system with more RAM and profile it?

What's in the actions table?

Also, next time you run it, run it with --debug which will dump all the exceptions to the console.

[Glucy-2]

I tried again with more free ram and it said it has done, but the size of the db hasn't changed.
log:

2022-03-23 00:06:55 INFO     Transferring table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO     Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO     Transferring table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO     Transferring table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO     Transferring table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO     Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:14 INFO     Transferring table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO     Adding unique index to column "name" in table sources
2022-03-23 00:07:15 INFO     Transferring table actions
2022-03-23 08:23:29 INFO     Adding index to column "x, y, z, world_id" in table actions
2022-03-23 08:23:30 INFO     Adding index to column "x, y, z, world_id" in table actions
2022-03-23 08:24:54 INFO     Adding foreign key to actions.player_id referencing players.id
2022-03-23 08:26:10 INFO     Adding foreign key to actions.source referencing sources.id
2022-03-23 08:27:20 INFO     Adding foreign key to actions.old_object_id referencing ObjectIdentifiers.id
2022-03-23 08:28:34 INFO     Adding foreign key to actions.object_id referencing ObjectIdentifiers.id
2022-03-23 08:29:51 INFO     Adding foreign key to actions.world_id referencing worlds.id
2022-03-23 08:31:00 INFO     Adding foreign key to actions.action_id referencing ActionIdentifiers.id
2022-03-23 08:32:10 INFO     Done!

[techouse]

Whoa, transferring the actions table for 8h 15min 😱 What do you have in there LOL?

I'm afraid I'll need more info here to give you some more feedback and/or potentially fix this.