Allow to control number of workers during import

[yohanboniface]

sqlite crashes when using 24 cores, so it may be helpful to limit them a bit.

[davidbgk]

Or change the multithreading options?

[yohanboniface]

Serialized. In serialized mode, SQLite can be safely used by multiple threads with no restriction.

Which seems to be the default mode.

Also, for the record, not sure here it's relevant (I guess they write "thread" like "things"), but we are doing multiprocessing, not multithreading.

[cquest]

I tried to add locks (acquire/release) with no effect :(

[davidbgk]

@cquest do you know exactly how many cores leads to errors? For the moment we only know that 8 works and 24 doesn't. Might be useful to get the exact limit point to investigate.

[cquest]

The limit is not that clear. I got crashes with 4 processes.
The more cores you have the more often you'll have some trouble, same applies with slower disks as it takes more time for sqlite to update its db/journal. In my case, I have 24 cores and the sqlite db/journal is stored on HDD (not SSD).
Limiting the number of process is not safe (unless we only have 1).

[davidbgk]

Do you have an error message from sqlite? A log or something? Hard to reproduce when you're somehow limited in terms of CPUs…

[davidbgk]

Enabling tracebacks might help debugging. Did you play with the isolation level?

[davidbgk]

More options from that Reddit thread using PRAGMA and verify that AUTOINCREMENT is disabled.

[davidbgk]

Many readers but only one writer? C'mon SQLite… We can try to adapt the busy_timeout option but I'm pessimistic about the resolution of that issue.