Changed LoadDataWriter to load data in batches

[AdalbertMemSQL]

This change will allow us to avoid table locking.
insertBatchSize config will be used to set batch size. Previously, we used it when OnDuplicateKey configuration was set.

[pmishchenko-ua]

With the default insertBatchSize of 10,000, can these changes negatively affect performance with default configuration? My guess is that at least 100,000 rows in batch are needed to get close to the best performance.

[AdalbertMemSQL]

With the default insertBatchSize of 10,000, can these changes negatively affect performance with default configuration? My guess is that at least 100,000 rows in batch are needed to get close to the best performance.

I did perf testing with tests from this repo. Batch size showed the same performance. 100k may work bad for insert queries as there should be a limit for a statement length. I bumped into it with fivetran connector and update queries.

[AdalbertMemSQL]

We have a connection pool, so creating a new statement is just sending a small LOAD DATA query.

[pmishchenko-ua]

for insert queries 100k is indeed too much, I'm referring to LOAD DATA queries. You're saying that sending batches of 10k, , 100k, 1M rows shows the same performance?

[AdalbertMemSQL]

10k and 1m (no batches) show the same performance. Here queries are not parallelized (inside of one job), so 100k should also have the same performance.

[AdalbertMemSQL]

Performance testing that we did before parallelized writes and created a separate connection for each query. This caused a situation when ~100k was the optimal batch size.

[AdalbertMemSQL]

I checked several more parameters. default_columnstore_table_lock_threshold by default will lock the whole table if the query touches more than 5k rows https://docs.singlestore.com/cloud/create-a-database/columnstore/locking-in-columnstores/#overriding-default-locking so it makes sense to use even smaller batch size.

[AdalbertMemSQL]

Ignore my previous comment, it refers to rows per partition, so if there are at least 2 partitions - 10k should be ok :)

[pmishchenko-ua]

nit: delete tab