batchInsert causes error : too many term in computed SELECT

[lmjgyy]

Environment

Knex version: 2.2.0
Database + version: better-sqlite3
OS: MAC

There are about 1500 hundred batches in and out, but errors are still reported using batchInsert

Bug

1. I am making a task manager for uploading files, and when I batch select a large number of files, I insert the corresponding number of entries into the database

2. Error message： too many term in computed SELECT

3. Code:

I also ran into an issue where I was using aws-sdk to execute upload.on(" httpUploadProgress "), so I was updating the data a lot, which caused a problem. When I execute

async update (table: Knex.QueryInterface, taskProps: Store.UpdateTaskProps) {
await table.where('uuid', this.uuid).update({ ...taskProps })
}

When multiple sql where statements are concatenated, that is, update xxx where uuid = "xxx" and uuid = "xxx" and uuid = "xxx" are concatenated indefinitely until the sql is too long

[mateon1]

I have the same exact problem. This issue was hard to find because the error is actually SQLITE_ERROR: too many terms in compound SELECT.

Inserting more than 500 items at once causes this error.
See this Stack Overflow page https://stackoverflow.com/questions/9527851/sqlite-error-too-many-terms-in-compound-select for context, apparently this can be worked around by using an INSERT INTO ... VALUES (v1),(v2),...(vN) statement instead of a compound select with unions, and this only mattered on ancient versions of sqlite3 before this syntax was supported.

EDIT: Apparently you can also use a series of insert statements, possibly wrapped in a transaction, this supposedly has little to no performance impact compared to a single insert statement, and this solution would allow for proper handling of default values in sqlite (by omitting column names).