Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(sqlite3.OperationalError) too many SQL variables exception #6402

Closed
GeigerJ2 opened this issue May 17, 2024 · 1 comment · Fixed by #6404
Closed

(sqlite3.OperationalError) too many SQL variables exception #6402

GeigerJ2 opened this issue May 17, 2024 · 1 comment · Fixed by #6404

Comments

@GeigerJ2
Copy link
Contributor

When doing an import dry-run of the MC3D provenance archive to an sqlite_dos profile via:
verdi archive import MC3D-provenance.aiida -n
I just got the following exception:

╰─ verdi archive import MC3D-provenance.aiida -n
Report: starting import: MC3D-provenance.aiida
Report: Parameters
-------------------------------  ---------------------
Archive                          MC3D-provenance.aiida
New Node Extras                  keep
Merge Node Extras (in database)  (k)eep
Merge Node Extras (in archive)   do (n)ot create
Merge Node Extras (in both)      (l)eave existing
Merge Comments                   leave
Computer Authinfos               exclude

Report: Test run: nothing will be added to the profile
Report: Adding 1 new user(s)
Report: Adding 3 new computer(s)
Report: Re-labelled 2 new Computer(s)
Report: Collecting Node(s) ...
Critical: an exception occurred while trying to import the archive MC3D-provenance.aiida: OperationalError: (sqlite3.OperationalError) too many SQL variables
[SQL: SELECT db_dbnode_1.uuid, db_dbnode_1.id
FROM db_dbnode AS db_dbnode_1
WHERE CAST(db_dbnode_1.node_type AS VARCHAR) LIKE ? ESCAPE '\' AND db_dbnode_1.uuid IN (?, ?, ?, ... 
...<question marks repeat>...
?, ?)
(Background on this error at: https://sqlalche.me/e/20/e3q8)

The command ran fine through for an psql_dos profile.

(Not sure if #5740 is somehow related)

@sphuber
Copy link
Contributor

sphuber commented May 18, 2024

The source of the error is exactly what #5740 addressed. While importing, a query is made to check what nodes to be imported already exist in the target storage, and this leads to many parameters, which exceeds the statically defined limit. What I don't fully understand yet is why this was not caught by the mentioned PR. Will have to find the exact line that throws this exception to see why that is not getting batched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants