New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
upsert_all() throws issue when upserting to empty table #73
Comments
I think this is because you forgot to include a |
Released as 2.0.1 https://github.com/simonw/sqlite-utils/releases/tag/2.0.1 |
I think I actually had several issues in play... The missing key was one, but I think there is also an issue as per below. For example, in the following: def init_testdb(dbname='test.db'):
if os.path.exists(dbname):
os.remove(dbname)
conn = sqlite3.connect(dbname)
db = Database(conn)
return conn, db
conn, db = init_testdb()
c = conn.cursor()
c.executescript('CREATE TABLE "test1" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')
c.executescript('CREATE TABLE "test2" ("Col1" TEXT, "Col2" TEXT, PRIMARY KEY ("Col1"));')
print('Test 1...')
for i in range(3):
db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
print('Test 2...')
for i in range(3):
db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
{'Col1':'c','Col2':'x'}], pk=('Col1'))
print('Done...')
---------------------------------------------------------------------------
Test 1...
Test 2...
IndexError: list index out of range
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-763-444132ca189f> in <module>
22 print('Test 2...')
23 for i in range(3):
---> 24 db['test1'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'}], pk=('Col1'))
25 db['test2'].upsert_all([{'Col1':'a', 'Col2':'x'},{'Col1':'b', 'Col2':'x'},
26 {'Col1':'c','Col2':'x'}], pk=('Col1'))
/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in upsert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, extracts)
1157 alter=alter,
1158 extracts=extracts,
-> 1159 upsert=True,
1160 )
1161
/usr/local/lib/python3.7/site-packages/sqlite_utils/db.py in insert_all(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, extracts, upsert)
1097 # self.last_rowid will be 0 if a "INSERT OR IGNORE" happened
1098 if (hash_id or pk) and self.last_rowid:
-> 1099 row = list(self.rows_where("rowid = ?", [self.last_rowid]))[0]
1100 if hash_id:
1101 self.last_pk = row[hash_id]
IndexError: list index out of range the first test works but the second fails. Is the length of the list of items being upserted leaking somewhere? |
Hmmm... just tried with installs from pip and the repo (v2.0.0 and v2.0.1) and I get the error each time (start of second run through the second loop). Could it be sqlite3? I'm on 3.30.1. UPDATE: just tried it on jupyter.org/try and I get the error there, too. |
So the conundrum continues.. The simple test case above now runs, but if I upsert a large number of new records (successfully) and then try to upsert a fewer number of new records to a different table, I get the same error. If I run the same upserts again (which in the first case means there are no new records to add, because they were already added), the second upsert works correctly. It feels as if the number of items added via an upsert >> the number of items I try to add in an upsert immediately after, I get the error. |
If I try to add a list of
dict
s to an empty table usingupsert_all
, I get an error:A hacky workaround in use is:
The text was updated successfully, but these errors were encountered: