Skip to content

Commit

Permalink
Optimize sqlite index data deletion
Browse files Browse the repository at this point in the history
On sqlite, deleting doesn't require painstakingly recreating the same
exact data that was used for putting, we can just sweep out everything
with the given header num at once. Cuts roughly 1/4 out of erase time.
  • Loading branch information
pmatilai committed Oct 18, 2019
1 parent 77af8e5 commit 5bf88af
Showing 1 changed file with 7 additions and 13 deletions.
20 changes: 7 additions & 13 deletions lib/backend/sqlite.c
Expand Up @@ -563,26 +563,20 @@ static rpmRC sqlite_idxdbPut(dbiIndex dbi, rpmTagVal rpmtag, unsigned int hdrNum
return tag2index(dbi, rpmtag, hdrNum, h, sqlite_idxdbPutOne);
}


static rpmRC sqlite_idxdbDelOne(dbiIndex dbi, dbiCursor dbc, const char *keyp, size_t keylen, dbiIndexItem rec)
static rpmRC sqlite_idxdbDel(dbiIndex dbi, rpmTagVal rpmtag, unsigned int hdrNum, Header h)
{
int rc = dbiCursorPrep(dbc,
"DELETE FROM '%q' WHERE key=? AND hnum=? AND idx=?",
dbi->dbi_file);

dbiCursor dbc = dbiCursorInit(dbi, DBC_WRITE);
int rc = dbiCursorPrep(dbc, "DELETE FROM '%q' WHERE hnum=?", dbi->dbi_file);

if (!rc)
rc = dbiCursorBindIdx(dbc, keyp, keylen, rec);
rc = dbiCursorBindPkg(dbc, hdrNum, NULL, 0);

if (!rc)
while ((rc = sqlite3_step(dbc->stmt)) == SQLITE_ROW) {};

return dbiCursorResult(dbc);
}

static rpmRC sqlite_idxdbDel(dbiIndex dbi, rpmTagVal rpmtag, unsigned int hdrNum, Header h)
{
return tag2index(dbi, rpmtag, hdrNum, h, sqlite_idxdbDelOne);
rc = dbiCursorResult(dbc);
dbiCursorFree(dbi, dbc);
return rc;
}

static const void * sqlite_idxdbKey(dbiIndex dbi, dbiCursor dbc, unsigned int *keylen)
Expand Down

3 comments on commit 5bf88af

@mlschroe
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to create an additional index on hnum, otherwise deleting is really really slow.

@pmatilai
Copy link
Member Author

@pmatilai pmatilai commented on 5bf88af Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, it's a fairly obvious one. Didn't show up in my early testing with smaller sets though, and I've been resisting the temptation to add indexes until proven gui^H^H^H necessary. Also was wondering, but hadn't gotten around to check, whether sqlite does automatic indexing on foreign keys. Apparently not.

@mlschroe
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recommend my little benchmark tool ;-)

Please sign in to comment.