shelve: Shelf.clear() has very poor performance

[jtcave]

Bug report

Calling the clear method on a shelve.Shelf object takes a very long time on databases that have thousands of entries.

It can be seen in this script, which creates a database with 10,000 entries and immediately clears it.

import os
import shelve
import tempfile

with tempfile.TemporaryDirectory() as tempdir:
    filename = os.path.join(tempdir, "test-shelf")
    with shelve.open(filename) as db:
        items = {str(x):x for x in range(10000)}
        db.update(items)
        db.clear()
print("ok")

On my M2 Mac Mini, this script takes about 2.1 seconds.

james@iris ~ % time python shelve-clear-test.py
ok
python shelve-clear-test.py  1.18s user 0.92s system 99% cpu 2.100 total

On a Debian VPS:

james@asteroid:~$ time python3 shelve-clear-test.py 
ok

real	0m43.665s
user	0m34.330s
sys	0m9.107s

Your environment

• CPython versions tested on
  • 3.11.2
  • 3.11.4
  • cpython/main branch
• Operating system and architecture
  • macOS/arm64 13.4.1
  • Debian/x86_64 12.1

Linked PRs

• gh-107089: Improve Shelf.clear method performance #107090

[pochmann]

How much of the 2.1 seconds is the clear method?

[jtcave]

How much of the 2.1 seconds is the clear method?

Virtually all of it. The example script running on my branch takes milliseconds.

james@iris cpython-build % time ./python.exe ~/shelve-clear-test.py
ok
./python.exe ~/shelve-clear-test.py  0.05s user 0.02s system 97% cpu 0.069 total