shelve del only removes the reference not the actual data

[todeskurve]

Documentation

Bug description:

cpython/Lib/shelve.py

Line 21 in aa3f11f

del d[key] # delete data stored at key (raises KeyError

this doesn't work on 3.12 under Windows 11

it looks like two files are used to store the shelf, .dir contains keys and offsets where to find the data in the .dat-file
del shelf[key] only removes the reference in the .dir-file, but the data stays untoched (i.e. still stored) in the .dat-file

does this work on other systems and the documentation just doesn't fit my windows

[rruuaanng]

I didn’t see any issues when running it on the main branch. My os is win11

import shelve

d = shelve.open('test.dbm')

key = 'c'

# 1
d[key] = '123'
data = d[key]
print(data)

# 2
keys_list = list(d.keys())
print(keys_list)

# 3
del d[key]

# 4
keys_list = list(d.keys())
print(keys_list)

# 5
data = d[key]
print(data)

d.close()

output:

123
['c']
[]
Traceback (most recent call last):
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
            ~~~~~~~~~~^^^^^
KeyError: 'c'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\code\cc\cpython\alpha\cpython-main\test.py", line 24, in <module>
    data = d[key]
           ~^^^^^
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\dbm\sqlite3.py", line 92, in __getitem__
    raise KeyError(key)
KeyError: b'c'

[rruuaanng]

I didn’t see any issues when running it on the main branch. My os is win11

import shelve

d = shelve.open('test.dbm')

key = 'c'

# 1
d[key] = '123'
data = d[key]
print(data)

# 2
keys_list = list(d.keys())
print(keys_list)

# 3
del d[key]

# 4
keys_list = list(d.keys())
print(keys_list)

# 5
data = d[key]
print(data)

d.close()

output:

123
['c']
[]
Traceback (most recent call last):
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\shelve.py", line 111, in __getitem__
    value = self.cache[key]
            ~~~~~~~~~~^^^^^
KeyError: 'c'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\code\cc\cpython\alpha\cpython-main\test.py", line 24, in <module>
    data = d[key]
           ~^^^^^
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\shelve.py", line 113, in __getitem__
    f = BytesIO(self.dict[key.encode(self.keyencoding)])
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\code\cc\cpython\alpha\cpython-main\Lib\dbm\sqlite3.py", line 92, in __getitem__
    raise KeyError(key)
KeyError: b'c'

The above code yields consistent results in the 3.12.6 python

[todeskurve]

sorry, i have not made myself clear enough.
i am not talking about data in python objects
i am talking about data in the stored files

i have found other cases, which doesnt even involve del but also look like a problem.
overwriting data (i assume the old data gets kind of removed) works only if you stay within the selected 'cluster size' (512 bytes here) from the intial adding the data to the shelf

i created some test code to be able to share the issue better:

import os

# remove old case files
if os.path.isfile("test_shelf.dir"): os.remove("test_shelf.dir")
if os.path.isfile("test_shelf.dat"): os.remove("test_shelf.dat")
if os.path.isfile("test_shelf.bak"): os.remove("test_shelf.bak")

import shelve
test_shelf = shelve.open('test_shelf')
test_shelf["key1"] = "this_sensitive_data_1"
test_shelf["key2"] = "this_sensitive_data_2"
test_shelf["key3"] = "this_sensitive_data_3"

# select the case here 1-5
case=1

if case==1:
    del test_shelf["key2"]
    # the file test_shelf.dat still contains "this_sensitive_data_2" at this point
    # excerpt from test_shelf.dat: €�•�       Œ�this_sensitive_data_2”.
elif case==2:
    # overwriting with empty string
    test_shelf["key2"] = ""
    del test_shelf["key2"]
    # the file test_shelf.dat still contains "is_sensitive_data_2" at this point
    # excerpt from test_shelf.dat: €�•�       Œ ”.is_sensitive_data_2”.
elif case == 3:
    # overwriting with spaces the size of the data in the shelf at that key
    old_data=test_shelf["key2"]
    test_shelf["key2"] = " "*len(old_data)
    del test_shelf["key2"]
    # the file test_shelf.dat at least doesnt contain the sensitive data anymore, but it has not shrunken in size ;-)
    # excerpt from test_shelf.dat: €�•�       Œ�                     ”.
elif case == 4:
    # overwriting with not enough data
    test_shelf["key2"] = ""
    # the file test_shelf.dat still contains "is_sensitive_data_2" at this point
    # excerpt from test_shelf.dat: €�•�       Œ ”.is_sensitive_data_2”.
elif case == 5:
    # overwriting with too much data also leaves the sensitive data in the test_shelf.dat file on the original place and just forgets having stored that in the first place
    test_shelf["key2"] = "this_sensitive_data_4"*100
    # excerpt from test_shelf.dat: €�•�       Œ�this_sensitive_data_2”.

the 'escalation' to stdlib-bug worries me a little bit, since fixing my 20mb test-data took a few seconds and i dont want to see this extra time added after a simple del every time.

my solution to really clean the shelf-files once was this:

shelf_with_orphaned_data=shelve.open('shelf_with_orphaned_data')
shelf_clean=shelve.open('shelf_clean')
for key in shelf_with_orphaned_data.keys():
    shelf_clean[key]=shelf_with_orphaned_data[key]

but this took way longer than anticipated! :-D

perhaps a combination of "overwriting the sensitive data in the .dat-file with 0000s" in those two cases (del and "assign new data to the key") and adjusting the documentation to reflect the missing reduction in filesize would be a nice approach?

perhaps this isn't even a shelve problem but the underlying storage. that's why i chose docs to start with.

if you like i can create a pull request of my quick fix: todeskurve@da3679f

[todeskurve]

this problem is similar with debian 12.7 and python 3.11

interestingly the keys are still in the .db-file even after del()

compared to windows there are a few less chars of sensitive data still in there, but the test_shelf.db suffers from security problems as well.

on a sidenote, the order of the keys is different between windows and debian (for example in case 4 of the test-script above)

[rruuaanng]

this problem is similar with debian 12.7 and python 3.11

interestingly the keys are still in the .db-file even after del()

compared to windows there are a few less chars of sensitive data still in there, but the test_shelf.db suffers from security problems as well.

on a sidenote, the order of the keys is different between windows and debian (for example in case 4 of the test-script above)

That is to say, Modification is not updated to the file.

[rhettinger]

Looking at Lib/dbm/dumb.py, the __del__ method only removes the index reference, not the actual data. The comments note that that space is never reclaimed. This matches what the OP observed.

Since dumb.py specificies the minimum requirements for other dbm implementations, they are also not guaranteed to erase the potentially sensitive information. (This is much like deleting a file on disk which may only remove the inode but not touch the actual data.)

I don't think there is a bug here, either in the shelve implementation or documentation. Instead, there is an unsupported desire for true data erasure which is something beyond Python's scope and which might not actually be possible on some devices such as SSDs.

The OP's efforts to mitigate the issue look like a solid effort. Zeroing out the data prior to deletion (case 3) looks to be effective for that environment.

Given the current API which makes very few guarantees, I think the only way to reclaim the space is to periodically build a new shelve and copy over only the active entries:

import shelve, os

# Build a DB with orphan entries
d = shelve.open('tmp1')
d['one'] = 1
d['two'] = 2
d['three'] = 3
del d['two']
d.close()

# Compaction
d = shelve.open('tmp1')
t = shelve.open('tmp2')
t.update(d)
d.close()
t.close()
os.replace('tmp2.db', 'tmp1.db')

[erlend-aasland]

I don't think there is a bug here, either in the shelve implementation or documentation. Instead, there is an unsupported desire for true data erasure which is something beyond Python's scope and which might not actually be possible on some devices such as SSDs.

Closing as not-a-bug.