RocksDB deleted, but locks still exist

[EliFinkelshteyn]

Currently, if I have a connection open to a RocksDB, even if I delete the entire RocksDB on disk, I still can't create a new RocksDB in the same location due to the No locks available error. Is this intended behavior, and if so, is there any way to force close all connections to a RocksDB or any other way to force the release of a lock from some connection that wasn't picked up during garbage collection?

[adamretter]

Are you using the Java API?

[EliFinkelshteyn]

Python API , but as far as I can tell, this part of the code is coming from Rocks DB itself.

[adamretter]

@EliFinkelshteyn Is this still an issue for you? Which platform are you on please?

[EliFinkelshteyn]

@adamretter yes. Was running into the issue on both OS X and Ubuntu. To get around it, I had to create and have to call a hacky .close() function:

    def close(self):
        if hasattr(self, '_db'):
            del self._db
            gc.collect()

I'm on v5.5.1. Happy to try a newer version if there was an expected fix for this.

[siying]

I'm confused by your link of "this part of the code is coming from Rocks DB itself.". I don't see how this code is from RocksDB itself. Note "python-rocksdb" is a third-party port completely on top of RocksDB. If you want to ask question related it, a better place to open an issue is there. We don't have the expertise here to answer questions related to it.

[EliFinkelshteyn]

Hi @siying,
Thanks for your response. I'm well aware that python-rocksdb is a third party port. I'm not asking questions related to it. I used it merely to demonstrate the issue. I linked you to the portion of the python-rocksdb code that calls out to RocksDB and, in the case of this issue, receives back an error with the statement No locks available.

This No locks available error is coming directly from RocksDB and just being relayed by python-rocksdb. It's been about a year since I created the issue and it seems like this language is no longer used in the current version of RocksDB, but you can still see references to it in older issues like here (note: this is on RocksDB, not python-rocksdb).

[siying]

@EliFinkelshteyn I still don't understand this. If calling

            del self._db
            gc.collect()

Solves the problem, it means the C++ object of RocksDB is not yet properly closed the moment you created a new DB. Then it's not a problem of RocksDB C++ library. C++ doesn't have any GC mechanism. If gc.collect() solves the problem, it means the finalizer called the right C++ function it is supposed to call earlier. Did I miss anything?

[gmossessian]

Hi @siying , thanks for your responses, I work with @EliFinkelshteyn so please excuse my butting in.

It seems that you are correct, that intended Cython behavior is to trigger the C++ destructor upon garbage collection, and there is no access to the delete C++ keyword from within Cython. However, I noticed there is a C API for Rocksdb in c.h, which includes extern ROCKSDB_LIBRARY_API void rocksdb_close(rocksdb_t* db);, would it be appropriate to merely use this method from Cython to close it?

At this point it's clearly not a rocksdb issue and I am just asking for your advice :) Thank you!

[siying]

@gmossessian yes, rocksdb_close() destroy the C++ DB object, so calling it should be sufficient.

[siying]

@gmossessian to clarify, if you create DB using C API, then use C API rocksdb_close() to destroy it. If you create it through C++ API, then you have to call C++ delete to destroy it. Are you able to do it in CPython?

[gmossessian]

@siying thank you for the clarification, that's very useful. It should not be very hard to write a Cython wrapper using only the C API to give CPython the ability to do this properly. I'll try to do it relatively soon, perhaps as a fork of python-rocksdb. Thanks again for your help!

[siying]

@gmossessian I guess even without accessing to "delete", you may be able to do db->~DB(); free db;?

[gmossessian]

wouldn't this would still imply access to C++ from CPython? I was hoping to avoid modifying RocksDB itself, but one way to address this issue that would make everything else easier is to add a Close() method to the C++ library, which does not seem to exist as far as I can find it.