Skip to content

Allow rocksdb to accept read_only option in order to share tables among workers#331

Merged
wbarnha merged 7 commits intofaust-streaming:masterfrom
wbarnha:rocksdb-read-only
Aug 10, 2022
Merged

Allow rocksdb to accept read_only option in order to share tables among workers#331
wbarnha merged 7 commits intofaust-streaming:masterfrom
wbarnha:rocksdb-read-only

Conversation

@wbarnha
Copy link
Member

@wbarnha wbarnha commented Jul 26, 2022

Closes #330, I've had success by running multiple Faust apps with the same data directory for RocksDB

app = faust.App("reuse-this-app-name")
app.GlobalTable('generic-table-name', options={'read_only': True})

I'd like to write some test cases/examples before merging this in order to understand how this works better. If this functionality is not intended, feel free to close this immediately.

@codecov-commenter
Copy link

codecov-commenter commented Jul 26, 2022

Codecov Report

Merging #331 (d4f03c2) into master (9eb770a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #331   +/-   ##
=======================================
  Coverage   94.42%   94.42%           
=======================================
  Files         100      100           
  Lines       10903    10905    +2     
  Branches     1529     1529           
=======================================
+ Hits        10295    10297    +2     
  Misses        533      533           
  Partials       75       75           
Impacted Files Coverage Δ
faust/stores/rocksdb.py 89.45% <100.00%> (+0.06%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9eb770a...d4f03c2. Read the comment docs.

Copy link
Collaborator

@patkivikram patkivikram left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this tested with https://github.com/faust-streaming/python-rocksdb? Thats the official rocksdb binding we are using for this project

@wbarnha
Copy link
Member Author

wbarnha commented Jul 27, 2022

I've been using https://github.com/NightTsarina/python-rocksdb with Faust. I made a PR for that update in faust-streaming/python-rocksdb#1 so that we can synchronize progress on all forks. Feel free to merge that whenever you feel comfortable.

I just tested this PR with the current release of https://github.com/faust-streaming/python-rocksdb and I can confirm everything works properly. Before using the changes in this PR, I got errors such as IO error: lock [DB path]/LOCK: Resource temporarily unavailable from RocksDB.

It'd be nice if I could make the read_only option more accessible.

@wbarnha wbarnha requested a review from patkivikram August 3, 2022 16:50
@wbarnha wbarnha merged commit 84e444d into faust-streaming:master Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-process reads for RocksDB among multiple Faust apps

3 participants