[WIP] single instance leveldb for all qbufs #1601
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks @hmmr! Settings---
minimum_reviewers: 2
merge: true
build_steps:
- make clean
- make deps
- make compile
- make test
- make xref
- make dialyzer
org_mode: true
timeout: 1800 |
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
hmmr
force-pushed
the
feature-az-single_instance_ldb_for_qbufs
branch
from
c7c1a02
to
24c5818
Compare
There seems to be an issue with build step **make_compile,make_test,make_xref,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_compile,make_test,make_xref,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_compile,make_test,make_xref,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
hmmr
force-pushed
the
feature-az-single_instance_ldb_for_qbufs
branch
from
79addc8
to
6b69789
Compare
There seems to be an issue with build step **merge,make_dialyzer** ! ☁️⛔ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
|
Work mostly done; putting it on hold pending leveldb changes from @matthewvon. |
There seems to be an issue with build step **merge,make_test,make_dialyzer** ! ☁️⛔ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
⛔ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **merge,make_compile,make_test,make_xref,make_dialyzer** ! ☁️⛔ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
hmmr
force-pushed
the
feature-az-single_instance_ldb_for_qbufs
branch
from
6b69789
to
59f4731
Compare
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
hmmr
force-pushed
the
feature-az-single_instance_ldb_for_qbufs
branch
from
59f4731
to
7a86559
Compare
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
hmmr
force-pushed
the
feature-az-single_instance_ldb_for_qbufs
branch
from
7a86559
to
784fdb6
Compare
There seems to be an issue with build step **make_test,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
⛔ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
matthewvon
reviewed
Feb 7, 2017
| %% total records stored (when complete) | ||
| total_records = 0 :: non_neg_integer(), | ||
|
|
||
| %% %% iterator cache (need a working iterator support in eleveldb) | ||
| %% TODO: iterator cache |
There was a problem hiding this comment.
Not sure what you think an "iterator cache" will be, but it could be a horrible thing ... leaving you in a position of never finding data. Iterators lock in a "snapshot" of the database. You likely want a fresh snapshot AFTER the query data finishes posting.
There was a problem hiding this comment.
The lifecycle of each qbuf table is such that after writing to it is completed, it becomes a read-only. table (effectively, the last snapshot by your definition?). This is when I intended to open some number of iterators, with the idea to have a map of {Pos, Iter}, so that when I need to fulfill a query with OFFSET N, I could pick the iterator whose associated Pos is closest to N and use it instead of iterating from the beginning.
However, there is no pressing need to develop on this further as query buffers are currently not reusable.
This was referenced Feb 8, 2017
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
added 5 commits
February 24, 2017 04:07
* parens around bools in guards; * no initialization of new vars inside if statement; * prefer `List /= []` over `length(List) > 0`.
# Conflicts: # src/riak_kv_qry_compiler.erl # src/riak_kv_ts_error_msgs.hrl
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
added 2 commits
February 27, 2017 23:19
# Conflicts: # src/riak_kv_qry_compiler.erl
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
added 5 commits
March 2, 2017 03:03
# Conflicts: # src/riak_kv_qry_compiler.erl
…ldb_for_qbufs # Conflicts: # src/riak_kv_qry_buffers.erl # src/riak_kv_qry_buffers_ldb.erl
There seems to be an issue with build step **make_compile,make_test,make_xref,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
|
FYI: I do have the first half of the leveldb/eleveldb code posted and ready. I cannot start the second half until all the reviewers finish with the bucket expiry branches. The code needed to request query id status is still being debated by the Erlangers. The request mechanism is common to both bucket expiry and query buffers. You will NOT directly use this option. But just letting you know I really have done a little of the work: https://github.com/basho/leveldb/wiki/mv-disable-recovery-log I will use this option during the generation of a special ExpiryModule for the query buffers. |
There seems to be an issue with build step **merge,make_compile,make_test,make_xref,make_dialyzer** ! ☁️⛔ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **merge,make_compile,make_test,make_xref,make_dialyzer** ! ☁️⛔ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
⛔ MAKE_COMPILE
⛔ MAKE_TEST
⛔ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
added 2 commits
March 8, 2017 02:20
…r_qbufs # Conflicts: # src/riak_kv_qry_buffers.erl # src/riak_kv_qry_buffers_ldb.erl
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_test,make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
⛔ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
There seems to be an issue with build step **make_dialyzer** ! ☁️✅ MERGE
✅ MAKE_CLEAN
✅ MAKE_DEPS
✅ MAKE_COMPILE
✅ MAKE_TEST
✅ MAKE_XREF
⛔ MAKE_DIALYZER
⬜ 0 of 2 Code reviews from organization basho |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://bashoeng.atlassian.net/browse/RTS-1690
In 1.5.0, each query buffer is kept in a separate leveldb table. This involves many a file and dir creation/deletion ops, with the only benefit that dropping operation is as easy as
eleveldb:close() andalso rmdir().With automatic bucket expiry becoming available in leveldb, we can keep a single leveldb instance always open (never having to call
eleveldb:openourself) and, now storing query results in{Bucket, Key}, offload deletion of per-query buckets to leveldb expiry mechanism.The PR is currently WIP: need to add code to properly enable the expiry.