Add /_partition/:partition/_info endpoint #1615
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This props list is recorded in each database shard as well as the shard document in the special _dbs database. Co-authored-by: Garren Smith <garren.smith@gmail.com> Co-authored-by: Robert Newson <rnewson@apache.org>
Co-authored-by: Garren Smith <garren.smith@gmail.com> Co-authored-by: Robert Newson <rnewson@apache.org>
Default to database's partitioned setting if not present in ddoc.
Co-authored-by: Robert Newson <rnewson@apache.org> Co-authored-by: Paul J. Davis <paul.joseph.davis@gmail.com>
Co-authored-by: Garren Smith <garren.smith@gmail.com> Co-authored-by: Robert Newson <rnewson@apache.org>
Adds tests to validate the all_docs optimisations works for partitions
* Block design documents with partitioned option in non-partitioned db * Prohibit javascript reduces in partitioned:true ddocs * Prohibit include_docs=true for _view in partitioned db
'skip' is implemented efficiently at the worker level but we've disabled it for clustered views because of the multiple shards (and not being able to calculate the right skip value to pass to each worker). With a partitioned query, this problem is gone, as the value the query specifies will be the right value for all workers (as they hit the same shard range). This commit removes the old fix_skip_and_limit function from fabric_rpc and moves the logic up to the coordinators.
rnewson
requested changes
Sep 20, 2018
Member
rnewson
left a comment
rnewson
left a comment
There was a problem hiding this comment.
The broad strokes are right but I'd like you to look at how db_info is done and follow it a bit more closely.
src/fabric/src/fabric.erl
Outdated
|
|
||
| %% @doc returns the size of a given partition | ||
| -spec get_partition_info(dbname(), Partition::binary()) -> | ||
| {ok, {partition_info, non_neg_integer()}}. |
Member
There was a problem hiding this comment.
that spec doesn't look quite right.
| case fabric_dict:size(Counters1) =:= 0 of | ||
| true -> | ||
| [FirstInfo | RestInfos] = Acc2, | ||
| PartitionInfo = get_max_partition_size(FirstInfo, RestInfos), |
Member
There was a problem hiding this comment.
take a look at fabric_db_info.erl for how it merges results.
| [FirstInfo | RestInfos] = Acc2, | ||
| PartitionInfo = get_max_partition_size(FirstInfo, RestInfos), | ||
| {stop, | ||
| [{db_name, Name} | format_partition(PartitionInfo)] |
Member
There was a problem hiding this comment.
peculiar indentation (here and elsewhere).
rnewson
reviewed
Sep 25, 2018
src/couch/src/couch_bt_engine.erl
Outdated
| {doc_del_count, DocDelCount}, | ||
| {active, SizeInfo#size_info.active}, | ||
| {external, SizeInfo#size_info.external} | ||
| {size, |
rnewson
reviewed
Sep 25, 2018
src/chttpd/src/chttpd_db.erl
Outdated
| send_json(Req, 202, {[{ok, true}]}). | ||
|
|
||
|
|
||
| handle_partition_req(#httpd{method='GET',path_parts=[DbName, <<"_partition">>, Partition, <<"_info">>]}=Req, _Db) -> |
Member
There was a problem hiding this comment.
let's put this at /dbname/_partition/partition as a parallel to getting db info at /dbname
rnewson
reviewed
Sep 25, 2018
src/couch/src/couch_db.erl
Outdated
| Sizes = couch_db_engine:get_partition_info(Db, Partition), | ||
| {ok, Sizes}; | ||
| get_partition_info(_Db, _Partition) -> | ||
| throw({badrequest, <<"`partition` is not valid">>}). |
rnewson
reviewed
Sep 25, 2018
src/couch/src/couch_db_engine.erl
Outdated
| -type epochs() :: [{Node::atom(), UpdateSeq::non_neg_integer()}]. | ||
| -type size_info() :: [{Name::atom(), Size::non_neg_integer()}]. | ||
| -type partition_info() :: [ | ||
| {Partition::atom(), Partition::binary()} | |
Member
There was a problem hiding this comment.
odd. why aren't we saying {partition, Partition::binary()} here? using the same name twice is confusing and probably doesn't even work?
rnewson
force-pushed
the
feature/user-partitioned-databases
branch
2 times, most recently
from
8b7f146 to
fcb1795
Compare
rnewson
reviewed
Oct 2, 2018
| {DocCount::atom(), DocCount::non_neg_integer()} | | ||
| {DocDelCount::atom(), DocDelCount::non_neg_integer()} | | ||
| {Size::atom(), size_info()} | ||
| {partition, Partition::binary()} | |
Member
Author
There was a problem hiding this comment.
we don't return the db name at this level. We do that at the fabric level.
rnewson
approved these changes
Oct 2, 2018
Member
rnewson
left a comment
rnewson
left a comment
There was a problem hiding this comment.
needs some squashing but otherwise +1
Add an endpoint that returns the partition size and doc count
garrensmith
force-pushed
the
add-partition-info-endpoint
branch
from
fac4775 to
2f3aa8b
Compare
Member
Author
|
This is merged into the partitions branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Add an endpoint that returns the partition size and doc count
Testing recommendations
Create documents in different partitions. Compare the partition size versus the
GET /dbsize information.Checklist