Overview documentation, documentation guidelines and formating. #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
text formatted it to 80 column width.
Right now it contains a documentation section that developers can use as guidelines to author secondary index documentation
Index Manager that resides in each index node Index Coordinator -- only one in set of index nodes Index Coordinator Replica -- backup nodes for the Index Manager
* Refactored index_manager documentation. * created projector and pubsub documentation.
ns-codereview
pushed a commit
that referenced
this pull request
Jul 11, 2017
Change-Id: Ieaa95c5a0e4a9cf4ab6d9b1f440cde589a114491
ns-codereview
pushed a commit
that referenced
this pull request
Jan 5, 2022
Fixes a problem where metadataClient's cached slice of scheduled plus existing indexes can be stale but it does not notice because the cache- invalidating changes are not synced with cache invalidation itself. This could lead to indexes showing as "building" forever in metadataClient output because it missed a cache-invalidating event and kept returning a stale cached list of scheduled plus existing indexes. The replacement algorithm requires all changes to 1. indexers.allIndexes[] -- all currently existing indexes 2. schedTokenMon.scheduledIndexes[] -- all currently scheduled indexes 3. comboIndexes[] -- cached combination of #1 and #2 to be done inside a new mutex, comboIndexesMut. Additionally, the cache must be invalidated (by setting comboIndexes = nil) inside the same instance of locking comboIndexesMut as any change to allIndexes or scheduledIndexes. These behaviors maintain the cache coherency of comboIndexes by ensuring that whenever it is computed, its content is in sync with both of its inputs (#1 and #2), and whenever one of its inputs is changed, comboIndexes gets set back to nil in the same critical section so cache-invalidating changes can never be missed. Change-Id: I64ab1a743fec76acb2e969089412039337382f33
ns-codereview
pushed a commit
that referenced
this pull request
Jan 12, 2022
1. cpu_throttle.go should use cgroup CPU usage stats when cgroups are supported. This is so the usage is correctly calculated against the number of CPUs allocated to the cgroup rather than the whole node. If not supported fall back to the standard algorithm of CPU ticks in different categories that is collected for the whole node. 2. Indexer set number of CPUs to min(node, cgroup, GSI) available CPUs, where cgroup is sigar num_cpu_prc when cgroups are supported and GSI is config value indexer.settings.max_cpu_percent (exposed on UI as Advanced Index Settings / Indexer Threads). 3. Log message to console if, in #2, GSI > min(node, cgroup) and use the latter instead. 4. planner/proxy.go RetrievePlanFromCluster was calling getIndexSettings() to set CpuQuota based on runtime.NumCPUs() and indexer.settings.max_cpu_percent, but the code in that method was incorrect. Replaced with a call to runtime.GOMAXPROCS(0) which will now get the logical min(node, cgroup, GSI) per #2 above. (Note that CpuQuota is not used in Planner anyway -- its enforcement is commented out in planner/planner.go Validate().) This changes eliminates REST calls to all remote indexer nodes and a use of ClusterInfoCache. 5. Replace direct lookup of indexer.settings.memory_quota with new Config.GetIndexerMemoryQuota() method that takes cgroup setting into account (sigar memory_max) if cgroups are supported. Change-Id: I3c906070d78ba51888a6c07c7dd05959fba9b27b
ns-codereview
pushed a commit
that referenced
this pull request
Jun 24, 2022
* Adding the print in fillScans and not setExplodePositions to capture protoscans received from the query * Index out of range panic was seen in setExplodedPositions function when accessing explodePositions array of size len(IndexInst.Defn.SecExprs) * -- panic: runtime error: index out of range [4] with length 4 * Print the scans details and return error than panic * Example print 2022-06-17T02:37:02.185+05:30 [Error] SCAN##2 ReqID: ee9939aa-d17c-4705-9d60-91d773053bbd Defn: 9151033855720878966 SecExprs: [`age`] Scans: <ud>([{ [150] 3 filterRange [{[{ 100 2}] [100] 3 } {[{50 150 3}] [50] [150] 3 }] <nil>} {[200] [200] 3 range [{[{200 200 3}] [200] [200] 3 }] <nil>}])</ud> MaxCompositeFilters: 1 ProtoScans: protobuf scan 0 filters:<low:"null" high:"100" inclusion:2 > protobuf scan 1 filters:<low:"50" high:"150" inclusion:3 > protobuf scan 2 filters:<low:"200" high:"200" inclusion:3 > Change-Id: Ibb017edb6e84af3d963e80f2549e28b69dc82d57
ns-codereview
pushed a commit
that referenced
this pull request
Jul 5, 2022
…eFilters > Num SecKeys * Adding the print in fillScans and not setExplodePositions to capture protoscans received from the query * Index out of range panic was seen in setExplodedPositions function when accessing explodePositions array of size len(IndexInst.Defn.SecExprs) * -- panic: runtime error: index out of range [4] with length 4 * Print the scans details and return error than panic * Example print 2022-06-17T02:37:02.185+05:30 [Error] SCAN##2 ReqID: ee9939aa-d17c-4705-9d60-91d773053bbd Defn: 9151033855720878966 SecExprs: [`age`] Scans: <ud>([{ [150] 3 filterRange [{[{ 100 2}] [100] 3 } {[{50 150 3}] [50] [150] 3 }] <nil>} {[200] [200] 3 range [{[{200 200 3}] [200] [200] 3 }] <nil>}])</ud> MaxCompositeFilters: 1 ProtoScans: protobuf scan 0 filters:<low:"null" high:"100" inclusion:2 > protobuf scan 1 filters:<low:"50" high:"150" inclusion:3 > protobuf scan 2 filters:<low:"200" high:"200" inclusion:3 > Change-Id: Ibb017edb6e84af3d963e80f2549e28b69dc82d57 (cherry picked from commit 20e09f1)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.