doc(server): sync backend-hstore with master - #486
Conversation
Add config/config-backend-hstore.md in English and Chinese. The hstore distributed backend had no configuration page: how to select it, the two hstore.* options with their defaults, the rest-server and graph keys that only take effect with PD, how the Server routes requests to Stores through PD, and the query features the backend does not support were all undocumented.
6c16fe8 to
987fa8c
Compare
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: no. Summary: The page is carefully sourced and most of it holds up against apache/hugegraph master at 36811483a, but two statements are wrong (the error string quoted for a rocksdb-only build, and nodes: 0 read as proof that PD was reached) and two are misleading as framed (section 7 is presented as a diff against the local backends when 18 of its 21 rows match RocksDBFeatures, and hstore.partition_count is described as applied once when it is re-sent on every Server start). The two hand-maintained section index pages were also not updated. Evidence: both files read at head 987fa8c; claims checked against apache/hugegraph master 36811483a in HstoreFeatures, RocksDBFeatures, BackendFeatures, HstoreProvider, HstoreOptions, HstoreSessionsImpl, HstoreMetrics, BackendProviderFactory, BackendStoreProvider, BackendStoreInfo, StandardHugeGraph, SchemaTransactionV2, RegisterUtil, GraphManager, InitStore, CoreOptions, ServerOptions, PartitionService, PDConfig, StoreNodeService, HgStoreNodePartitionerImpl, hugegraph-dist/pom.xml, backend.properties, hstore.properties.template, Dockerfile-hstore and both application.yml files; and against apache/hugegraph-doc master 493ef79 for themes/docsy, config.toml and content/{en,cn}/docs/config/.
| `hstore` does not support every query form the local backends do. The differences visible to a user: | ||
|
|
||
| | Feature | Supported | | ||
| |----------------------------|-----------| | ||
| | Scan by key prefix | yes | | ||
| | Scan by key range | yes | | ||
| | Query with range condition | yes | | ||
| | Query with order by | yes | | ||
| | Query by page | yes | | ||
| | OLAP properties | yes | | ||
| | Task and server vertex | yes | | ||
| | Scan token | no | | ||
| | Query schema by name | no | | ||
| | Query by label | no | | ||
| | Query with `in` condition | no | | ||
| | Query with `contains` | no | | ||
| | Query with `contains key` | no | | ||
| | Sort results by input ids | no | | ||
| | Delete edge by label | no | | ||
| | Update vertex property | no | | ||
| | Update edge property | no | | ||
| | Transaction | no | | ||
| | Number type | no | | ||
| | Aggregate property | no | | ||
| | TTL | no | | ||
|
|
||
| Sorting by input ids is off because multi node batch scans group the input keys by store and lose the global | ||
| order. Vertex and edge property updates are off because the properties are stored in a single cell. |
There was a problem hiding this comment.
RocksDBFeatures on apache/hugegraph master (36811483a). Only three entries actually differ:
supportsQuerySortByInputIds: rocksdbtrue, hstorefalsesupportsTransaction: rocksdbtrue, hstorefalsesupportsTaskAndServerVertex: rocksdbfalsethrough theBackendFeaturesdefault, hstoretrue
Every other no row is false on rocksdb too. RocksDBFeatures.supportsUpdateVertexProperty() even carries the same // Vertex properties are stored in a cell(column value) comment that the closing paragraph attributes to hstore.
The table also omits supportsMergeVertexProperty() (false), listing 21 of the 22 overrides in HstoreFeatures.
Please retitle the section to describe this backend's own capabilities, or add a rocksdb column marking the three real differences. Same at content/cn/docs/config/config-backend-hstore.md:190-219.
| A locally built distribution has the `hstore` provider compiled in by default. The `rocksdb-only` Maven | ||
| profile narrows the compiled backend list to `rocksdb`, and a distribution built that way rejects | ||
| `backend=hstore` with `Unsupported backend type`. |
There was a problem hiding this comment.
rocksdb-only build produces.
Unsupported backend type '%s' comes from RegisterUtil.registerBackend(), whose switch only sees the entries of DistOptions.BACKENDS. Under -Drocksdb-only that list is narrowed at hugegraph-dist/pom.xml:302 (<hugegraph.backends>rocksdb</hugegraph.backends>) and reaches the runtime through hugegraph-dist/src/main/resources/backend.properties (backends=[${hugegraph.backends}]), so hstore is never passed to that switch and the message never fires.
BackendProviderFactory.ALLOWED_BACKENDS is a static List.of("memory", "rocksdb", "hbase", "hstore"), so the illegal-backend guard passes and execution reaches:
BackendException.check(clazz != null, "Not exists BackendStoreProvider: %s", backend);Please quote Not exists BackendStoreProvider: hstore instead, so the string a reader greps for in the server log is the one actually written. Same at content/cn/docs/config/config-backend-hstore.md:115-116.
| The `nodes` value in the response is the count of active stores PD returns. A `nodes` value of `0` means the | ||
| Server reached PD but PD has no store in state `Up`, which usually means the Store nodes have not registered | ||
| yet, or registered as `Pending` because they are not in PD's `pd.initial-store-list`. |
There was a problem hiding this comment.
nodes value of 0 does not establish that the Server reached PD.
HstoreMetrics.metrics() reports session.getActiveStoreSize(), and that method in HstoreSessionsImpl swallows the failure:
public int getActiveStoreSize() {
try {
return defaultPdClient.getActiveStores().size();
} catch (PDException ignore) {
}
return 0;
}Any PDException produces the same 0: PD down, PD unreachable from the Server, wrong pd.peers, or credentials rejected. As written the page tells a reader to rule out the most likely cause first, and there is no log line at this call site to correct them.
Please add the PD-unreachable case. Same at content/cn/docs/config/config-backend-hstore.md:229-231.
| The number is applied when the store is first registered with PD, so changing it later in the properties file | ||
| does not repartition an existing graph. |
There was a problem hiding this comment.
infoInitializedGraph, the guard around setGraph(...) in HstoreSessionsImpl.open(), is a private static final Set<String> on the class. It is empty again in a fresh JVM, so each start sends whatever hstore.partition_count currently holds. PartitionService.updateGraph() accepts any value in 1..totalCount and rewrites the graph's partitionCount, logging partition count changed from {} to {}.
That matters because newPartition() derives boundaries from the current value:
int partitionSize = PartitionUtils.MAX_VALUE / graph.getPartitionCount();Partitions created afterwards use the new count while existing ones keep their old boundaries. Saying it does not repartition an existing graph is true, but it reads as a reassurance that the key is safe to edit.
Please say the value is re-sent at each start and must not change once the graph holds data. Same at content/cn/docs/config/config-backend-hstore.md:140.
|
|
||
| - `0`, the default, means let PD decide. For a graph data store PD uses its own cluster wide partition total, | ||
| which it derives from the number of entries in `pd.initial-store-list`, `partition.store-max-shard-count` | ||
| and `partition.default-shard-count`. For the `/m` and `/s` stores it uses a fixed count of `1`. |
There was a problem hiding this comment.
🧹 The /s store here contradicts section 1 of this page.
Line 35 states that hstore has no system store, and HstoreProvider.newSystemStore() does return null, so an hstore graph never registers a /s entry with PD. Line 87 correctly lists only /g and /m. The /s branch in PartitionService.updateGraph exists for graphs that do have a system store.
Please drop /s from this sentence, or note that it does not arise for hstore. Same at content/cn/docs/config/config-backend-hstore.md:136.
| --- | ||
| title: "Configuring the HStore Distributed Backend" | ||
| linkTitle: "Config HStore Backend" | ||
| weight: 6 |
There was a problem hiding this comment.
🧹 The description says the page is not yet linked from the sidebar, but the linkage actually missing is elsewhere.
Docsy generates the section nav from the page tree: themes/docsy/layouts/partials/sidebar-tree.html:48 takes (union $s.Pages $s.Sections).ByWeight and filters on toc_hide alone. This page sets no toc_hide, so it does appear under Server Config on merge, after Config HTTPS (weight 4).
What does not update itself is content/en/docs/config/_index.md, which lists the section's pages as four hand-written bullets, and its CN counterpart. Both still list four pages after this merges.
Please add a bullet for this page to both _index.md files, and correct that line in the description.
| search_keywords: | ||
| - hstore | ||
| - pd.peers | ||
| - usePD | ||
| - hstore.partition_count |
There was a problem hiding this comment.
🧹 search_keywords is not read by anything in this repository.
No other markdown page under content/ uses it, layouts/ holds only 404.html, and nothing in themes/docsy or config.toml references it. The only front matter fields docsy reads for its index are title, categories, tags, description and exclude_search (themes/docsy/assets/json/offline-search-index.json). All three search backends are off in config.toml besides: offlineSearch = false, algolia_docsearch = false, and gcs_engine_id commented out. The sibling pages in this section carry title, linkTitle and weight and nothing else.
Please drop the block, or replace it with a description, which docsy does read for the search excerpt and the page meta description. Same at content/cn/docs/config/config-backend-hstore.md:5-9.
| only change behavior, when PD and the `hstore` backend are in use. The source column gives the file and line | ||
| on the HugeGraph master branch where the option is declared. |
There was a problem hiding this comment.
🧹 The source column pins line numbers in another repository's moving branch.
I checked all eight ranges against apache/hugegraph master at 36811483a, the two CoreOptions.java rows included, and every one is correct today. Nothing keeps them correct after the next edit to either file, and this repository has no check that would catch the drift, so the column decays into wrong pointers on a published page.
Please keep the file names and drop the line ranges. Same at content/cn/docs/config/config-backend-hstore.md:151-162.
| - `serializer=binary` is required. Registering the `hstore` backend adds a config space and a store provider | ||
| but no serializer of its own, and the adapter is written against the binary serializer. The built-in default | ||
| of `serializer` is `text`, so this value has to be written out. |
There was a problem hiding this comment.
🧹 Worth a follow-up: this contradicts the manual in the same section.
The statement here is right, CoreOptions.SERIALIZER declares default text. But content/en/docs/config/config-option.md:74 lists serializer with default value binary, and the CN manual carries the same row at the same line. Once this merges, two pages under Server Config state different defaults for one key, and the manual is the one readers reach for first.
That file is outside this diff, so nothing needs to change here. Please open a follow-up for the serializer row and link it from the description.
| `init-store.sh` never initializes an `hstore` graph. On the enabled path it scans `conf/graphs` and skips | ||
| every graph whose backend is `hstore`. If you turn the whole step off with `init_store.enabled=false`, it | ||
| validates instead that the admin account can still be created on the PD startup path: `usePD` has to be true, | ||
| the auth graph has to exist locally with backend `hstore`, and `auth.admin_pa` has to be set to an explicit | ||
| non-empty value. Otherwise startup fails rather than handing out the public default password. |
There was a problem hiding this comment.
🧹 These three conditions only apply when the built-in local authenticator is configured.
InitStore.checkAdminBootstrapReachable() returns before any of them otherwise:
if (!requiresLocalBuiltinAdmin(conf)) {
return;
}The javadoc above it states that remote auth and custom authenticators keep their identities elsewhere and are exempt. As written the paragraph reads as an unconditional requirement, so an operator on remote auth would set usePD, move the auth graph to hstore and configure auth.admin_pa for no reason.
Please add that gate to the sentence. Same at content/cn/docs/config/config-backend-hstore.md:164-167.
imbajin
left a comment
There was a problem hiding this comment.
中文部分与源码保持一致且表述清晰,评分 9.2/10,按规则可直接合并。
Adds the missing configuration page for the
hstoredistributed backend, in English and Chinese. Every facttraces to the HugeGraph master branch.
The page is not yet linked from the sidebar, because nav registration for all three new backend
pages has to land as a single later change.
hstorehad no configuration page at all, so the backend was only reachable through the PD and Store quickstart pages.hugegraph-hstore/src/main/java/org/apache/hugegraph/backend/store/hstore/HstoreProvider.java:30-53,hugegraph-core/src/main/java/org/apache/hugegraph/StandardHugeGraph.java:266-271,281-285,613-621,664-669,1651-1656,hugegraph-core/src/main/java/org/apache/hugegraph/backend/tx/SchemaTransactionV2.java:72,84,648-650,hugegraph-core/src/main/java/org/apache/hugegraph/backend/store/BackendStoreInfo.java:42-48,hugegraph-dist/src/main/java/org/apache/hugegraph/cmd/InitStore.java:113-122UpversusPendingstore registration rule, and the PD/Store gRPC and REST default ports. Links to the PD and Store quickstart pages.hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/StoreNodeService.java:160-167,218-220,hugegraph-pd/hg-pd-dist/src/assembly/static/conf/application.yml:38-55,hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml:32-47serializer=binaryis needed, howstorebecomes the graph name PD sees, or thatpd.peersis read from the graph config.usePDswitch inrest-server.properties, the inheritance ofpd.peersfrom the server config, the shippedhstore.properties.template, the hstore Docker image swap and therocksdb-onlyMaven profile.conf/graphs/hstore.properties.template:25-33,conf/rest-server.properties:26-27,hugegraph-core/src/main/java/org/apache/hugegraph/config/CoreOptions.java:47-82,649-654,679-685,hugegraph-core/src/main/java/org/apache/hugegraph/backend/store/BackendProviderFactory.java:48-49,61-62,70,hugegraph-core/src/main/java/org/apache/hugegraph/backend/store/BackendStoreProvider.java:28-30,hugegraph-dist/src/main/java/org/apache/hugegraph/dist/RegisterUtil.java:77-126,hugegraph-api/src/main/java/org/apache/hugegraph/core/GraphManager.java:251-260,342-372,1603-1642,hugegraph-server/Dockerfile-hstore:39-41,hugegraph-dist/pom.xml:38,294-304,hugegraph-dist/src/assembly/travis/start-server.sh:72-75hstore.partition_countandhstore.shard_countwere documented nowhere, including in the complete configuration manual.0and their descriptions, the validation error for a negative partition count, how PD interprets0and clamps a value above its own total, and the fact that no Server code readshstore.shard_countin this release.hugegraph-hstore/src/main/java/org/apache/hugegraph/backend/store/hstore/HstoreOptions.java:27-38,.../HstoreSessionsImpl.java:126-140,hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/PartitionService.java:425-445,hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/config/PDConfig.java:104-107,263-278init-store.shapplies wheninit_store.enabledisfalse.hugegraph-api/src/main/java/org/apache/hugegraph/config/ServerOptions.java:187-201,355-361,371-380,390-396,591-598,hugegraph-core/src/main/java/org/apache/hugegraph/config/CoreOptions.java:649-654,679-685,hugegraph-dist/src/main/java/org/apache/hugegraph/cmd/InitStore.java:86-92,113-122,202-238.../HstoreSessionsImpl.java:109-122,hugegraph-store/hg-store-client/src/main/java/org/apache/hugegraph/store/HgStoreClient.java:92-101,hugegraph-store/hg-store-client/src/main/java/org/apache/hugegraph/store/client/HgStoreNodePartitionerImpl.java:54-193hugegraph-hstore/src/main/java/org/apache/hugegraph/backend/store/hstore/HstoreFeatures.java:24-135GET /metrics/backendand what anodesvalue of0means.hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java:85,113-134,.../HstoreMetrics.java:37-43,.../HstoreSessionsImpl.java:810-816Paths above are relative to
hugegraph-server/unless they start withhugegraph-pd/,hugegraph-store/,docker/orconf/.