Skip to content

[IOTDB-2559][IOTDB-1840][IOTDB-2215][IOTDB-2216] Persistent Schema Management#4732

Merged
qiaojialin merged 322 commits intoapache:masterfrom
MarcosZyk:persistent_schema
Apr 11, 2022
Merged

[IOTDB-2559][IOTDB-1840][IOTDB-2215][IOTDB-2216] Persistent Schema Management#4732
qiaojialin merged 322 commits intoapache:masterfrom
MarcosZyk:persistent_schema

Conversation

@MarcosZyk
Copy link
Copy Markdown
Contributor

@MarcosZyk MarcosZyk commented Jan 8, 2022

Description

Docs

All related docs are in the shared folder linked by the following url.
https://y8dp9fjm8f.feishu.cn/drive/folder/fldcnoxcC1ZU9Ux43DCfImzXYqg?from=from_copylink

Implement MTree Cache Framework

  1. Rewrite attribute children management in MNode and implement MNodeContainer
  2. Separate service and store of MTree
    1. implement MemMTreeStore and CachedMTreeStore for memory mode and persistent mode respectively
    2. rewrite service logic in MTreeBelowSG and Traverser based on the persistent mode
  3. Implement CacheManager for CachedMTreeStore
    1. implement cache framework and two cache strategy, plain and LRU
    2. implement Pin/Unpin to manage mnode cache and buffer

Implement SchemaFile

  1. Implement mnode operations
  2. Implement page space management

Currently, we have not ensured the consistency and reliance of schemafile when encountering unexpected abort. It could be understand as we extend the workspace of metadata from memory to disk. Now we still recover SchemaRegion based on redoing mlog.

New config parameters

  1. enable_persistent_schema: whether to use persistent schema mode
  2. cached_metadata_size_in_persistent_mode: the memory used for metadata cache when using persistent schema, currently this represents the num of cached mnode
  3. max_schema_flush_thread: the max num of thread used for flushing metadata to schema file
  4. minimum_schema_file_segment_in_bytes: the minimum size (in bytes) for segment inside a schema file page
  5. page_cache_in_schema_file: the cache size for schema page in one schema file

Future work

implement SchemaRegion recover based on schemafile

@coveralls
Copy link
Copy Markdown

coveralls commented Jan 8, 2022

Coverage Status

Coverage decreased (-2.3%) to 65.428% when pulling 0f3c1a7 on MarcosZyk:persistent_schema into 3f915d0 on apache:master.

MarcosZyk and others added 29 commits February 9, 2022 12:53
…ent_schema

� Conflicts:
�	docs/SystemDesign/SchemaManager/SchemaManager.md
�	docs/zh/SystemDesign/SchemaManager/SchemaManager.md
�	server/src/assembly/resources/conf/iotdb-engine.properties
�	server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
�	server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
�	server/src/main/java/org/apache/iotdb/db/metadata/mnode/InternalMNode.java
@qiaojialin qiaojialin merged commit 1c28868 into apache:master Apr 11, 2022
xinzhongtianxia pushed a commit to xinzhongtianxia/iotdb that referenced this pull request Apr 11, 2022
…er_meta

* remotes/upstream/master:
  [IOTDB-2687] Base partition policy of data (apache#5464)
  [IOTDB-2868] Construct ExchangeOperator in LocalExecutionPlanner (apache#5466)
  implement serialize and deserialize of SchemaTree (apache#5474)
  [IOTDB-2559][IOTDB-1840][IOTDB-2215][IOTDB-2216] Persistent Schema Management (apache#4732)
  [IOTDB-2876] Definition of common thrift struct (apache#5471)

# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/mpp/common/DataRegion.java
#	server/src/main/java/org/apache/iotdb/db/mpp/sql/planner/LogicalPlanner.java
#	server/src/test/java/org/apache/iotdb/db/mpp/sql/plan/DistributionPlannerTest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants