Skip to content

[IOTDB-294]online upgrade from v0.8.0 to current version#467

Merged
qiaojialin merged 27 commits into
apache:masterfrom
EJTTianYu:master
Nov 12, 2019
Merged

[IOTDB-294]online upgrade from v0.8.0 to current version#467
qiaojialin merged 27 commits into
apache:masterfrom
EJTTianYu:master

Conversation

@EJTTianYu
Copy link
Copy Markdown
Contributor

No description provided.

Comment thread server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkBuffer.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
Comment thread server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/service/UpgradeSevice.java
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeCheckStatus.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
tsfilePathBefore + DOT_SEPERATOR + tsfilePathAfter + DOT_SEPERATOR
+ UpgradeCheckStatus.AFTER_UPGRADE_FILE, upgradePostion);
} catch (IOException e) {
logger.error("generate upgrade file failed, the file to be upgraded:{}", tsfilePathBefore);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the upgrade occurs an error, you just throw an Exception and move the incomplete file to the data directory?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in recovery process, system will remove the imcomplete tsfile

Copy link
Copy Markdown
Contributor

@fanhualta fanhualta Nov 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it occurs in the running of the system, the system engine may read this file which lead to errors.

Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/service/UpgradeSevice.java
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/utils/UpgradeUtils.java Outdated
Comment thread tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
oldVersionTsfileDirs.add("/Users/tianyu/incubator-iotdb/data/data/sequence/root.group_10");
newVersionTsfileDirs.add("/Users/tianyu/incubator-iotdb/data/data/sequence/root.group_8");
for (int i = 0; i < oldVersionTsfileDirs.size(); i++) {
updateTsfiles(oldVersionTsfileDirs.get(i), newVersionTsfileDirs.get(i));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making this a real tool.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea,I will make it a sh script then.

@EJTTianYu EJTTianYu changed the title online upgrade from v0.8.0 to current version [IOTDB-294]online upgrade from v0.8.0 to current version Nov 7, 2019

public static boolean writeUpgradeLogFile(String content) {
UpgradeUtils.getUpgradeLogLock().writeLock().lock();
try (BufferedWriter upgradeLogWriter = new BufferedWriter(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not necessary to open a writer every time, you can reuse one.

Comment thread server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated
Comment thread server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated
tsfilePathBefore + DOT_SEPERATOR + tsfilePathAfter + DOT_SEPERATOR
+ UpgradeCheckStatus.AFTER_UPGRADE_FILE, upgradePostion);
} catch (IOException e) {
logger.error("generate upgrade file failed, the file to be upgraded:{}", tsfilePathBefore);
Copy link
Copy Markdown
Contributor

@fanhualta fanhualta Nov 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it occurs in the running of the system, the system engine may read this file which lead to errors.

Copy link
Copy Markdown
Member

@qiaojialin qiaojialin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the "reading TsFile-0.8.0" bug fixed?

@EJTTianYu
Copy link
Copy Markdown
Contributor Author

Is the "reading TsFile-0.8.0" bug fixed?
fixed, when a tsfile is old version, we consider this tsfile a complete file. We would generate totalChunkNum using tsDeviceMetaData and set invalidChunkNum as 0.
related file:
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/TsFileMetaDataTest.java
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TsFileMetaData.java

However, it still has some conflict with merge as you have said. I will discuss with jt to solve the problem.

@EJTTianYu EJTTianYu requested a review from qiaojialin November 10, 2019 02:38
@qiaojialin qiaojialin merged commit 8682fe2 into apache:master Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants