Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IOTDB-294]online upgrade from v0.8.0 to current version #467

Merged
merged 27 commits into from Nov 12, 2019
Merged

[IOTDB-294]online upgrade from v0.8.0 to current version #467

merged 27 commits into from Nov 12, 2019

Conversation

EJTTianYu
Copy link
Contributor

No description provided.

jt2594838
jt2594838 requested changes Oct 24, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/TsfileUpgradeToolV0_8_0.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkBuffer.java Outdated Show resolved Hide resolved
fanhualta
fanhualta requested changes Oct 25, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileResource.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/service/UpgradeSevice.java Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/encoding/decoder/PlainDecoder.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
fanhualta
fanhualta requested changes Nov 6, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeCheckStatus.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
tsfilePathBefore + DOT_SEPERATOR + tsfilePathAfter + DOT_SEPERATOR
+ UpgradeCheckStatus.AFTER_UPGRADE_FILE, upgradePostion);
} catch (IOException e) {
logger.error("generate upgrade file failed, the file to be upgraded:{}", tsfilePathBefore);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the upgrade occurs an error, you just throw an Exception and move the incomplete file to the data directory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in recovery process, system will remove the imcomplete tsfile

Copy link
Contributor

@fanhualta fanhualta Nov 7, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it occurs in the running of the system, the system engine may read this file which lead to errors.

server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/service/UpgradeSevice.java Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Show resolved Hide resolved
jt2594838
jt2594838 requested changes Nov 6, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/utils/UpgradeUtils.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated Show resolved Hide resolved
tsfile/src/main/java/org/apache/iotdb/tsfile/tool/upgrade/UpgradeTool.java Outdated
oldVersionTsfileDirs.add("/Users/tianyu/incubator-iotdb/data/data/sequence/root.group_10");
newVersionTsfileDirs.add("/Users/tianyu/incubator-iotdb/data/data/sequence/root.group_8");
for (int i = 0; i < oldVersionTsfileDirs.size(); i++) {
updateTsfiles(oldVersionTsfileDirs.get(i), newVersionTsfileDirs.get(i));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about making this a real tool.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea,I will make it a sh script then.

@EJTTianYu EJTTianYu changed the title online upgrade from v0.8.0 to current version [IOTDB-294]online upgrade from v0.8.0 to current version Nov 7, 2019
jt2594838
jt2594838 requested changes Nov 7, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeLog.java Outdated

public static boolean writeUpgradeLogFile(String content) {
UpgradeUtils.getUpgradeLogLock().writeLock().lock();
try (BufferedWriter upgradeLogWriter = new BufferedWriter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is not necessary to open a writer every time, you can reuse one.

fanhualta
fanhualta requested changes Nov 7, 2019
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java Outdated Show resolved Hide resolved
server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java Outdated
tsfilePathBefore + DOT_SEPERATOR + tsfilePathAfter + DOT_SEPERATOR
+ UpgradeCheckStatus.AFTER_UPGRADE_FILE, upgradePostion);
} catch (IOException e) {
logger.error("generate upgrade file failed, the file to be upgraded:{}", tsfilePathBefore);
Copy link
Contributor

@fanhualta fanhualta Nov 7, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When it occurs in the running of the system, the system engine may read this file which lead to errors.

qiaojialin
qiaojialin requested changes Nov 10, 2019
View reviewed changes
Copy link
Member

@qiaojialin qiaojialin left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the "reading TsFile-0.8.0" bug fixed?

@EJTTianYu
Copy link
Contributor Author

Is the "reading TsFile-0.8.0" bug fixed?
fixed, when a tsfile is old version, we consider this tsfile a complete file. We would generate totalChunkNum using tsDeviceMetaData and set invalidChunkNum as 0.
related file:
tsfile/src/test/java/org/apache/iotdb/tsfile/file/metadata/TsFileMetaDataTest.java
tsfile/src/main/java/org/apache/iotdb/tsfile/read/TsFileSequenceReader.java
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TsFileMetaData.java

However, it still has some conflict with merge as you have said. I will discuss with jt to solve the problem.

@qiaojialin qiaojialin merged commit 8682fe2 into apache:master Nov 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qiaojialin qiaojialin qiaojialin approved these changes

@jt2594838 jt2594838 jt2594838 approved these changes

@fanhualta fanhualta fanhualta approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@EJTTianYu @qiaojialin @jt2594838 @fanhualta