Skip to content

Premerge for the distributed version #1169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
qiaojialin merged 43 commits into from
Jun 8, 2020
Merged

Premerge for the distributed version #1169

qiaojialin merged 43 commits into from
Jun 8, 2020

Conversation

@jt2594838
Copy link
Contributor

@jt2594838 jt2594838 commented May 8, 2020

No description provided.

jixuan1989
jixuan1989 reviewed May 13, 2020
View reviewed changes
jdbc/src/main/java/org/apache/iotdb/jdbc/Config.java Outdated
public static final String JDBC_DRIVER_NAME = "org.apache.iotdb.jdbc.IoTDBDriver";

public static boolean rpcThriftCompressionEnable = false;
public static boolean rpcThriftCompressionEnable = true;
Copy link
Member

@jixuan1989 jixuan1989 May 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why set it as true?

Copy link
Contributor Author

@jt2594838 jt2594838 May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is said in thrift that when the thrift structures are large and sent frequently, it will be overall better to enable compression, so I made the default compression true.

@jt2594838
Merge branch 'master' into cluster_premerge
76acdd5 
# Conflicts:
#	session/src/main/java/org/apache/iotdb/session/Session.java
@jt2594838 jt2594838 added the Module - Cluster PRs for the cluster module label May 18, 2020
jt2594838 added 2 commits May 18, 2020 10:42
@jt2594838
Merge branch 'master' into cluster_premerge
448c5b8 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/engine/upgrade/UpgradeTask.java
#	server/src/main/java/org/apache/iotdb/db/engine/version/SimpleFileVersionController.java
@jt2594838
fix sonar isssues
425ec9d
HTHou
HTHou reviewed May 18, 2020
View reviewed changes
jt2594838 and others added 8 commits May 18, 2020 17:48
@jt2594838
fix sonar issues
780b426
@jt2594838 @HTHou
Update service-rpc/src/main/thrift/cluster.thrift
898cede 
Co-authored-by: Haonan <hhaonan@outlook.com>
@jt2594838 @HTHou
Update service-rpc/src/main/thrift/cluster.thrift
135b845 
Co-authored-by: Haonan <hhaonan@outlook.com>
@jt2594838 @HTHou
Update service-rpc/src/main/thrift/cluster.thrift
207b279 
Co-authored-by: Haonan <hhaonan@outlook.com>
@jt2594838 @HTHou
Update service-rpc/src/main/thrift/cluster.thrift
12b7aec 
Co-authored-by: Haonan <hhaonan@outlook.com>
@jt2594838
fix by review
a22c3b5
@jt2594838
Merge branch 'cluster_premerge' of https://github.com/apache/incubato…
3892e24 
…r-iotdb into cluster_premerge
@jt2594838
Merge branch 'master' into cluster_premerge
495685e 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/qp/constant/SQLConstant.java
@jixuan1989
Copy link
Member

jixuan1989 commented May 19, 2020

Seems new files are defined.
partitionDirectFileVersions, partitionMaxFileVersions.

Is there some design docs?

@jt2594838
Copy link
Contributor Author

jt2594838 commented May 20, 2020

Seems new files are defined.
partitionDirectFileVersions, partitionMaxFileVersions.

Is there some design docs?

These are not persisted because they can be calculated by TsFileResources and there are no new files.

qiaojialin
qiaojialin approved these changes May 20, 2020
View reviewed changes
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
TsFileResource existingTsFile = iterator.next();
if (newTsFile.getHistoricalVersions().containsAll(existingTsFile.getHistoricalVersions())
&& !newTsFile.getHistoricalVersions().equals(existingTsFile.getHistoricalVersions())
&& existingTsFile.tryWriteLock()) {
Copy link
Member

@qiaojialin qiaojialin May 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we handle if trylock failed? ignore?

Copy link
Contributor Author

@jt2594838 jt2594838 May 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and we will leave this for merge.

Copy link
Member

@jixuan1989 jixuan1989 May 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where to leave the message?

Copy link
Contributor Author

@jt2594838 jt2594838 May 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment added

jixuan1989
jixuan1989 requested changes May 20, 2020
View reviewed changes
Copy link
Member

@jixuan1989 jixuan1989 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work. See my comments.
many of them are minor issues but some of them are critical, IMO.

server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java

((LeafMNode) measurementNode)
.updateCachedLast(plan.composeTimeValuePair(i), true, latestFlushedTime);
if (measurementNode != null) {
Copy link
Member

@jixuan1989 jixuan1989 May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why check null here? is there a bug before?

Copy link
Contributor Author

@jt2594838 jt2594838 May 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is because we do not always have MNodes and data on the same node together in the distributed version.

jt2594838 and others added 6 commits May 21, 2020 14:01
@jt2594838
add comments for replace users and roles
b16379d
@jt2594838
Merge branch 'master' into cluster_premerge
54f3de4 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/engine/StorageEngine.java
#	server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
#	server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
#	session/src/main/java/org/apache/iotdb/session/Session.java
add comments
3fa3907
Merge branch 'cluster_premerge' of github.com:apache/incubator-iotdb …
9aa51cc 
…into cluster_premerge
@jt2594838
use relative path in mod file serialization
e78fabc
@jt2594838
Merge branch 'master' into cluster_premerge
598718e 
# Conflicts:
#	server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
jt2594838 and others added 14 commits June 3, 2020 10:43
@jt2594838
Merge branch 'master' into cluster_premerge
57f1cbf 
# Conflicts:
#	server/src/main/antlr4/org/apache/iotdb/db/qp/strategy/SqlBase.g4
#	server/src/main/java/org/apache/iotdb/db/auth/authorizer/BasicAuthorizer.java
#	server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
#	server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
#	server/src/main/java/org/apache/iotdb/db/qp/physical/crud/InsertPlan.java
#	server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
#	server/src/test/java/org/apache/iotdb/db/qp/plan/PhysicalPlanTest.java
#	server/src/test/java/org/apache/iotdb/db/tools/WalCheckerTest.java
@jt2594838
fix wal checker
998e97e
@jt2594838
fix WalCheckerTest
ae777e6
@jt2594838
disable Thrift compression by default
8b39ef9
@jt2594838
disable thrift compression in onfig
ea5d72a
@jt2594838
remove exception in remote cache
2d9e103
@jt2594838
disable compression in session pool
be02cba
@jt2594838
fix SessionPoolTest
a39afec
@jt2594838
Merge branch 'master' into cluster_premerge
9b9821f 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
#	server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
@qiaojialin
remove while
d3b1d43
@jt2594838
fix when schema is null
e259a2f
@jt2594838
Merge branch 'master' into cluster_premerge
51c2ade 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
#	server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
#	server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
@jt2594838
Merge branch 'master' into cluster_premerge
a9b8650 
# Conflicts:
#	server/src/main/java/org/apache/iotdb/db/qp/executor/PlanExecutor.java
#	server/src/main/java/org/apache/iotdb/db/qp/strategy/LogicalGenerator.java
@jt2594838
fix test
47047a3
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jun 8, 2020

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 2 Security Hotspots to review)
Code Smell A 3 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

warning The version of Java (1.8.0_242) you have used to run this analysis is deprecated and we will stop accepting it from October 2020. Please update to at least Java 11.
Read more here

@qiaojialin qiaojialin merged commit 1079e2e into master Jun 8, 2020
@qiaojialin qiaojialin deleted the cluster_premerge branch June 8, 2020 06:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jixuan1989 jixuan1989 jixuan1989 requested changes

@HTHou HTHou HTHou left review comments

@qiaojialin qiaojialin qiaojialin approved these changes

Assignees

No one assigned

Labels

Module - Cluster PRs for the cluster module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jt2594838 @jixuan1989 @qiaojialin @HTHou