Refactor TsFile #736

HTHou · 2020-01-14T02:24:38Z

https://issues.apache.org/jira/browse/IOTDB-243

LeiRui · 2020-01-14T02:42:33Z

The travis tests fail

HTHou · 2020-01-14T02:47:47Z

The travis tests fail

Yes, there are still lots of works to complete. So it's only a draft.

yuyuankang · 2020-01-14T06:18:53Z

tsfile/src/main/java/org/apache/iotdb/tsfile/compress/ICompressor.java

-        return new SnappyCompressor();
-      default:
-        throw new CompressionTypeNotSupportedException(name.toString());
+    case UNCOMPRESSED:


Please check the indention.

yuyuankang · 2020-01-14T06:58:00Z

tsfile/src/main/java/org/apache/iotdb/tsfile/read/controller/IMetadataQuerier.java

   */
-  TSDataType getDataType(String measurement) throws NoMeasurementException;
+  TSDataType getDataType(Path path) throws NoMeasurementException, IOException;


The comment should be updated too.

…to new_tsFile_v2

HTHou · 2020-02-11T01:36:30Z

The new structure of TsFileMetadata is shown below.

More detailed doc to be added soon.

liutaohua · 2020-02-11T07:30:47Z

I think TsOffsetArray is a bad design, because it is hard to understanding it, and each DeviceIndexArray takes 8 byte to store the index.
If you put the contents of TsOffsetArray directly into the DeviceIndexArray, it would only take 4 byte to record the number, and it would be easier to understand

tsfile/src/main/java/org/apache/iotdb/tsfile/write/writer/TsFileIOWriter.java

tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/ChunkGroupWriterImpl.java

tsfile/src/main/java/org/apache/iotdb/tsfile/write/chunk/IChunkGroupWriter.java

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TimeseriesMetaData.java

qiaojialin

please format the code first

qiaojialin · 2020-02-12T04:05:46Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TimeseriesMetaData.java

+  private int numOfChunkMetaDatas;
+
+  private String measurementId;
+  private List<ChunkMetaData> chunkMetaDataList = new ArrayList<>();


No need to store this in TimeseriesMetaData. I expect TimeseriesMetadata be used in this way:

List TsFileSequenceReader.getAllChunkMetadata(TimeseriesMetaDataIndex metadataIndex)

Thank you, fixed.

qiaojialin · 2020-02-12T04:06:37Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TimeseriesMetaData.java

+
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+public class TimeseriesMetaData {


Suggested change

public class TimeseriesMetaData {

public class TimeseriesMetadataIndex {

Actually this is an index to ChunkMetadatas of one series. Similar to the TsDeviceMetadataIndex

qiaojialin · 2020-02-12T04:08:22Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/TimeseriesMetaData.java

+
+  private long startOffsetOfChunkMetaDataList;
+  private int chunkMetaDataListDataSize;
+  private int numOfChunkMetaDatas;


Do we need this field?

We could read ChunkMetadata from a ByteBuffer like this:

List metadatas = new ArrayList();
for (bytebuffer.remaining > 0) {
metadatas.add(bytebuffer.readChunkMetadata());
}

Fixed. Thank you.

qiaojialin · 2020-02-12T04:14:22Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/TSDataType.java

-        return 5;
-      default:
-        return -1;
+    case BOOLEAN:


Do not change this, our google-style contains the blanks.

qiaojialin · 2020-02-12T04:14:46Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/CompressionType.java

-        return PLA;
-      default:
-        return UNCOMPRESSED;
+    case 0:


change back

qiaojialin · 2020-02-12T04:15:32Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/TSFreqType.java

-        return IRREGULAR_FREQ;
-      default:
-        return IRREGULAR_FREQ;
+    case 0:


change back

qiaojialin · 2020-02-12T04:16:28Z

tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/statistics/Statistics.java

-        return new FloatStatistics();
-      default:
-        throw new UnknownColumnTypeException(type.toString());
+    case INT32:


change back

qiaojialin · 2020-02-12T04:17:47Z

tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/fileInputFactory/HDFSInputFactory.java

-      logger.error(
-          "Failed to get HDFSInput in Hadoop file system. Please check your dependency of Hadoop module.",
-          e);
+      logger.error("Failed to get HDFSInput in Hadoop file system. Please check your dependency of Hadoop module.", e);


I think you need to use idea to format the codes......... Maybe this line exceeds the max length.

HTHou · 2020-02-12T10:50:54Z

The updated structure is here.

…to new_tsFile_v2

HTHou · 2020-02-24T18:53:51Z

I changed some UnitTests in IoTDB server, because I added a new kind of MNode DeviceMNode in MTree. As a result, the deviceId cannot be storage group name any more. The storage group must be on the higher level of device.

HTHou added 9 commits January 6, 2020 17:02

fix a bug when recover the last crashed file

5a4e225

add exception for something gets wrong

ec64d27

fix test errors

68296fc

change the comment

ba57aab

add comment

e7d4626

Merge branch 'master' of https://github.com/apache/incubator-iotdb

835cbfa

new tsfile

8a5793a

Merge branch 'master' of https://github.com/apache/incubator-iotdb

ee2414a

refactor tsfile

4dab61b

yuyuankang reviewed Jan 14, 2020

View reviewed changes

HTHou added 4 commits January 22, 2020 15:35

refactor tsfile

dad198a

resolve conflicts

d17aa55

Merge branch 'master' of https://github.com/apache/incubator-iotdb in…

9edff86

…to new_tsFile_v2

refactor tsfile

17ead2e

HTHou marked this pull request as ready for review February 10, 2020 08:03

fanhualta requested changes Feb 11, 2020

View reviewed changes

qiaojialin requested changes Feb 12, 2020

View reviewed changes

HTHou added 4 commits February 17, 2020 23:27

new interface and new tsfile structure

b44955a

Merge branch 'master' of https://github.com/apache/incubator-iotdb in…

8c04837

…to new_tsFile_v2

fix tsfile problems

aacdc9d

fix some problems

3105712

fanhualta approved these changes Feb 24, 2020

View reviewed changes

qiaojialin mentioned this pull request Feb 24, 2020

[IOTDB-163] Support create device template and create device #527

Closed

resolve conflicts

84272b7

fix some of bugs

e0dc626

HTHou closed this Feb 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor TsFile #736

Refactor TsFile #736

HTHou commented Jan 14, 2020

LeiRui commented Jan 14, 2020

HTHou commented Jan 14, 2020

yuyuankang Jan 14, 2020

yuyuankang Jan 14, 2020

HTHou commented Feb 11, 2020

liutaohua commented Feb 11, 2020

qiaojialin left a comment

qiaojialin Feb 12, 2020

HTHou Feb 23, 2020

qiaojialin Feb 12, 2020

qiaojialin Feb 12, 2020

HTHou Feb 21, 2020

qiaojialin Feb 12, 2020

qiaojialin Feb 12, 2020

qiaojialin Feb 12, 2020

qiaojialin Feb 12, 2020

qiaojialin Feb 12, 2020

HTHou commented Feb 12, 2020

HTHou commented Feb 24, 2020


		import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;

		public class TimeseriesMetaData {

	public class TimeseriesMetaData {
	public class TimeseriesMetadataIndex {

Refactor TsFile #736

Refactor TsFile #736

Conversation

HTHou commented Jan 14, 2020

LeiRui commented Jan 14, 2020

HTHou commented Jan 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HTHou commented Feb 11, 2020

liutaohua commented Feb 11, 2020

qiaojialin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HTHou commented Feb 12, 2020

HTHou commented Feb 24, 2020