Skip to content

[IOTDB-292]Add load external tsfile feature#597

Merged
fanhualta merged 13 commits intomasterfrom
add_load_tsfile_feature
Dec 5, 2019
Merged

[IOTDB-292]Add load external tsfile feature#597
fanhualta merged 13 commits intomasterfrom
add_load_tsfile_feature

Conversation

@fanhualta
Copy link
Copy Markdown
Contributor

This pr adds a new feature of loading external tsfile, it allows the following three operations:

  1. load new tsfiles to the Apache IoTDB engine instance.
  2. remove a tsfile from the Apache IoTDB engine instance.
  3. move a tsfile from the Apache IoTDB engine instance to a target directory.

Corresponding jira link: https://issues.apache.org/jira/browse/IOTDB-292.

@jixuan1989
Copy link
Copy Markdown
Member

[INFO] 
[ERROR] Errors: 
[ERROR]   IoTDBLoadExternalTsfileTest.moveTsfileTest:130 � IoTDBSQL Cannot move file dat...
[INFO] 

pay attention to the error. is there something not released?

@fanhualta
Copy link
Copy Markdown
Contributor Author

[INFO] 
[ERROR] Errors: 
[ERROR]   IoTDBLoadExternalTsfileTest.moveTsfileTest:130 � IoTDBSQL Cannot move file dat...
[INFO] 

pay attention to the error. is there something not released?

I will try the ut in the windows environment.

Copy link
Copy Markdown
Contributor

@HTHou HTHou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks great.

Copy link
Copy Markdown
Member

@jixuan1989 jixuan1989 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need to upgrade to antlr4

@fanhualta
Copy link
Copy Markdown
Contributor Author

need to upgrade to antlr4

I have upgraded to antlr4.

Copy link
Copy Markdown
Member

@qiaojialin qiaojialin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

修改一下加载tsfile的功能吧:

用户有一个 tsfile,放在任意位置,使用 load工具 加载文件,
传入三个参数

tsfile 文件路径
2,是否根据tsfile的schema创建元数据(可选,默认创建)
3,存储组名字(可选,默认以路径前二层创建存储组, root.sg1 这种)
load工具做的功能:
判断这个文件是否为 sequence文件,然后移动到对应文件夹,生成 resource

@fanhualta
Copy link
Copy Markdown
Contributor Author

修改一下加载tsfile的功能吧:

用户有一个 tsfile,放在任意位置,使用 load工具 加载文件,
传入三个参数

tsfile 文件路径
2,是否根据tsfile的schema创建元数据(可选,默认创建)
3,存储组名字(可选,默认以路径前二层创建存储组, root.sg1 这种)
load工具做的功能:
判断这个文件是否为 sequence文件,然后移动到对应文件夹,生成 resource

I don't think so. In a complete tsfile, the data type of timeseries and the storage group are fixed. It cannot change by users. I can add the feature of creating schemas automatically while the schemas don't exist.

@fanhualta
Copy link
Copy Markdown
Contributor Author

修改一下加载tsfile的功能吧:

用户有一个 tsfile,放在任意位置,使用 load工具 加载文件,
传入三个参数

tsfile 文件路径
2,是否根据tsfile的schema创建元数据(可选,默认创建)
3,存储组名字(可选,默认以路径前二层创建存储组, root.sg1 这种)
load工具做的功能:
判断这个文件是否为 sequence文件,然后移动到对应文件夹,生成 resource

I have added the feature you requested, please check.

@fanhualta fanhualta force-pushed the add_load_tsfile_feature branch from 1962de5 to f4b4b80 Compare December 5, 2019 08:41
@fanhualta fanhualta merged commit 1c8aa34 into master Dec 5, 2019
@fanhualta fanhualta deleted the add_load_tsfile_feature branch December 5, 2019 11:25
Map<String, MeasurementSchema> schemaMap = new HashMap<>();
List<ChunkGroupMetaData> chunkGroupMetaData = new ArrayList<>();
try (TsFileSequenceReader reader = new TsFileSequenceReader(file.getAbsolutePath(), false)) {
reader.selfCheck(schemaMap, chunkGroupMetaData, false);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my understanding, the reader.selfCheck method with the parameter fastFinish assigned false will sequentially read the whole data part of the tsfile.
If the method here is only used to generate the schemaMap and chunkGroupMetaData, why not just read the metadata at the end of the tsfile, since the tsfile is always complete here?

@fanhualta fanhualta restored the add_load_tsfile_feature branch December 26, 2019 01:46
@fanhualta fanhualta deleted the add_load_tsfile_feature branch December 26, 2019 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants