Skip to content

[IOTDB-2205] Optimize unaligned int/long read/write functions in BytesUtils#4676

Merged
JackieTien97 merged 4 commits intoapache:masterfrom
543202718:unaligned
Jan 13, 2022
Merged

[IOTDB-2205] Optimize unaligned int/long read/write functions in BytesUtils#4676
JackieTien97 merged 4 commits intoapache:masterfrom
543202718:unaligned

Conversation

@543202718
Copy link
Copy Markdown
Contributor

@543202718 543202718 commented Dec 31, 2021

注意到在现有的代码中,BytesUtils中非对齐的int/long的读写函数一次循环只能处理一个比特,效率较低,可以对其进行优化,使得一次循环可以处理一个字节。

该优化改动了下面的函数:

  • void longToBytes(long srcNum, byte[] result, int pos, int width)
  • void intToBytes(int srcNum, byte[] result, int pos, int width)
  • long bytesToLong(byte[] result, int pos, int width)
  • int bytesToInt(byte[] result, int pos, int width)

通过对用例的查找,发现该优化将会对使用TS_2DIFF编码的数据读写产生影响。

单元性能测试表明,该优化可以提高函数接近5倍的性能。总体性能测试表明,对于采用TS_2DIFF编码且不压缩的INT32/INT64数据,该优化可以减少约40%的读取时间开销。


In the current codes, the read and write functions of unaligned int/long in BytesUtils can only process one bit per loop, which is inefficient. It can be optimized to one byte per loop.

The following functions are optimized:

  • void longToBytes(long srcNum, byte[] result, int pos, int width)
  • void intToBytes(int srcNum, byte[] result, int pos, int width)
  • long bytesToLong(byte[] result, int pos, int width)
  • int bytesToInt(byte[] result, int pos, int width)

Through the use case search, the optimization influences the reading and writing of data encoded by TS_2DIFF.

In the unit performance test, the optimized functions are about 5x faster the current ones. In the overall test, about 40% time cost is reduced in the query of the uncompressed TS_2DIFF-coded INT32/INT64 data due to the optimization .

@HTHou HTHou changed the title Optimize unaligned int/long read/write functions in BytesUtils [IOTDB-2205] Optimize unaligned int/long read/write functions in BytesUtils Dec 31, 2021
Copy link
Copy Markdown
Contributor

@JackieTien97 JackieTien97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! I wonder that how much improvement does this change bring to write performance? Since you only mentioned the query performance improvements in your pr description.

543202718 and others added 2 commits January 4, 2022 10:33
Co-authored-by: Jackie Tien <JackieTien@foxmail.com>
@543202718
Copy link
Copy Markdown
Contributor Author

With the help of IoTDB-benchmark, I measured the performance improvement. Due to the difference in test data, the results are slightly different from those previously reported.

  • The write throughput before improvement is 232,268 pt/s and 237,046 pt/s after improvement. The gap is within the fluctuation range because the write bottleneck is not the optimized function.
  • The query throughput before improvement is 5,648,345 pt/s and 6,767,949 pt/s after improvement. Thers is a 20% improvement.

The modified config of IoTDB-benchmark is listed as follows while others are still the default.

LOOP=1000
SENSOR_NUMBER=4
CLIENT_NUMBER=1
POINT_STEP=1000
INSERT_DATATYPE_PROPORTION=0:1:1:1:1:0
COMPRESSOR=UNCOMPRESSED
ENCODING_INT32=TS_2DIFF
ENCODING_INT64=TS_2DIFF
ENCODING_FLOAT=TS_2DIFF
ENCODING_DOUBLE=TS_2DIFF
LINE_RATIO=0
SIN_RATIO=0
SQUARE_RATIO=0
RANDOM_RATIO=1
CONSTANT_RATIO=0
BATCH_SIZE_PER_WRITE=10000
QUERY_INTERVAL=100000000000

OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0 (for write)
OPERATION_PROPORTION=0:0:1:0:0:0:0:0:0:0:0 (for query)

@HTHou
Copy link
Copy Markdown
Contributor

HTHou commented Jan 12, 2022

Hi, plz run mvn spotless:apply to fix the format issue.

@543202718
Copy link
Copy Markdown
Contributor Author

Hi, plz run mvn spotless:apply to fix the format issue.

Thanks. The format issue is fixed in the new commit.

@JackieTien97 JackieTien97 merged commit 37bb7e2 into apache:master Jan 13, 2022
@543202718 543202718 deleted the unaligned branch January 13, 2022 02:32
@jihaitangguo
Copy link
Copy Markdown

Note: There was a bug in the code submitted here, IOTDB-4633, which has been fixed in #7669, where

byte y = (byte) (srcNum >> width);

has been corrected to

byte y = (byte) (srcNum >>> width);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants