Implements varint + delta encoding on CSR to optimize the fragment memory usage #1372

vegetableysm · 2023-05-12T06:41:58Z

What do these changes do?

Related issue number

github-actions · 2023-05-12T06:48:22Z

🎊 PR Preview 1b66da5 has been successfully built and deployed to https://deploy-preview-pr-1372--v6d.netlify.app
_{🤖 By netlify}

sighingnow · 2023-05-17T02:52:11Z

modules/graph/fragment/varint_impl.h

+}
+
+template <typename T>
+size_t varint_decode(const uint8_t* input, T& output) {


Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

…mpile time and add field to control the process of load graph. Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

sighingnow · 2023-05-22T12:33:15Z

modules/graph/fragment/property_graph_types.h

+  const uint8_t* begin_ptr_;
+  const uint8_t* end_ptr_;
+  size_t size_ = 0;
+


Don't touch Nbr and NbrList as you already have a EncodedNbr and EncodeNbrList variant.

Unify the terminology to only use Encoded or only use Compacted. I personally prefer Compacted.

sighingnow · 2023-05-22T12:34:59Z

modules/graph/fragment/property_graph_utils_impl.h

+               [&e_list, &e_offsets_lists_, &encoded_id_sub_lists](int64_t k) {
+                  VID_T pre_vid = 0;
+                  encoded_id_sub_lists[k].resize(9 * (e_offsets_lists_[k + 1] - e_offsets_lists_[k]));
+                  encoded_id_sub_lists[k].resize(0);


??? resize to 9*xxxx then resize to 0?

std::vector::reserve().

sighingnow · 2023-05-22T12:40:11Z

modules/graph/fragment/property_graph_utils_impl.h

+               [&encoded_id_sub_lists, &encoded_id_list, &encoded_offsets_list](int64_t i) {
+                 memcpy(encoded_id_list.data() + encoded_offsets_list[i], encoded_id_sub_lists[i].data(), encoded_id_sub_lists[i].size());
+               },
+               concurrency);


Could you please report numbers of time used by these two stages (encode, copy) and given a comparison about parallel and non-parallel version?

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

sighingnow · 2023-05-26T02:39:17Z

modules/graph/tools/fragment_loader_impl.h

+          traverse_fragment(fragment, options.dump_dry_run_rounds);
+        } else {
+          dump_fragment(fragment, target_directory + "/output_graph_f" +
+                                      std::to_string(fragment->fid()));


Try add the fragment_t as the template type parameter for traverse_fragment and dump_fragment to avoid the duplication.

sighingnow · 2023-05-26T02:40:01Z

modules/graph/tools/fragment_loader_impl.h

+          });
+    } else {
+      return loadfn().value();
+    }


Cannot see why you need copy the whole file, rather than some template tricks.

sighingnow · 2023-05-26T02:41:39Z

Please fixes the format and lint error as well.

sighingnow · 2023-05-26T02:42:49Z

modules/graph/fragment/arrow_fragment_builder_impl.h

+            << " seconds";
+  LOG(INFO) << "Generate varint time usage: "
+            << (generate_varint_time / 1000000.0) << " seconds";
+  LOG(INFO) << "Memcpy time usage: " << (memcpy_time / 1000000.0) << " seconds";


Use VLOG(100) as I think those profiling logs shouldn't be so verbose.

Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>

vegetableysm force-pushed the main branch from db459bf to b29dc3c Compare May 12, 2023 08:13

sighingnow requested review from mengke-mk and sighingnow May 12, 2023 08:29

vegetableysm force-pushed the main branch from 1295d99 to aac4f0b Compare May 16, 2023 06:58

sighingnow reviewed May 17, 2023

View reviewed changes

vegetableysm force-pushed the main branch from e2af27f to b467ac1 Compare May 17, 2023 09:20

vegetableysm added 23 commits May 17, 2023 17:30

add data field for encoding oe lists

bf3709b

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

add encode/decode function for varint

967fc71

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

add functions to generate varint edges

a71e250

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

construct varint edges

b7ce8a9

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

seal varint oe in vineyard

8dcc520

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

fix a bug when construct varint header

aa81cc1

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

add decode support for Nbr

75cc340

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

make GetOutgoingAdjList support decoding compressed edges

3969c55

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove oe_lists_ from the seal list

9972c0b

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Add data field for encoding ie lists

bb0536e

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Encode edges for ie_lists.

e03af3e

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Seal encoded ie_lists in vineyard

bff5b89

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

make GetInComingAdjList support decoding compressed edges

3e43614

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove ie_lists_ from the seal list

8d551a3

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Add licnese

a154966

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Format code

5a8db0c

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Add field to store real offset of eid && vid in encoded edge lists

953eca2

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Fill offset lists of encoded edges

606fea2

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Record the offset of encoded edges data in vineyard

6ed7cf7

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Speed up the process of accessing encoded edges by ptr_lists.

a664161

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove the header of encoded edges.

3445ab9

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove oe(ie)_offsets_ptr_lists_ from vineyard

1e96274

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Format code

5e04750

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

vegetableysm added 14 commits May 17, 2023 17:30

Add EncodedNbr struct and EncodedAdjList class.

861df66

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove the old design of encoded edges.

c0e8f1b

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Format code.

ab2aee8

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove excess code.

ec41d21

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Delta encoding.

6915ea0

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Format code.

b6c10fa

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Controls the specialization of the function of accessing graphs at co…

eb14254

…mpile time and add field to control the process of load graph. Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Remove the delta encoding for eid.

1e4c5c6

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Rename encode_edges to compact_edges

c6facb2

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Disable RawAdjLists APIs when compact edges.

fe81fcc

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Clean up and format code.

0b16ed0

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Change varint_decode into a inline function.

1b73a35

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Move template param from member function to ArrowFragment class.

cc01630

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Parallel for encoding edges.

185f09d

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

vegetableysm force-pushed the main branch 2 times, most recently from 50113dd to b894006 Compare May 22, 2023 06:56

Optimized encoding process.

548d42c

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

vegetableysm force-pushed the main branch from b894006 to 548d42c Compare May 22, 2023 08:19

sighingnow reviewed May 22, 2023

View reviewed changes

vegetableysm added 3 commits May 25, 2023 15:54

Use config file to control compact.

a29274e

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Clean code.

4d90b1c

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

Format code.

b5cd047

Signed-off-by: vegetableysm <yuanshumin.ysm@alibaba-inc.com>

sighingnow reviewed May 26, 2023

View reviewed changes

Dedup some copy, fixes logging and lint errors

1b66da5

Signed-off-by: Tao He <linzhu.ht@alibaba-inc.com>

sighingnow changed the title ~~Varint encoding csr~~ Implements varint + delta encoding on CSR to optimize the fragment memory usage May 26, 2023

sighingnow merged commit 0c9c16e into v6d-io:main May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implements varint + delta encoding on CSR to optimize the fragment memory usage #1372

Implements varint + delta encoding on CSR to optimize the fragment memory usage #1372

vegetableysm commented May 12, 2023 •

edited by sighingnow

Loading

github-actions bot commented May 12, 2023 •

edited

Loading

sighingnow May 17, 2023

vegetableysm May 17, 2023

sighingnow May 22, 2023

vegetableysm May 26, 2023

sighingnow May 22, 2023

sighingnow May 23, 2023 •

edited

Loading

vegetableysm May 26, 2023

sighingnow May 22, 2023

sighingnow May 26, 2023

sighingnow May 26, 2023

sighingnow commented May 26, 2023

sighingnow May 26, 2023

Implements varint + delta encoding on CSR to optimize the fragment memory usage #1372

Implements varint + delta encoding on CSR to optimize the fragment memory usage #1372

Conversation

vegetableysm commented May 12, 2023 • edited by sighingnow Loading

What do these changes do?

Related issue number

github-actions bot commented May 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sighingnow May 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sighingnow commented May 26, 2023

Choose a reason for hiding this comment

vegetableysm commented May 12, 2023 •

edited by sighingnow

Loading

github-actions bot commented May 12, 2023 •

edited

Loading

sighingnow May 23, 2023 •

edited

Loading