Skip to content

add compressed graph datacell with Elias-Fano Encoder#596

Merged
nedchu merged 7 commits into
mainfrom
compressed-graph-datacell
Apr 29, 2025
Merged

add compressed graph datacell with Elias-Fano Encoder#596
nedchu merged 7 commits into
mainfrom
compressed-graph-datacell

Conversation

@nedchu

@nedchu nedchu commented Apr 16, 2025

Copy link
Copy Markdown
Collaborator

closes #595

Summary of changes

  1. src/data_cell/compressed_graph_datacell.h Graph datacell with Elias-Fano encoder that can compress the adjacency list
  2. src/data_cell/graph_storage_type.h Add GraphStorageTypes, can choose to use compressed graph, or sparse graph, or flat one
enum class GraphStorageTypes {
    GRAPH_STORAGE_TYPE_FLAT = 0,
    GRAPH_STORAGE_TYPE_COMPRESSED = 1
};
  1. src/data_cell/graph_interface_test.h Sort neighbor sets in BasicTest if needed, elias-fano encoder will sort neighbor sets
  2. src/impl/elias_fano_encoder.h: Update elias-fano encoder
    a. Data visibility change to public (to serialize CompressedGraphDatacell)
    b. Change DecompressAll interface (to fit CompressedGraphDatacell)
from
    Vector<InnerIdType>
    DecompressAll(Allocator* allocator) const;
to
    void
    DecompressAll(Vector<InnerIdType>& neighbors) const;

@codecov

codecov Bot commented Apr 16, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 94.61538% with 7 lines in your changes missing coverage. Please review.

@@            Coverage Diff             @@
##             main     #596      +/-   ##
==========================================
- Coverage   91.37%   91.36%   -0.02%     
==========================================
  Files         180      182       +2     
  Lines       10761    10837      +76     
==========================================
+ Hits         9833     9901      +68     
- Misses        928      936       +8     
Flag Coverage Δ
cpp 91.36% <94.61%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 93.28% <ø> (-0.09%) ⬇️
datacell 92.03% <91.86%> (+0.04%) ⬆️
index 89.52% <ø> (-0.06%) ⬇️
simd 100.00% <ø> (ø)

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 22cbb09...84b4278. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wxyucs wxyucs added version/0.15 1. Add Attribute Filter 2. Quantization and transformation 1. 新增 Attribute Filter 2. 量化与变换 kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 labels Apr 18, 2025

@wxyucs wxyucs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@nedchu
nedchu force-pushed the compressed-graph-datacell branch from 3750fb0 to e433982 Compare April 18, 2025 13:50
Comment thread src/data_cell/compressed_graph_datacell.cpp Outdated
Comment thread src/data_cell/compressed_graph_datacell.cpp Outdated
Comment thread src/impl/elias_fano_encoder.h Outdated
Comment thread src/impl/elias_fano_encoder.cpp
Comment thread src/data_cell/graph_interface_parameter.h

@jiaweizone jiaweizone left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread src/impl/elias_fano_encoder.cpp
@nedchu
nedchu force-pushed the compressed-graph-datacell branch from 7d89c25 to 1fc6376 Compare April 25, 2025 06:32
nedchu added 7 commits April 29, 2025 14:13
Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
…e_parameter.h, add coverage

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
…pdate graph, throw exception on neighbor size exceeds

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
@nedchu
nedchu force-pushed the compressed-graph-datacell branch from 1fc6376 to 84b4278 Compare April 29, 2025 06:13
@nedchu
nedchu merged commit 614ac17 into main Apr 29, 2025
@nedchu
nedchu deleted the compressed-graph-datacell branch April 29, 2025 12:26
LHT129 pushed a commit to LHT129/vsag that referenced this pull request Apr 16, 2026
* feat(data_cell): implement compressed_graph_datacell

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* style minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* refactor(graph_interface): graph_storage_type_ move to graph_interface_parameter.h, add coverage

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(elias-fano-encoder): wrong const

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(compressed-graph-datacell): wrong InsertNeighborsById, add test update graph, throw exception on neighbor size exceeds

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix comments: remove _, fix lint

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

---------

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
LHT129 pushed a commit that referenced this pull request May 11, 2026
* feat(data_cell): implement compressed_graph_datacell

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* style minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* refactor(graph_interface): graph_storage_type_ move to graph_interface_parameter.h, add coverage

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(elias-fano-encoder): wrong const

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(compressed-graph-datacell): wrong InsertNeighborsById, add test update graph, throw exception on neighbor size exceeds

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix comments: remove _, fix lint

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

---------

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: LHT129 <tianlan.lht@antgroup.com>
Sia-Sheerland pushed a commit to Sia-Sheerland/vsag that referenced this pull request Jun 26, 2026
* feat(data_cell): implement compressed_graph_datacell

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* style minor

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* refactor(graph_interface): graph_storage_type_ move to graph_interface_parameter.h, add coverage

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(elias-fano-encoder): wrong const

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix(compressed-graph-datacell): wrong InsertNeighborsById, add test update graph, throw exception on neighbor size exceeds

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

* fix comments: remove _, fix lint

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>

---------

Signed-off-by: Deming Chu <chudeming.cdm@antgroup.com>
Signed-off-by: Sia Sheerland <x1075956441x@163.com>

Signed-off-by: Sia Sheerland <x1075956441x@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/improvement Optimizations, UX polish, or minor improvements 性能优化、体验打磨或细节改良 size/L version/0.15 1. Add Attribute Filter 2. Quantization and transformation 1. 新增 Attribute Filter 2. 量化与变换

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add compressed graph datacell

4 participants