Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'large_attr' data structure in op definition for large chunk (data of pandas/numpy) #1303

Merged
merged 6 commits into from
Jan 27, 2022

Conversation

lidongze0629
Copy link
Collaborator

@lidongze0629 lidongze0629 commented Jan 26, 2022

What do these changes do?

size time(s)
934M (vertices) 42.56s

Related issue number

links #1290

@codecov-commenter
Copy link

codecov-commenter commented Jan 26, 2022

Codecov Report

Merging #1303 (a233274) into main (3dc75f4) will decrease coverage by 3.14%.
The diff coverage is 93.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1303      +/-   ##
==========================================
- Coverage   76.75%   73.60%   -3.15%     
==========================================
  Files          88       88              
  Lines        9510     9508       -2     
==========================================
- Hits         7299     6998     -301     
- Misses       2211     2510     +299     
Impacted Files Coverage Δ
python/graphscope/framework/loader.py 83.73% <62.50%> (-0.14%) ⬇️
python/graphscope/client/rpc.py 85.24% <100.00%> (+0.12%) ⬆️
python/graphscope/client/utils.py 89.18% <100.00%> (+0.14%) ⬆️
python/graphscope/framework/dag_utils.py 62.66% <100.00%> (-0.13%) ⬇️
python/graphscope/framework/graph_utils.py 80.48% <100.00%> (-0.39%) ⬇️
python/graphscope/framework/operation.py 86.11% <100.00%> (+0.60%) ⬆️
python/graphscope/tests/unittest/test_download.py 20.00% <0.00%> (-80.00%) ⬇️
python/graphscope/dataset/ppi.py 31.25% <0.00%> (-56.25%) ⬇️
python/graphscope/dataset/cora.py 31.25% <0.00%> (-56.25%) ⬇️
python/graphscope/dataset/ogbl_ddi.py 31.25% <0.00%> (-56.25%) ⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3dc75f4...a233274. Read the comment docs.

@@ -1058,7 +1058,7 @@ bl::result<void> GrapeInstance::registerGraphType(const rpc::GSParams& params) {
bl::result<std::shared_ptr<DispatchResult>> GrapeInstance::OnReceive(
const CommandDetail& cmd) {
auto r = std::make_shared<DispatchResult>(comm_spec_.worker_id());
rpc::GSParams params(cmd.params);
rpc::GSParams params(cmd.params, cmd.large_attr);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the cmd.large_attr be copied? or just a reference?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will double-check any possible copy both in c++ and python side.

analytical_engine/core/io/property_parser.h Outdated Show resolved Hide resolved
analytical_engine/core/io/property_parser.h Outdated Show resolved Hide resolved
analytical_engine/core/server/command_detail.h Outdated Show resolved Hide resolved
analytical_engine/core/server/command_detail.h Outdated Show resolved Hide resolved
analytical_engine/core/server/rpc_utils.h Show resolved Hide resolved
@lidongze0629 lidongze0629 changed the title [WIP] Add 'large_attr' data structure in op definition for large chunk (data of pandas/numpy) Add 'large_attr' data structure in op definition for large chunk (data of pandas/numpy) Jan 27, 2022
@sighingnow sighingnow merged commit 10f2580 into alibaba:main Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants