Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-2551: [Plasma] Improve notification logic #2031

Closed

Conversation

zhijunfu
Copy link
Contributor

This change targets to improve a few places in current plasma notification code:

  1. When a client subscribes to Plasma, the store pushes notifications about existing objects to ALL subscribers, while it should only push to the new subscriber.
  2. And in the above scenario, it should only push "sealed" objects to the new subscriber, while currently it pushes all objects regardless of the state.
  3. When a client disconnects, it will no longer be able to receive notifications, thus the NotificationQueue for the client should be removed from global map.

@codecov-io
Copy link

codecov-io commented May 11, 2018

Codecov Report

Merging #2031 into master will increase coverage by 0.02%.
The diff coverage is 94.73%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2031      +/-   ##
==========================================
+ Coverage   86.35%   86.37%   +0.02%     
==========================================
  Files         242      230      -12     
  Lines       41184    40474     -710     
==========================================
- Hits        35565    34960     -605     
+ Misses       5619     5514     -105
Impacted Files Coverage Δ
cpp/src/plasma/store.h 100% <ø> (ø) ⬆️
cpp/src/plasma/store.cc 88.71% <94.73%> (-0.05%) ⬇️
python/pyarrow/array.pxi 66.04% <0%> (-2.5%) ⬇️
python/pyarrow/feather.pxi 67.44% <0%> (-1.45%) ⬇️
cpp/src/arrow/util/bit-util.h 98.01% <0%> (-1.23%) ⬇️
python/pyarrow/types.pxi 57.9% <0%> (-1.09%) ⬇️
python/pyarrow/table.pxi 68.69% <0%> (-1.07%) ⬇️
python/pyarrow/__init__.py 57.14% <0%> (-0.76%) ⬇️
python/pyarrow/io.pxi 62.23% <0%> (-0.53%) ⬇️
cpp/src/arrow/io/hdfs-internal.cc 33.62% <0%> (-0.45%) ⬇️
... and 52 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6646864...84f4935. Read the comment docs.

@@ -104,7 +104,7 @@ GetRequest::GetRequest(Client* client, const std::vector<ObjectID>& object_ids)
num_objects_to_wait_for = unique_ids.size();
}

Client::Client(int fd) : fd(fd) {}
Client::Client(int fd) : fd(fd), notification_fd(boost::none) {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make notification_fd = -1 by default if not set (and get rid of the boost dependency)

Copy link
Contributor

@pcmoritz pcmoritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! Can you get rid of the boost::option dependency and use the value -1 instead of boost::none? At the moment plasma doesn't depend on boost and I'd like to keep it that way unless there is a good reason to introduce it.

Also can you rebase the PR? There is a small conflict at the moment.

@zhijunfu
Copy link
Contributor Author

@pcmoritz Thanks for reviewing :) Changed boost::none to -1 and did a rebase to resolve the conflict.

@pcmoritz
Copy link
Contributor

@zhijunfu Sorry, the other PR caused a merge conflict, can you rebase/merge so we can get this PR in? Thanks :)

@zhijunfu
Copy link
Contributor Author

@pcmoritz Sorry for the long delay in reply, I just rebased the change on top of latest code. The build failures w/ parquet and tool-chain should be unrelated with the change.


/// The file descriptor used to push notifications to client. This is only valid
/// if client subscribes to plasma store. -1 indicates invalid.
int notification_fd;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to the content of this patch: could we regularize the variable name convention in this header and elsewhere in Plasma? Would help with readability

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that makes a lot of sense:)

@zhijunfu
Copy link
Contributor Author

zhijunfu commented Jun 6, 2018

Re-triggered the build and it passed this time.

Copy link
Member

@wesm wesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Merging this, and also opened https://issues.apache.org/jira/browse/ARROW-2690

@wesm wesm closed this in 8156e25 Jun 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants