Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-3116: [Plasma] Add "ls" to object store #2470

Closed
wants to merge 22 commits into from

Conversation

@pcmoritz
Copy link
Contributor

pcmoritz commented Aug 25, 2018

This adds plasma_client.list to the plasma client API.

It can be used like so:

  import pyarrow.plasma as plasma
  import time

  client = plasma.connect("/tmp/plasma", "", 0)

  client.put("hello, world")
  # Sleep a little so we get different creation times
  time.sleep(2)
  client.put("another object")
  # Create an object that is not sealed yet
  object_id = plasma.ObjectID.from_random()
  client.create(object_id, 100)
  print(client.list())

  >>> {ObjectID(4cba8f80c54c6d265b46c2cdfcee6e32348b12be): {'construct_duration': 0,
  >>>  'create_time': 1535223642,
  >>>  'data_size': 460,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 0,
  >>>  'state': 'sealed'},
  >>> ObjectID(a7598230b0c26464c9d9c99ae14773ee81485428): {'construct_duration': 0,
  >>>  'create_time': 1535223644,
  >>>  'data_size': 460,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 0,
  >>>  'state': 'sealed'},
  >>> ObjectID(e603ab0c92098ebf08f90bfcea33ff98f6476870): {'construct_duration': -1,
  >>>  'create_time': 1535223644,
  >>>  'data_size': 100,
  >>>  'metadata_size': 0,
  >>>  'ref_count': 1,
  >>>  'state': 'created'}}
@pcmoritz

This comment has been minimized.

Copy link
Contributor Author

pcmoritz commented Aug 25, 2018

This is now ready to give feedback or merge (I marked it as experimental so it can be evolved).

fix
@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Aug 25, 2018

Codecov Report

❗️ No coverage uploaded for pull request base (master@06e7a1b). Click here to learn what that means.
The diff coverage is 82.35%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #2470   +/-   ##
=========================================
  Coverage          ?   87.47%           
=========================================
  Files             ?      246           
  Lines             ?    43412           
  Branches          ?        0           
=========================================
  Hits              ?    37975           
  Misses            ?     5437           
  Partials          ?        0
Impacted Files Coverage Δ
cpp/src/plasma/client.h 100% <ø> (ø)
cpp/src/plasma/protocol.h 100% <ø> (ø)
cpp/src/plasma/common.h 100% <ø> (ø)
cpp/src/plasma/plasma.h 100% <ø> (ø)
cpp/src/plasma/client.cc 84.43% <100%> (ø)
python/pyarrow/tests/test_plasma.py 96.6% <100%> (ø)
cpp/src/plasma/store.cc 89.14% <50%> (ø)
cpp/src/plasma/protocol.cc 95.36% <60%> (ø)
python/pyarrow/_plasma.pyx 61.57% <95%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06e7a1b...5ff4e35. Read the comment docs.

Copy link
Contributor

robertnishihara left a comment

Looks good to me!

x, _, _ = create_object(plasma_client, 3, metadata_size=0, seal=False)
t2 = time.time()
l6 = plasma_client.list()
assert math.floor(t1) <= l6[x]["create_time"] <= math.ceil(t2)

This comment has been minimized.

Copy link
@robertnishihara

robertnishihara Aug 25, 2018

Contributor

There's some chance this could be flaky due to non-monotonic clocks.

This comment has been minimized.

Copy link
@pcmoritz

pcmoritz Aug 26, 2018

Author Contributor

Yeah, I'd say let's keep this clock for now, since it needs to be consistent with the timestamp from the store, where it makes more sense to use the system time than a monotonic clock. If it turns out to be flaky, we can find a different solution.

@pcmoritz pcmoritz closed this in 628b74b Aug 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.