Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bindings/python): read APIs return memoryview instead of bytes to avoid copy #3310

Merged
merged 1 commit into from
Oct 16, 2023

Conversation

messense
Copy link
Member

Before

In [1]: import opendal

In [2]: op = opendal.Operator("memory")

In [3]: op.write("test", bytes(10 * 1024 * 1024))

In [4]: %timeit op.read("test")
627 µs ± 3.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

After

In [1]: import opendal

In [2]: op = opendal.Operator("memory")

In [3]: op.write("test", bytes(10 * 1024 * 1024))

In [4]: %timeit op.read("test")
291 µs ± 761 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Unfortunately better performance comes with a price, we're no longer able to build abi3 wheels which means we need to build and publish one wheel per Python version, OS and CPU architecture.

We can remove the unsafe once PyO3/pyo3#3514 is released.

…s` to avoid copy

Before

```python
In [1]: import opendal

In [2]: op = opendal.Operator("memory")

In [3]: op.write("test", bytes(10 * 1024 * 1024))

In [4]: %timeit op.read("test")
627 µs ± 3.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```

After

```python
In [1]: import opendal

In [2]: op = opendal.Operator("memory")

In [3]: op.write("test", bytes(10 * 1024 * 1024))

In [4]: %timeit op.read("test")
291 µs ± 761 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
```
@messense messense marked this pull request as ready for review October 16, 2023 13:20
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@Xuanwo Xuanwo merged commit 5d0194b into apache:main Oct 16, 2023
36 checks passed
@messense messense deleted the python-memoryview branch October 17, 2023 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants