Skip to content

Commit

Permalink
GH-31506: [Python] Address docstrings in Streams and File Access (Fac…
Browse files Browse the repository at this point in the history
…tory Functions) (#33609)

# Which issue does this PR close?
Closes #31506

# Rationale for this change
Ensure docstrings for [Streams and File Access](https://arrow.apache.org/docs/python/api/files.html) - Factory Functions - have an Examples section:

# What changes are included in this PR?
Added docstring examples for:
- `pyarrow.input_stream`
- `pyarrow.output_stream`
- `pyarrow.memory_map`
- `pyarrow.create_memory_map` 

 Also pytest argument ` --disable-warnings` for `pytest-cython` is added, see lgpage/pytest-cython#3: 

```python
=============================== warnings summary ===============================
opt/conda/envs/arrow/lib/python3.9/site-packages/pytest_cython/plugin.py:57
  /opt/conda/envs/arrow/lib/python3.9/site-packages/pytest_cython/plugin.py:57: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to DoctestModule is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return DoctestModule.from_parent(parent, fspath=path)
```

# Are these changes tested?
Yes, locally with `pytest --doctest-cython --disable-warnings pyarrow` and on the CI with `Python / AMD64 Conda Python 3.9 Sphinx & Numpydoc ` build.

# Are there any user-facing changes?
No.
* Closes: #31506

Authored-by: Alenka Frim <frim.alenka@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
  • Loading branch information
AlenkaF committed Jan 17, 2023
1 parent 5580f27 commit ef98a97
Show file tree
Hide file tree
Showing 2 changed files with 104 additions and 1 deletion.
5 changes: 4 additions & 1 deletion docker-compose.yml
Expand Up @@ -1101,7 +1101,10 @@ services:
LANG: "C.UTF-8"
BUILD_DOCS_CPP: "ON"
BUILD_DOCS_PYTHON: "ON"
PYTEST_ARGS: "--doctest-modules --doctest-cython"
# GH-31506/GH-33609: Remove --disable-warnings once
# https://github.com/lgpage/pytest-cython/issues/24 is resolved
# and a new version that includes the fix is released.
PYTEST_ARGS: "--doctest-modules --doctest-cython --disable-warnings"
volumes: *conda-volumes
command:
["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
Expand Down
100 changes: 100 additions & 0 deletions python/pyarrow/io.pxi
Expand Up @@ -943,6 +943,20 @@ def memory_map(path, mode='r'):
Returns
-------
mmap : MemoryMappedFile
Examples
--------
Reading from a memory map without any memory allocation or copying:
>>> import pyarrow as pa
>>> with pa.output_stream('example_mmap.txt') as stream:
... stream.write(b'Constructing a buffer referencing the mapped memory')
...
51
>>> with pa.memory_map('example_mmap.txt') as mmap:
... mmap.read_at(6,45)
...
b'memory'
"""
_check_is_file(path)

Expand Down Expand Up @@ -971,6 +985,18 @@ def create_memory_map(path, size):
Returns
-------
mmap : MemoryMappedFile
Examples
--------
Create a file with a memory map:
>>> import pyarrow as pa
>>> with pa.create_memory_map('example_mmap_create.dat', 27) as mmap:
... mmap.write(b'Create a memory-mapped file')
... mmap.read_at(10, 9)
...
27
b'memory-map'
"""
return MemoryMappedFile.create(path, size)

Expand Down Expand Up @@ -2218,6 +2244,40 @@ def input_stream(source, compression='detect', buffer_size=None):
buffer_size : int, default None
If None or 0, no buffering will happen. Otherwise the size of the
temporary read buffer.
Examples
--------
Create a readable BufferReader (NativeFile) from a Buffer or a memoryview object:
>>> import pyarrow as pa
>>> buf = memoryview(b"some data")
>>> with pa.input_stream(buf) as stream:
... stream.read(4)
...
b'some'
Create a readable OSFile (NativeFile) from a string or file path:
>>> import gzip
>>> with gzip.open('example.gz', 'wb') as f:
... f.write(b'some data')
...
9
>>> with pa.input_stream('example.gz') as stream:
... stream.read()
...
b'some data'
Create a readable PythonFile (NativeFile) from a a Python file object:
>>> with open('example.txt', mode='w') as f:
... f.write('some text')
...
9
>>> with pa.input_stream('example.txt') as stream:
... stream.read(6)
...
b'some t'
"""
cdef NativeFile stream

Expand Down Expand Up @@ -2270,6 +2330,46 @@ def output_stream(source, compression='detect', buffer_size=None):
buffer_size : int, default None
If None or 0, no buffering will happen. Otherwise the size of the
temporary write buffer.
Examples
--------
Create a writable NativeFile from a pyarrow Buffer:
>>> import pyarrow as pa
>>> data = b"buffer data"
>>> empty_obj = bytearray(11)
>>> buf = pa.py_buffer(empty_obj)
>>> with pa.output_stream(buf) as stream:
... stream.write(data)
...
11
>>> with pa.input_stream(buf) as stream:
... stream.read(6)
...
b'buffer'
or from a memoryview object:
>>> buf = memoryview(empty_obj)
>>> with pa.output_stream(buf) as stream:
... stream.write(data)
...
11
>>> with pa.input_stream(buf) as stream:
... stream.read()
...
b'buffer data'
Create a writable NativeFile from a string or file path:
>>> with pa.output_stream('example_second.txt') as stream:
... stream.write(b'Write some data')
...
15
>>> with pa.input_stream('example_second.txt') as stream:
... stream.read()
...
b'Write some data'
"""
cdef NativeFile stream

Expand Down

0 comments on commit ef98a97

Please sign in to comment.