-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DataPipe] Update docstring for functional form of DataPipes #100446
Changes from all commits
24e7e3d
6278c9d
679156b
4b86a3a
1a32296
05aa283
1ce0ca1
fc82cb9
b9c79c1
8457423
0c6a6c8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,6 +5,7 @@ | |
import os | ||
import os.path | ||
import pickle | ||
import pydoc | ||
import random | ||
import sys | ||
import tempfile | ||
|
@@ -831,6 +832,43 @@ def _fn3(x): | |
with self.assertRaises((pickle.PicklingError, AttributeError)): | ||
pickle.dumps(datapipe) | ||
|
||
def test_docstring(self): | ||
""" | ||
Ensure functional form of IterDataPipe has the correct docstring from | ||
the class form. | ||
|
||
Regression test for https://github.com/pytorch/data/issues/792. | ||
""" | ||
input_dp = dp.iter.IterableWrapper(range(10)) | ||
|
||
for dp_funcname in [ | ||
"batch", | ||
"collate", | ||
"concat", | ||
"demux", | ||
"filter", | ||
"fork", | ||
"map", | ||
"mux", | ||
"read_from_stream", | ||
# "sampler", | ||
"shuffle", | ||
"unbatch", | ||
"zip", | ||
]: | ||
if sys.version_info >= (3, 9): | ||
docstring = pydoc.render_doc( | ||
thing=getattr(input_dp, dp_funcname), forceload=True | ||
) | ||
elif sys.version_info < (3, 9): | ||
# pydoc works differently on Python 3.8, see | ||
# https://docs.python.org/3/whatsnew/3.9.html#pydoc | ||
docstring = getattr(input_dp, dp_funcname).__doc__ | ||
|
||
assert f"(functional name: ``{dp_funcname}``)" in docstring | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Yes I see the error, though it seems to be only for certain combinations of the build matrix which is weird. Traceback at https://github.com/pytorch/pytorch/actions/runs/4929875273/jobs/8810511389#step:13:628:
Logs aren't very helpful, let me try and track down the issue. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Merging in the changes from #100503 to see if it helps. I can't seem to reproduce this locally on my setup, and not quite sure what the shard/num_shards config means in the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Have tried using __________________ TestFunctionalIterDataPipe.test_docstring ___________________
Traceback (most recent call last):
File "test_datapipe.py", line 863, in test_docstring
assert f"(functional name: ``{dp_funcname}``)" in docstring
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_datapipe.py", line 868, in test_docstring
raise ValueError(dp_funcname, "IterDataPipe docstring incorrect")
ValueError: ('batch', 'IterDataPipe docstring incorrect')
----------------------------- Captured stdout call -----------------------------
***Begin docstring for batch
Python Library Documentation: partial in module torch.utils.data.datapipes.iter.grouping object
class p�pa�ar�rt�ti�ia�al�l(builtins.object)
| partial(func, *args, **keywords) - new function with partial application
| of the given arguments and keywords.
|
| Methods defined here:
|
| _�__�_c�ca�al�ll�l_�__�_(self, /, *args, **kwargs)
| Call self as a function.
|
| _�__�_d�de�el�la�at�tt�tr�r_�__�_(self, name, /)
| Implement delattr(self, name).
|
| _�__�_g�ge�et�ta�at�tt�tr�ri�ib�bu�ut�te�e_�__�_(self, name, /)
| Return getattr(self, name).
|
| _�__�_r�re�ed�du�uc�ce�e_�__�_(...)
| Helper for pickle.
|
| _�__�_r�re�ep�pr�r_�__�_(self, /)
| Return repr(self).
|
| _�__�_s�se�et�ta�at�tt�tr�r_�__�_(self, name, value, /)
| Implement setattr(self, name, value).
|
| _�__�_s�se�et�ts�st�ta�at�te�e_�__�_(...)
|
| ----------------------------------------------------------------------
| Static methods defined here:
|
| _�__�_n�ne�ew�w_�__�_(*args, **kwargs) from builtins.type
| Create and return a new object. See help(type) for accurate signature.
|
| ----------------------------------------------------------------------
| Data descriptors defined here:
|
| _�__�_d�di�ic�ct�t_�__�_
|
| a�ar�rg�gs�s
| tuple of arguments to future partial calls
|
| f�fu�un�nc�c
| function object to use in future partial calls
|
| k�ke�ey�yw�wo�or�rd�ds�s
| dictionary of keyword arguments to future partial calls
***End docstring for batch
__________________ TestFunctionalIterDataPipe.test_docstring ___________________
Traceback (most recent call last):
File "test_datapipe.py", line 863, in test_docstring
assert f"(functional name: ``{dp_funcname}``)" in docstring
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test_datapipe.py", line 868, in test_docstring
raise ValueError(dp_funcname, "IterDataPipe docstring incorrect")
ValueError: ('batch', 'IterDataPipe docstring incorrect') Not sure why the docstring output is repeated in some parts 😕 I could xfail those tests for now and investigate them later? |
||
assert "Args:" in docstring | ||
assert "Example:" in docstring or "Examples:" in docstring | ||
|
||
def test_iterable_wrapper_datapipe(self): | ||
|
||
input_ls = list(range(10)) | ||
|
@@ -1894,6 +1932,34 @@ def _fn1(x): | |
with self.assertRaises((pickle.PicklingError, AttributeError)): | ||
pickle.dumps(datapipe) | ||
|
||
def test_docstring(self): | ||
""" | ||
Ensure functional form of MapDataPipe has the correct docstring from | ||
the class form. | ||
|
||
Regression test for https://github.com/pytorch/data/issues/792. | ||
""" | ||
input_dp = dp.map.SequenceWrapper(range(10)) | ||
|
||
for dp_funcname in [ | ||
"batch", | ||
"concat", | ||
"map", | ||
"shuffle", | ||
"zip", | ||
]: | ||
if sys.version_info >= (3, 9): | ||
docstring = pydoc.render_doc( | ||
thing=getattr(input_dp, dp_funcname), forceload=True | ||
) | ||
elif sys.version_info < (3, 9): | ||
# pydoc works differently on Python 3.8, see | ||
# https://docs.python.org/3/whatsnew/3.9.html#pydoc | ||
docstring = getattr(input_dp, dp_funcname).__doc__ | ||
assert f"(functional name: ``{dp_funcname}``)" in docstring | ||
assert "Args:" in docstring | ||
assert "Example:" in docstring or "Examples:" in docstring | ||
|
||
def test_sequence_wrapper_datapipe(self): | ||
seq = list(range(10)) | ||
input_dp = dp.map.SequenceWrapper(seq) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Noticed that
SamplerIterDataPipe
is missing the functional form atpytorch/torch/utils/data/datapipes/iter/combinatorics.py
Lines 17 to 25 in 8f6951c
Should add the
@functional_datapipe('sample')
decorator in a separate PR?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we left it out for a reason, we can add the functional form for
sampler
later if there is demand for me