-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support datasets library #7
Conversation
cls_doc = getattr(callable_, "__doc__", None) | ||
return "" if cls_doc is None else cls_doc | ||
|
||
|
||
def append_parent_docstr(cls: Type = None, parent_id: int = 0): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optionally, we can adopt the common convention as "klass" for such naming.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally don't like "klass". Moreover, type hint reveal its type and cls is also commonly used including IDE stubs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cls
is common keyword for classmethods that's why I suggested a change though it's kinda optional ofc.
|
||
|
||
def append_callable_docstr( | ||
cls: Type = None, callable_: Union[Type, Callable] = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 above
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 above (my comment)
…DatasetReader make DatasetReader their common base class.
- Drop the support for the regular file based dataset readers. - Introduce DatasetLoader class that wraps the load_dataset function from the datasets library. - Introduce TransformerDataProcessor class that performs the previous dataset reader's most job.
a4c01ac
to
766e40e
Compare
Replace the dataset reader classes with data processors and a dataset loader class. TransformerDataProcessor contains the optimizer and is responsible from tokenization and pre-processing e.g. append special tokens and truncation etc. The DatasetLoader is currently a single class that can load datasets from the datasets library through the load_dataset function. It also processes the datasets using its TransformerDataProcessor attribute and can directly provide IndexedDataset to the TransformerTrainer.
Fixes:
#2
#5
#6