Add not-in-place implementations for several dataset transforms #1883
Conversation
src/datasets/arrow_dataset.py
Outdated
| """ | ||
| dataset = copy.deepcopy(self) | ||
| dataset._fingerprint = new_fingerprint | ||
| dataset.flatten_(max_depth=max_depth) |
There was a problem hiding this comment.
flatten_ already updates the fingerprint so we're updating the fingerprint twice here.
We could simply copy paste the code from flatten_. I think it's ok since we may deprecate flatten_ at one point, and the code is short and straightforward.
This also applies for the other transforms. Let me know what you think
There was a problem hiding this comment.
I also added a @deprecated decorator that emits a DeprecationWarning when calling the methods
There was a problem hiding this comment.
Nice thanks !
Although usually deprecation warnings are only emitted once, maybe we can just give an id parameter to the deprecated decorator so that the second time a deprecated function is called we can say that the warning has already been emitted and therefore keep it silent ? We can use a dictionary to keep track of already emitted deprecation warning as in transformers.
|
@lhoestq I am not sure how to test |
|
I can take a look at dictionary_encode_column tomorrow. |
Warn only once and add a replaced_by arg Refactor Use logger from logging utils
SBrandeis
force-pushed
the
rename_column
branch
from
47f3495
to
0a3b53a
Compare
|
Now let's update the documentation to use the new methods x) |
Should we deprecate in-place versions of such methods?