Skip to content

Raise warning for unpickable local function (#547) #80232

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Raise warning for unpickable local function (#547) #80232

wants to merge 1 commit into from

Conversation

ejguan
Copy link
Contributor

@ejguan ejguan commented Jun 24, 2022

Summary:
X-link: meta-pytorch/data#547

Fixes meta-pytorch/data#538

  • Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
  • The inner function from functools.partial object is extracted as well for validation
  • Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'

This Diff also fixes the Error introduced by #79344

Test Plan:
CI on PyTorch and TorchData
Manually validated the tests from TorchVision

Differential Revision: D37417556

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jun 24, 2022 •
edited
Loading

🔗 Helpful links

✅ No Failures (0 Pending)

As of commit 373109c (more details on the Dr. CI page):

Expand to see more

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37417556

ejguan added a commit to ejguan/data that referenced this pull request Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
0cc4106 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: meta-pytorch#547

Fixes meta-pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: 7213ee84b34092e0c2cf293ff8bf1dc56659fc83
ejguan added a commit to ejguan/data that referenced this pull request Jun 24, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
16f963e 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: meta-pytorch#547

Fixes meta-pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Differential Revision: D37417556

fbshipit-source-id: c17e475bde703f2f55af140f2155d41efb45f049
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37417556

@ejguan ejguan force-pushed the export-D37417556 branch from 909ada0 to f4a19a7 Compare June 24, 2022 19:08
NivekT
NivekT approved these changes Jun 24, 2022
View reviewed changes
Copy link
Contributor

@NivekT NivekT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ejguan @facebook-github-bot
Raise warning for unpickable local function (pytorch#80232)
373109c 
Summary:
Pull Request resolved: pytorch#80232

X-link: meta-pytorch/data#547

Fixes meta-pytorch/data#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch#79344

Test Plan:
```
buck test //caffe2/test:datapipe
buck test //pytorch/data/test:tests
```
Tested in OSS
```
# PT
pytest test/test_datapipe.py -v
# TD
pytest test/test_iterdatapipe.py -v
pytest test/test_mapdatapipe.py -v
pytest test/test_serialization.py -v
# TV
pytest test/test_prototype_builtin_datasets.py -v
```

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 6fae4059285b8c742feda739cc5fe590b2e20c5e
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D37417556

@ejguan ejguan force-pushed the export-D37417556 branch from f4a19a7 to 373109c Compare June 27, 2022 16:18
ejguan added a commit to ejguan/data that referenced this pull request Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
ad3474e 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: meta-pytorch#547

Fixes meta-pytorch#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 6babb73a7daad470ec93c3bd0a0c08d71849d3c8
@ejguan ejguan mentioned this pull request Jun 27, 2022
Closed
@facebook-github-bot
Copy link
Contributor

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

@pytorchmergebot
Copy link
Collaborator

@pytorchbot successfully started a merge job. Check the current status here

@github-actions
Copy link
Contributor

Hey @ejguan.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

facebook-github-bot pushed a commit to meta-pytorch/data that referenced this pull request Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
14b2291 
Summary:
X-link: pytorch/pytorch#80232

Pull Request resolved: #547

Fixes #538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by pytorch/pytorch#79344

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 388d04004da43aaf14729cbe465a9247bef6c780
facebook-github-bot pushed a commit that referenced this pull request Jun 27, 2022
@ejguan @facebook-github-bot
Raise warning for unpickable local function (#80232)
3d96af7 
Summary:
Pull Request resolved: #80232

X-link: meta-pytorch/data#547

Fixes meta-pytorch/data#538
- Improve the validation function to raise warning about unpickable function when either lambda or local function is provided to DataPipe.
- The inner function from functools.partial object is extracted as well for validation
- Mimic the behavior of pickle module for local lambda function: It would only raise Error for the local function rather than lambda function. So, we will raise warning about local function not lambda function.
```py

>>> import pickle
>>> def fn():
...     lf = lambda x: x
...     pickle.dumps(lf)
>>> pickle.dumps(fn)
AttributeError: Can't pickle local object 'fn.<locals>.<lambda>'
```

This Diff also fixes the Error introduced by #79344

Test Plan:
```
buck test //caffe2/test:datapipe
buck test //pytorch/data/test:tests
```
Tested in OSS
```
# PT
pytest test/test_datapipe.py -v
# TD
pytest test/test_iterdatapipe.py -v
pytest test/test_mapdatapipe.py -v
pytest test/test_serialization.py -v
# TV
pytest test/test_prototype_builtin_datasets.py -v
```

Reviewed By: NivekT

Differential Revision: D37417556

fbshipit-source-id: 388d04004da43aaf14729cbe465a9247bef6c780
@ejguan ejguan added release notes: dataloader release notes category topic: improvements topic category labels Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@NivekT NivekT NivekT approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cla signed fb-exported Merged release notes: dataloader release notes category topic: improvements topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Warn about pickle-ablity when using dp.map(some_local_function) ?

4 participants

@ejguan @facebook-github-bot @pytorchmergebot @NivekT