Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fx] introduce __fx_create_arg__ dunder method for controlling custom classes are handled as node args #61780

Closed
wants to merge 1 commit into from
Closed

[fx] introduce __fx_create_arg__ dunder method for controlling custom classes are handled as node args #61780

wants to merge 1 commit into from

Conversation

bradleyhd
Copy link
Contributor

Summary: These changes would allow objects to control how they are handled when they are an argument to a torch.fx call_module node from within their source. Previously, we have been using a custom Tracer with an overridden create_arg() method and branching based on class name to handle args that are unusual (data classes, etc).

Differential Revision: D27976120

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Jul 16, 2021 •
edited

💊 CI failures summary and remediations

As of commit 2048f40 (more details on the Dr. CI page and at hud.pytorch.org/pr/61780):

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See GitHub Actions build Windows CI (pytorch-win-vs2019-cuda10-cudnn7-py3) / test (default, 1, 2, windows.8xlarge.nvidia.gpu) (1/3)

Step: "Store PyTorch Test Reports" (full log | diagnosis details | 🔁 rerun)

2021-07-21T16:33:47.3793932Z ls: cannot access ...d/win_tmp/ci_scripts/*': No such file or directory 
2021-07-21T16:33:47.3021057Z + PYTORCH_FINAL_PACKAGE_DIR=/c/1052879422/build-results/
2021-07-21T16:33:47.3102485Z ++ cygpath -w /c/1052879422/build-results/
2021-07-21T16:33:47.3239520Z + PYTORCH_FINAL_PACKAGE_DIR_WIN='C:\1052879422\build-results\'
2021-07-21T16:33:47.3240708Z + export PYTORCH_FINAL_PACKAGE_DIR_WIN
2021-07-21T16:33:47.3241714Z + export PYTORCH_TEST_SKIP_NOARCH=1
2021-07-21T16:33:47.3242572Z + PYTORCH_TEST_SKIP_NOARCH=1
2021-07-21T16:33:47.3243349Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/build/win_tmp/build/torch
2021-07-21T16:33:47.3440818Z + CI_SCRIPTS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/build/win_tmp/ci_scripts
2021-07-21T16:33:47.3442035Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/build/win_tmp/ci_scripts
2021-07-21T16:33:47.3695010Z ++ ls '/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/build/win_tmp/ci_scripts/*'
2021-07-21T16:33:47.3793932Z ls: cannot access '/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/build/win_tmp/ci_scripts/*': No such file or directory
2021-07-21T16:33:47.3797117Z + '[' -n '' ']'
2021-07-21T16:33:47.3799107Z + export SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/.jenkins/pytorch/win-test-helpers
2021-07-21T16:33:47.3800751Z + SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879422/.jenkins/pytorch/win-test-helpers
2021-07-21T16:33:47.3801605Z + IN_PULL_REQUEST=
2021-07-21T16:33:47.3801967Z + '[' -n '' ']'
2021-07-21T16:33:47.3802612Z + [[ pytorch-win-vs2019-cuda10-cudnn7-py3 == *cuda11* ]]
2021-07-21T16:33:47.3803546Z + run_tests
2021-07-21T16:33:47.3804351Z + for path in '/c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe' /c/Windows/System32/nvidia-smi.exe
2021-07-21T16:33:47.3805338Z + [[ -x /c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe ]]
2021-07-21T16:33:47.3806143Z + '/c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe'

See GitHub Actions build Windows CI (pytorch-win-vs2019-cpu-py3) / test (default, 2, 2, windows.4xlarge) (2/3)

Step: "Store PyTorch Test Reports" (full log | diagnosis details | 🔁 rerun)

2021-07-21T16:24:58.6299721Z test_add_done_ca...arg() takes 0 positional arguments but 1 was given 
2021-07-21T16:24:58.6255435Z   c:\jenkins\miniconda3\lib\site-packages\coverage\execfile.py(247): run
2021-07-21T16:24:58.6256108Z   c:\jenkins\miniconda3\lib\site-packages\coverage\cmdline.py(746): do_run
2021-07-21T16:24:58.6256784Z   c:\jenkins\miniconda3\lib\site-packages\coverage\cmdline.py(588): command_line
2021-07-21T16:24:58.6257469Z   c:\jenkins\miniconda3\lib\site-packages\coverage\cmdline.py(871): main
2021-07-21T16:24:58.6258103Z   C:\Jenkins\Miniconda3\Scripts\coverage.exe\__main__.py(7): <module>
2021-07-21T16:24:58.6258645Z   c:\jenkins\miniconda3\lib\runpy.py(87): _run_code
2021-07-21T16:24:58.6259155Z   c:\jenkins\miniconda3\lib\runpy.py(194): _run_module_as_main
2021-07-21T16:24:58.6259454Z 
2021-07-21T16:24:58.6259688Z ok (0.000s)
2021-07-21T16:24:58.6284913Z   test_add_done_callback_maintains_callback_order (__main__.TestFuture) ... ok (0.016s)
2021-07-21T16:24:58.6299721Z   test_add_done_callback_no_arg_error_is_ignored (__main__.TestFuture) ... [E pybind_utils.h:200] Got the following error when running the callback: TypeError: no_arg() takes 0 positional arguments but 1 was given
2021-07-21T16:24:58.6300530Z ok (0.000s)
2021-07-21T16:24:58.6328623Z   test_add_done_callback_simple (__main__.TestFuture) ... ok (0.000s)
2021-07-21T16:24:58.6384680Z   test_chained_then (__main__.TestFuture) ... ok (0.000s)
2021-07-21T16:24:58.7568399Z   test_collect_all (__main__.TestFuture) ... ok (0.129s)
2021-07-21T16:24:58.7587853Z   test_done (__main__.TestFuture) ... ok (0.001s)
2021-07-21T16:24:58.7620509Z   test_done_exception (__main__.TestFuture) ... ok (0.000s)
2021-07-21T16:24:58.7661753Z   test_interleaving_then_and_add_done_callback_maintains_callback_order (__main__.TestFuture) ... ok (0.000s)
2021-07-21T16:24:58.7688694Z   test_interleaving_then_and_add_done_callback_propagates_error (__main__.TestFuture) ... [E pybind_utils.h:200] Got the following error when running the callback: ValueError: Expected error
2021-07-21T16:24:58.7689379Z 
2021-07-21T16:24:58.7689620Z At:

See GitHub Actions build Windows CI (pytorch-win-vs2019-cpu-py3) / test (default, 1, 2, windows.4xlarge) (3/3)

Step: "Store PyTorch Test Reports" (full log | diagnosis details | 🔁 rerun)

2021-07-21T15:46:40.3073515Z ls: cannot access ...d/win_tmp/ci_scripts/*': No such file or directory 
2021-07-21T15:46:40.2011501Z + PYTORCH_FINAL_PACKAGE_DIR=/c/1052879421/build-results/
2021-07-21T15:46:40.2070813Z ++ cygpath -w /c/1052879421/build-results/
2021-07-21T15:46:40.2165897Z + PYTORCH_FINAL_PACKAGE_DIR_WIN='C:\1052879421\build-results\'
2021-07-21T15:46:40.2166440Z + export PYTORCH_FINAL_PACKAGE_DIR_WIN
2021-07-21T15:46:40.2166855Z + export PYTORCH_TEST_SKIP_NOARCH=1
2021-07-21T15:46:40.2167243Z + PYTORCH_TEST_SKIP_NOARCH=1
2021-07-21T15:46:40.2167819Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/build/win_tmp/build/torch
2021-07-21T15:46:40.2567701Z + CI_SCRIPTS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/build/win_tmp/ci_scripts
2021-07-21T15:46:40.2568591Z + mkdir -p /c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/build/win_tmp/ci_scripts
2021-07-21T15:46:40.2752828Z ++ ls '/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/build/win_tmp/ci_scripts/*'
2021-07-21T15:46:40.3073515Z ls: cannot access '/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/build/win_tmp/ci_scripts/*': No such file or directory
2021-07-21T15:46:40.3076382Z + '[' -n '' ']'
2021-07-21T15:46:40.3077629Z + export SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/.jenkins/pytorch/win-test-helpers
2021-07-21T15:46:40.3078646Z + SCRIPT_HELPERS_DIR=/c/actions-runner/_work/pytorch/pytorch/pytorch-1052879421/.jenkins/pytorch/win-test-helpers
2021-07-21T15:46:40.3079277Z + IN_PULL_REQUEST=
2021-07-21T15:46:40.3079545Z + '[' -n '' ']'
2021-07-21T15:46:40.3079965Z + [[ pytorch-win-vs2019-cpu-py3 == *cuda11* ]]
2021-07-21T15:46:40.3080386Z + run_tests
2021-07-21T15:46:40.3080990Z + for path in '/c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe' /c/Windows/System32/nvidia-smi.exe
2021-07-21T15:46:40.3081747Z + [[ -x /c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe ]]
2021-07-21T15:46:40.3082353Z + '/c/Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe'

🚧 1 ongoing upstream failure:

These were probably caused by upstream breakages that are not fixed yet.

🚧 2 fixed upstream failures:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Preview docs built from this PR

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D27976120

@bradleyhd bradleyhd requested review from jamesr66a and suo July 16, 2021 18:52
suo
suo approved these changes Jul 16, 2021
View reviewed changes
Copy link
Member

@suo suo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! This approach feels more Pythonic to me (using special methods to customize behavior through a protocol vs. overriding the behavior the tracer) and increases the composability of custom tracers.

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D27976120

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D27976120

@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D27976120

@bradleyhd @facebook-github-bot
[fx] introduce __fx_create_arg__ dunder method for controlling cust…
2048f40 
…om classes are handled as node args (pytorch#61780)

Summary:
Pull Request resolved: pytorch#61780

These changes would allow objects to control how they are handled when they are an argument to a torch.fx call_module node from within their source. Previously, we have been using a custom Tracer with an overridden create_arg() method and branching based on class name to handle args that are unusual (data classes, etc).

Reviewed By: suo, houseroad

Differential Revision: D27976120

fbshipit-source-id: e10a27f48af9c18626f8b65508cd12660aa56e94
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D27976120

@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 8880f3d.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@suo suo suo approved these changes

@jamesr66a jamesr66a Awaiting requested review from jamesr66a

Assignees
No one assigned
Labels
cla signed fb-exported Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bradleyhd @facebook-github-bot @suo