Add python stack tracing option on on-demand flow

[slgong-fb]

Summary:
Changes:

1. add an option in Config; can use 'PYTHON_STACK_TRACE=true' option (via .conf)
2. deliver PYTHON_STACK_TRACE value to kineto_client_interface start()
3. abstract class also changed.

Test Plan:

1. launch a python test case with the following command for on-demand flow:
   echo -e "PYTHON_STACK_TRACE=true" > /tmp/scott_kineto.conf && dyno gputrace --gputrace_duration 300ms --gpuconf /tmp/scott_kineto.conf

2. Then, we can see with_stack enabled as intended from output log:
   INFO:2022-06-27 15:00:16 1009443:1011716 kineto_client_interface.cpp:22] withStack : 1

Differential Revision: D37410204

[facebook-github-bot]

🔗 Helpful links

• 🧪  See artifacts and rendered test results at hud.pytorch.org/pr/80919
• 📄  Preview Python docs built from this PR
• 📄  Preview C++ docs built from this PR
• ❓Need help or want to give feedback on the CI? Visit our office hours

✅ No Failures (0 Pending)

As of commit 7f411f3 (more details on the Dr. CI page):

Expand to see more

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[robieta]

Approving, with an action item to clean up the interface in a follow up PR.

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

This pull request was exported from Phabricator. Differential Revision: D37410204

[facebook-github-bot]

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

[pytorchmergebot]

@pytorchbot successfully started a merge job. Check the current status here

[github-actions]

Hey @slgong-fb.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

[janeyx99]

@pytorchbot revert -m "Sorry, reverting as this broke buck build/test https://hud.pytorch.org/pytorch/pytorch/commit/f50a248a5eacb9a9aa475a9e610486aea136e4f5" -c nosignal

Could you please add ciflow/periodic to the reland PR to capture the signal?

[pytorchmergebot]

@pytorchbot successfully started a revert job. Check the current status here

[pytorchmergebot]

@slgong-fb your PR has been successfully reverted.

[kit1980]

@pytorchbot revert -m "Sorry, reverting as this broke buck build/test https://hud.pytorch.org/pytorch/pytorch/commit/f50a248a5eacb9a9aa475a9e610486aea136e4f5" -c nosignal

To be clear, buck build helped to catch an actual issue with the code: void start(bool withStack) override has one bool arg, while the parent's start has no arguments, so you can't override.

[chaekit]

the buck build failure is legit but not sure why linux-bionic-py3.7-clang9

ERROR [0.001s]: test_bce_with_logits_has_correct_forward_grad (__main__.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_nn.py", line 9557, in test_bce_with_logits_has_correct_forward_grad
    (output, target), check_forward_ad=True)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 3148, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 1418, in gradcheck
    return _gradcheck_helper(**args)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 1427, in _gradcheck_helper
    func_out = func(*tupled_inputs)
  File "test_nn.py", line 9556, in <lambda>
    gradcheck(lambda self, target: nn.BCEWithLogitsLoss(reduction=reduction)(self, target),
  File "test_nn.py", line 9556, in <lambda>
    gradcheck(lambda self, target: nn.BCEWithLogitsLoss(reduction=reduction)(self, target),
  File "/var/lib/jenkins/torchdynamo/torchdynamo/eval_frame.py", line 93, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 655, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 277, in __call__
    raise e
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 267, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given

this is strictly profiler change and does not touch any PyTorch core. also it passed in CI before landing

[robieta]

the buck build failure is legit but not sure why linux-bionic-py3.7-clang9

ERROR [0.001s]: test_bce_with_logits_has_correct_forward_grad (__main__.TestNN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_nn.py", line 9557, in test_bce_with_logits_has_correct_forward_grad
    (output, target), check_forward_ad=True)
  File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 3148, in gradcheck
    return torch.autograd.gradcheck(fn, inputs, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 1418, in gradcheck
    return _gradcheck_helper(**args)
  File "/opt/conda/lib/python3.7/site-packages/torch/autograd/gradcheck.py", line 1427, in _gradcheck_helper
    func_out = func(*tupled_inputs)
  File "test_nn.py", line 9556, in <lambda>
    gradcheck(lambda self, target: nn.BCEWithLogitsLoss(reduction=reduction)(self, target),
  File "test_nn.py", line 9556, in <lambda>
    gradcheck(lambda self, target: nn.BCEWithLogitsLoss(reduction=reduction)(self, target),
  File "/var/lib/jenkins/torchdynamo/torchdynamo/eval_frame.py", line 93, in _fn
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 655, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 277, in __call__
    raise e
  File "/opt/conda/lib/python3.7/site-packages/torch/fx/graph_module.py", line 267, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1186, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() takes 2 positional arguments but 3 were given

this is strictly profiler change and does not touch any PyTorch core. also it passed in CI before landing

It's unrelated to the BCE test. The issue is that before the class hierarchy looked something like:

struct base {
  virtual void foo();
};

struct derived {
  void foo() override;
};

And it got changed to:

struct base {
  virtual void foo();
};

struct derived {
  // Not a legitimate override.
  void foo(bool) override;
};

Honestly, I think the question isn't why did buck fail but rather "why didn't all other CI fail"? (My suspicion is that PyTorch is using Kineto's CMake files so whatever lets it through Kineto CI also lets it through PyTorch CI. But just a hunch.) It didn't fail CI because buck CI doesn't run on PRs, it runs periodically. If it was an issue with buck I would have fought the revert (I have also been bitten by this and it's very annoying), but in this case buck was flagging a real problem as @kit1980 points out.

Speaking of which, this seems like a good reason to put buck build in the per PR group.