Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add NVTX tracing hooks for profiling with Nsight Systems #2723

Merged
merged 15 commits into from
Mar 29, 2021
Merged

Add NVTX tracing hooks for profiling with Nsight Systems #2723

merged 15 commits into from
Mar 29, 2021

Commits on Mar 28, 2021

  1. Add NVTX in cmake 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    20bbad2 View commit details
    Browse the repository at this point in the history

  2. Some cleanup suggested by clang tidy 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    f15f97c View commit details
    Browse the repository at this point in the history

  3. fix clearing tensor state in Timeline::End 

    (assertion in Timeline::NegotiateStart would fail otherwise)

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    bbf626f View commit details
    Browse the repository at this point in the history

  4. Integrate NVTX tracing into Horovod timeline 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    e3037fa View commit details
    Browse the repository at this point in the history

  5. Fix unused variable warning 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    1f50e9d View commit details
    Browse the repository at this point in the history

  6. Add basic NVTX tracing to tensorflow/mpi_ops.cc 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    ae15365 View commit details
    Browse the repository at this point in the history

  7. Clean up Enqueue* function signatures (clang-tidy suggestions) 

    (remove ineffective const qualifiers, pass strings by reference if possible)

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    a0c7790 View commit details
    Browse the repository at this point in the history

  8. Minor cleanup in Status, TensorShape, OpContext 

    (remove some redundant qualifiers, avoid some copies)

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    ac7ebdd View commit details
    Browse the repository at this point in the history

  9. Replace TensorFlow-specific NVTX annotations by framework-agnostic an… 

    …notations

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    a2921d4 View commit details
    Browse the repository at this point in the history

  10. Replace TensorFlow-specific NVTX annotations by framework-agnostic an… 

    …notations II

(add missing files)

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    51cd685 View commit details
    Browse the repository at this point in the history

  11. Introduce shared_ptr ref counting to fix ending HorovodGroupedAllredu… 

    …ce ranges.

Also avoid some shared_ptr copies in Enqueue*().

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    72c13d0 View commit details
    Browse the repository at this point in the history

  12. Put NvtxOpRange shared_ptr into an opaque wrapper 

    (helps the compiler optimize it away when building without NVTX)

Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    df53e71 View commit details
    Browse the repository at this point in the history

  13. Introduce environment variable HOROVOD_DISABLE_NVTX_RANGES 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    c9a99c8 View commit details
    Browse the repository at this point in the history

  14. Update changelog 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    bd5fb42 View commit details
    Browse the repository at this point in the history

  15. Revert std::move calls in Enqueue* 

    Signed-off-by: Max H. Gerlach <git@maxgerlach.de>
    @maxhgerlach
    maxhgerlach committed Mar 28, 2021
    Configuration menu
    Copy the full SHA
    691db99 View commit details
    Browse the repository at this point in the history