Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RNN-T and TDT inference: use CUDA graphs by default #8972

Merged
merged 39 commits into from
May 3, 2024
Merged

RNN-T and TDT inference: use CUDA graphs by default #8972

merged 39 commits into from
May 3, 2024

Commits on Apr 18, 2024

  1. Use Cuda graphs by default for transcription 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    7441a2f View commit details
    Browse the repository at this point in the history

  2. RNN-T Loop Labels + Cuda graphs user-friendly 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    4761e68 View commit details
    Browse the repository at this point in the history

  3. Fix Cuda graphs mode 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    2b13f31 View commit details
    Browse the repository at this point in the history

  4. Fuse graphs 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    d391e98 View commit details
    Browse the repository at this point in the history

  5. Enable by default Cuda graphs for TDT 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    0205101 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2024

  1. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    0cac599 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 19, 2024
    Configuration menu
    Copy the full SHA
    cb68701 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    b38ff6f View commit details
    Browse the repository at this point in the history

  2. Add test 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    25235bb View commit details
    Browse the repository at this point in the history

  3. Speedup init state 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    f175c8a View commit details
    Browse the repository at this point in the history

  4. Add comments 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    1672739 View commit details
    Browse the repository at this point in the history

  5. Speedup tests 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    9e18150 View commit details
    Browse the repository at this point in the history

  6. Add comments 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    94ba6af View commit details
    Browse the repository at this point in the history

  7. Fix tests for alignments 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7b9d619 View commit details
    Browse the repository at this point in the history

  8. Fix test 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    2d3b083 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 25, 2024
    Configuration menu
    Copy the full SHA
    7e805bc View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. Test decoder in forced mode 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    43c01ac View commit details
    Browse the repository at this point in the history

  2. Set max_symbols to 10 if None. Add comments 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    030be86 View commit details
    Browse the repository at this point in the history

  3. Fix issue with confidence + bfloat16 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    100fd9c View commit details
    Browse the repository at this point in the history

  4. Test with confidence 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    6b5d1d2 View commit details
    Browse the repository at this point in the history

  5. Add comment about setting variables in config 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 27, 2024
    Configuration menu
    Copy the full SHA
    07bd665 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    638823e View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Enable CUDA graphs everywhere. Disable explicitly in training pipeline. 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    464dd51 View commit details
    Browse the repository at this point in the history

  2. Revert redundant changes 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    2b7cd73 View commit details
    Browse the repository at this point in the history

  3. Fix comment 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e730f91 View commit details
    Browse the repository at this point in the history

  4. Fix typo 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    cc06bf0 View commit details
    Browse the repository at this point in the history

  5. Fix enabling CUDA graphs 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    cf38241 View commit details
    Browse the repository at this point in the history

  6. Instantiate RNNTGreedyDecodeCudaGraph only when all conditions are met 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    19ca09d View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Fix hybrid ASR-TTS model 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d98b8fc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored Apr 30, 2024
    Configuration menu
    Copy the full SHA
    05eb103 View commit details
    Browse the repository at this point in the history

  3. Move toggling CUDA graphs to ASRModel 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    7c6f7f0 View commit details
    Browse the repository at this point in the history

  4. Remove redundant import 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    cb6d500 View commit details
    Browse the repository at this point in the history

  5. Clean up 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    35564df View commit details
    Browse the repository at this point in the history

  6. Clean up 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    7192acc View commit details
    Browse the repository at this point in the history

  7. Clean up 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    d4a27f6 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. Extract toggling CUDA graphs logic to WithOptionalCudaGraphs. Fix C… 

    …UDA graphs in `ASRModel`

Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed May 2, 2024
    Configuration menu
    Copy the full SHA
    c0877f2 View commit details
    Browse the repository at this point in the history

  2. Fix unused imports 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed May 2, 2024
    Configuration menu
    Copy the full SHA
    b880510 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into rnnt_cuda_graphs_default

    @artbataev
    artbataev authored May 2, 2024
    Configuration menu
    Copy the full SHA
    82f83dc View commit details
    Browse the repository at this point in the history

  4. Fix hook (failing tests) 

    Signed-off-by: Vladimir Bataev <vbataev@nvidia.com>
    @artbataev
    artbataev committed May 2, 2024
    Configuration menu
    Copy the full SHA
    4e47010 View commit details
    Browse the repository at this point in the history