Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Asynchronous support for ctc/hlg decoding #923

Closed
wants to merge 7 commits into from
Closed

Asynchronous support for ctc/hlg decoding #923

wants to merge 7 commits into from

Conversation

pkufool
Copy link
Collaborator

@pkufool pkufool commented Feb 22, 2022

Asynchronous decoding means that the sequences of consecutive chunks can come from different sources, for the production scenarios, not all the waves would come at the same time. With asynchronous decoding we can randomly combine the received chunks into batches.

@pkufool
Copy link
Collaborator Author

pkufool commented Feb 22, 2022 •
edited
Loading

The running logs for number streams equals to 2 :

Click to see the log 
 
./bin/online_decode --use_gpu true --use_ctc_decoding true --jit_pt /ceph-
hw/kangwei/code/pre_trained/jit-bpe-500/exp_500/cpu_jit.pt --bpe_model /ceph-hw/kangwei/code/pre_trained/jit-bpe-500/data/lang_bpe_500/bpe.model /ceph-hw/kang
wei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav /ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav /ceph-hw/kangwei/
code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav /ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav /ceph-hw/kangwei/code
/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav
2022-02-22 12:01:21.470 [I] k2/torch/bin/online_decode.cu:142:int main(int, char**) Device: cuda:0
2022-02-22 12:01:27.431 [I] k2/torch/bin/online_decode.cu:161:int main(int, char**) Load wave files
2022-02-22 12:01:27.435 [I] k2/torch/bin/online_decode.cu:168:int main(int, char**) Compute features
2022-02-22 12:01:27.440 [I] k2/torch/bin/online_decode.cu:176:int main(int, char**) Load neural network model
2022-02-22 12:01:28.749 [I] k2/torch/bin/online_decode.cu:195:int main(int, char**) Compute nnet_output
2022-02-22 12:01:29.390 [I] k2/torch/bin/online_decode.cu:205:int main(int, char**) Build CTC topo
2022-02-22 12:01:29.399 [I] k2/torch/bin/online_decode.cu:215:int main(int, char**) Decoding
[W BinaryOps.cpp:601] Warning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc'
function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (function
operator())
2022-02-22 12:01:29.620 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)
Partial result:
/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav


2022-02-22 12:01:29.931 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BE


2022-02-22 12:01:30.366 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI


2022-02-22 12:01:30.624 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI ANY GOOD


2022-02-22 12:01:30.922 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BEL


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI ANY GOOD IN YOUR


2022-02-22 12:01:31.323 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BELLY COUN


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI ANY GOOD IN YOUR MIND


2022-02-22 12:01:31.584 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BELLY COUNSELLED HIM


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI ANY GOOD IN YOUR MIND


2022-02-22 12:01:31.883 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BELLY COUNSELLED HIM


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUM


2022-02-22 12:01:32.321 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BELLY COUNSELLED HIM


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER T


2022-02-22 12:01:32.546 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav


2022-02-22 12:01:32.835 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHA


2022-02-22 12:01:33.209 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN


2022-02-22 12:01:33.499 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS


2022-02-22 12:01:33.797 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARD


2022-02-22 12:01:34.206 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EX


2022-02-22 12:01:34.490 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISH


2022-02-22 12:01:34.788 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITS


2022-02-22 12:01:35.241 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU GOOD NIGHT


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS


2022-02-22 12:01:35.454 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU GOOD NIGHT HUSBA


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COL


2022-02-22 12:01:35.778 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU GOOD NIGHT HUSBAND


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD


2022-02-22 12:01:36.241 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDI


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER


2022-02-22 12:01:36.393 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER E


2022-02-22 12:01:36.692 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT KNOWLED


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHT


2022-02-22 12:01:37.143 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT KNOWLEDGE OF


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL


2022-02-22 12:01:37.353 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT KNOWLEDGE OF HIMSELF


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YE


2022-02-22 12:01:37.676 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT KNOWLEDGE OF HIMSELF


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMP


2022-02-22 12:01:38.134 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD


2022-02-22 12:01:38.353 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP


2022-02-22 12:01:38.667 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND


2022-02-22 12:01:39.130 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE


2022-02-22 12:01:39.331 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUA


2022-02-22 12:01:39.646 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID


2022-02-22 12:01:40.110 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER


2022-02-22 12:01:40.336 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER OF THE B


2022-02-22 12:01:40.642 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER OF THE BROTHELS


2022-02-22 12:01:41.107 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER OF THE BROTHELS


2022-02-22 12:01:41.291 [I] k2/torch/bin/online_decode.cu:357:int main(int, char**)

Partial result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER OF THE BROTHELS


2022-02-22 12:01:41.291 [I] k2/torch/bin/online_decode.cu:367:int main(int, char**)

Decoding result:

/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0001.wav

STUFFOR IT INTO YOU HIS BELLY COUNSELLED HIM


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0003.wav

HELLO BERTI ANY GOOD IN YOUR MIND


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0004.wav

NUMBER TEN FRESH NELLIE IS WAITING ON YOU GOOD NIGHT HUSBAND


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0008.wav

THE CHAOS IN WHICH HIS ARDOUR EXTINGUISHED ITSELF WAS A COLD INDIFFERENT KNOWLEDGE OF HIMSELF


/ceph-hw/kangwei/code/pre_trained/jit-bpe-500/test_wave/1089-134686-0002.wav

AFTER EARLY NIGHTFALL THE YELLOW LAMPS WOULD LIGHT UP HERE AND THERE THE SQUALID QUARTER OF THE BROTHELS

danpovey
danpovey reviewed Feb 22, 2022
View reviewed changes
k2/csrc/intersect_dense_pruned.cu Outdated
final_t =
is_final ? final_t + b_chunk_size : final_t + b_chunk_size - 1;
final_t_data[i] = final_t;
final_t_data[i] = his_t + b_chunk_size - 1;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps you mean this_t?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean history_t, this_t may be a better name, or start_t

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, history_t is fine.
google style guide prefers using whole words, not shortened ones.

@pkufool pkufool mentioned this pull request Apr 27, 2022
Merged
@pkufool pkufool mentioned this pull request Nov 4, 2022
Merged
@pkufool
Copy link
Collaborator Author

pkufool commented Nov 4, 2022

See #1091

@pkufool pkufool closed this Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@danpovey danpovey danpovey left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@pkufool @danpovey