GPU-based vectorized SpecAug
#9041
Conversation
Signed-off-by: Piotr Żelasko <petezor@gmail.com>
| ) -> torch.Tensor: | ||
| if isinstance(width, float): | ||
| width = length * width | ||
| value = torch.rand(x.shape[0], device=x.device, dtype=x.dtype) * width |
There was a problem hiding this comment.
I think that using dtype=x.dtype, followed by multiplication by width can cause a very subtle mistake here.
Basically, the goal is uniformly generate an integer in [0, width). Suppose our datatype was bfloat16. The smallest integer not representable by bfloat16 is 257, which seems precariously small. If width were 258, then I don't think this code could ever generate a value such that value.long() would equal 257 (but I could be wrong).
I think it is probably safest to do the entirety of this math using integers. The only concern is if width is a floating point value, which it sometimes is (I guess width is intended as a "stretch" factor on the length when it is a floating point value?). I can probably do this for you if you want.
There was a problem hiding this comment.
Great point. Because of width typically being a float in practice, maybe instead we can hardcode float32 as the dtype for randn.
Signed-off-by: Piotr Żelasko <petezor@gmail.com>
|
Closing in favor of #9155 |
What does this PR do ?
Context: together with @galv we found feature normalization and specaug to take approx 30% of the total forward step time in Canary training, due to CPU-bottlenecked implementations. Feature normalization is addressed in #8964.
This PR adds a GPU based SpecAugment. The original implementation loops over every example and every mask, then waits on CPU RNG to sample the numbers. The new fast implementation still applies masks sequentially, but is vectorized on batch size and uses GPU's RNG. We found approx. 5x speedup (70ms -> 17ms in profiling, but both numbers include profiler overhead). I also added a flag to be able to revert to the old implementation in case anybody encounters a compatibility issue.
I validated visually that the new impl behavior is as expected.
Old:
New:
Collection: ASR
Changelog
Usage
# Add a code snippet demonstrating how to use thisJenkins CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
There's no need to comment
jenkinson the PR to trigger Jenkins CI.The GitHub Actions CI will run automatically when the PR is opened.
To run CI on an untrusted fork, a NeMo user with write access must click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information