Chunking over frequency instead of time #39
Conversation
max chunks is the number of freqs.
| def __call__(self, Obs, acitivity_freq): | ||
|
|
||
| initialization = cp.asarray(acitivity_freq, dtype=cp.float64) | ||
| def __call__(self, Obs, activity_freq): |
There was a problem hiding this comment.
I ve just corrected spelling here
|
Awesome! I'll merge once you can verify that the performance remains unchanged (which I believe it should) :) |
|
There seems to be some inconsistency when I change the number of GPUs (I don't think this depends on this PR however). With 3 GPUs: With 2 GPU: With 1 GPU: |
That's strange. I have never seen this happen before. Can you check if your GPUs are configured to not share memory? |
Yep they were in DEFAULT mode. Changed to exclusive mode and it does not happen anymore. |
Something to prevent it would be great. I had this issue in CHiME-7. |
|
I think it should be sufficient to add this in the README (perhaps as an FAQ), instead of restricting certain modes in the processing. OOM issues can happen for a variety of reasons, such as if the GPU memory is not cleared from a previous running process or misconfigured nodes, and we cannot expect to solve all such problems. |
| initialization = cp.where(initialization == 0, 1e-10, initialization) | ||
| initialization = initialization / cp.sum(initialization, keepdims=True, axis=0) | ||
| initialization = cp.repeat(initialization[None, ...], 513, axis=0) | ||
| initialization = cp.repeat(initialization[None, ...], F, axis=0) |
|
I think I can grep the processing mode from |
You could put these instructions in the README, so that users running the code can check. No need to add it in the code, IMO. |
|
Discussing offline with @boeddeker, we added a new utils called gpu_check. $cmd JOB=1:$nj ${exp_dir}/${dset_name}/${dset_part}/log/enhance.JOB.log \
gss utils gpu_check $nj $cmd \& gss enhance cuts \
${exp_dir}/${dset_name}/${dset_part}/cuts.jsonl.gz ${exp_dir}/${dset_name}/${dset_part}/split$nj/cuts_per_segment.JOB.jsonl.gz \
${exp_dir}/${dset_name}/${dset_part}/enhanced \
--bss-iterations $gss_iterations \
--context-duration 15.0 \
--use-garbage-class \
--min-segment-length 0.0 \
--max-segment-length $max_segment_length \
--max-batch-duration $max_batch_duration \
--max-batch-cuts 1 \
--num-buckets 4 \
--num-workers 4 \
--force-overwrite \
--duration-tolerance 3.0 \
${affix} || exit 1However it will not exit when it raises an exception used as this. |
The |
|
In the meantime, I confirm i get same results with old version on CHiME-7: ###################################################
### Metrics for all Scenarios ###
###################################################
+----+------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------+
| | scenario | num spk hyp | num spk ref | tot utterances hyp | tot utterances ref | hits | substitutions | deletions | insertions | wer |
|----+------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------|
| 0 | chime6 | 8 | 8 | 6644 | 6644 | 42884 | 11672 | 4325 | 3107 | 0.324451 |
| 0 | dipco | 20 | 20 | 3673 | 3673 | 22175 | 5817 | 1974 | 2210 | 0.333745 |
| 0 | mixer6 | 118 | 118 | 14804 | 14804 | 126632 | 15991 | 6358 | 7815 | 0.202469 |
+----+------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------+
####################################################################
### Macro-Averaged Metrics across all Scenarios (Ranking Metric) ###
####################################################################
+----+---------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------+
| | scenario | num spk hyp | num spk ref | tot utterances hyp | tot utterances ref | hits | substitutions | deletions | insertions | wer |
|----+---------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------|
| 0 | macro-average | 48.6667 | 48.6667 | 8373.67 | 8373.67 | 63897 | 11160 | 4219 | 4377.33 | 0.286888 |
+----+---------------+---------------+---------------+----------------------+----------------------+--------+-----------------+-------------+--------------+----------+ |
|
Added some lines in the README.md |
Most code comes from @boeddeker .
Also he raised the issue here #33
I am re-running the code on CHiME-7 to see if it will match the previous version.