Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix multiprocessing bug #12

Merged
merged 1 commit into from Mar 4, 2022
Merged

fix multiprocessing bug #12

merged 1 commit into from Mar 4, 2022

Conversation

leminhnguyen
Copy link
Contributor

@leminhnguyen leminhnguyen commented Feb 28, 2022

Hi @RayeRen, today I've tried your preprocess.py code for LJSpeech dataset. And I realized that there were some processed items in metadata.json doesn't have ph_token key or len(ph.split()) is not equal to len(ph_token). So I've checked your code and found the problem with the line 123 in utils/commons/multiprocess_utils.py.

for job_i, res in manager.get_results():
results[job_i] = res
while i_now < n_jobs and (not isinstance(results[i_now], str) or results[i_now] != '<WAIT>'):
yield job_i, results[i_now]
results[i_now] = None
i_now += 1

For my understanding, you tried to return the indices and results as passed order instead of processed order, so I think i_now should be yielded instead of job_i. I also code a small snippet to debug your code, you can use it as reference.

def test_map_func(idx):
    import time
    time.sleep(0.2)
    return {"number": idx*2, "id": idx}

if __name__ == "__main__":
    args = [{"idx": idx} for idx in range(100)]
    ids = []
    for idx, x in multiprocess_run_tqdm(test_map_func, args):
        # args[idx].update(x)
        ids.append(idx)
   print(ids) 
   # print [0, 1, 2, 3, 4, 5, 5, 7, 7, 7, 7, 7, ...] with yield job_i
   # print [0, 1, 2, 3, 4, 6, 7, 8, 9, 10, ...] with yield i_now

@RayeRen RayeRen merged commit 707c1ff into NATSpeech:main Mar 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants