Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to train with multi GPUs? #7

Open
HoJ-Onle opened this issue Aug 4, 2022 · 0 comments
Open

How to train with multi GPUs? #7

HoJ-Onle opened this issue Aug 4, 2022 · 0 comments

Comments

@HoJ-Onle
Copy link

HoJ-Onle commented Aug 4, 2022

Hello! I tried to train the model with multi GPUs. And I found that you have released train_distributed.py
So I tried to use tf.distribute.MirroredStrategy() as strategy to achieve distributed training. But I got an error as follows:

 RuntimeError: `merge_call` called while defining a new graph or a tf.function. This can often happen if the function `fn` passed to `strategy.experimental_run()` is decorated with `@tf.function` (or contains a nested `@tf.function`), and `fn` contains a synchronization point, such as aggregating gradients. This behavior is not yet supported. Instead, please wrap the entire call `strategy.experimental_run(fn)` in a `@tf.function`, and avoid nested `tf.function`s that may potentially cross a synchronization boundary.

Looking forward to your help!!

@HoJ-Onle HoJ-Onle changed the title How to train with multi GPU? How to train with multi GPUs? Aug 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant