Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix error on exiting #5053

Merged
merged 7 commits into from
May 31, 2021
Merged

fix error on exiting #5053

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
01345ed
fix error on exiting
daquexian May 31, 2021
848c832
Merge branch 'master' into fix_atexit_error
daquexian May 31, 2021
27e3fbb
lazily get rank
daquexian May 31, 2021
68f4acd
Merge branch 'master' into fix_atexit_error
oneflow-ci-bot May 31, 2021
97d9872
Merge branch 'master' into fix_atexit_error
oneflow-ci-bot May 31, 2021
8938f8f
Merge branch 'master' into fix_atexit_error
oneflow-ci-bot May 31, 2021
b6cd1da
Merge branch 'master' into fix_atexit_error
oneflow-ci-bot May 31, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 13 additions & 4 deletions oneflow/init.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,13 @@
del env_util


def SyncOnMaster():
if oneflow.distributed.get_rank() == 0:
oneflow._oneflow_internal.eager.single_client.Sync
# capture oneflow methods so that they can be still accessed after `del oneflow`
def _SyncOnMasterFn(get_rank, sync):
def SyncOnMaster():
if get_rank() == 0:
sync()

return SyncOnMaster


atexit.register(oneflow._oneflow_internal.SetShuttingDown)
Expand All @@ -80,7 +84,12 @@ def SyncOnMaster():
# Global<ResourceDesc, ForSession>::Get(), used by vm in background thread,
# will be set to nullptr by TryCloseDefaultSession,
# so sync vm in advance to avoid data race
atexit.register(SyncOnMaster)
atexit.register(
_SyncOnMasterFn(
oneflow.python.framework.distribute.get_rank,
oneflow._oneflow_internal.eager.single_client.Sync,
)
)
del atexit

import sys
Expand Down