Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repulsion branch DistributedEnvironment exception produces no output #333

Open
bernstei opened this issue Mar 1, 2024 · 1 comment
Open

Comments

@bernstei
Copy link
Collaborator

bernstei commented Mar 1, 2024

I think the logger need to be set up before the calls that create DistributedEnvironment in cli/run_train.py, otherwise there's no output. Also, since it's an error, it should probably go to logger.error rather than .info, and return a non-zero status when the script terminates.

@bernstei
Copy link
Collaborator Author

bernstei commented Mar 1, 2024

I see now that the logger depends on rank, which isn't known until after the distributed env is created, and I'm not sure what's the simplest way of dealing with that. Maybe just setting the logging level very early, and setting the full logger later. But I still think the DistributedEnvironment exception message should be an error, no?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant