Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Enable distributed inference #761

Merged
merged 38 commits into from
Jan 25, 2023
Merged

ENH: Enable distributed inference #761

merged 38 commits into from
Jan 25, 2023

Conversation

kenza-bouzid
Copy link
Contributor

Add --max_num_gpus_inference flag to be able to use all gpus during inference. DDP processes are only killed if max_num_gpus_inference == 1

@kenza-bouzid
use unrepeted sampler for validation
99a5750
@kenza-bouzid
wip verbose
64067f5
@kenza-bouzid
Merge branch 'main' into kenzab/ddp_sampler_without_duplications
25c4102
@kenza-bouzid
wip
3757cef
@kenza-bouzid
add more logs
0caebae
@kenza-bouzid
rebase deepmil to remove debugging logs
bdb5851
@kenza-bouzid
Merge branch 'main' into kenzab/ddp_sampler_without_duplications
018bb61
@kenza-bouzid
update datamodule test
39d8069
@kenza-bouzid
rebase runner
8ba5cc0
@kenza-bouzid
fix warning
6f0a2d6
@kenza-bouzid
deprecate val_set_is_dist
76822c4
@kenza-bouzid
fix imports and linting
2a36ae3
@kenza-bouzid
Merge branch 'main' into kenzab/outputs_gathering
482b7b2
@kenza-bouzid
add test for uneven samples for tiles selection
d7afda0
@kenza-bouzid
add uneven samples to outputs handler tests
8c49c46
@kenza-bouzid
add uneven samples case
712f279
@kenza-bouzid
add uneven test case to loss callback
709a755
@kenza-bouzid
fix test
d66ef63
@kenza-bouzid
add max_num_gpus_inference flag
a5efd73
@kenza-bouzid
linting
87a9673
@kenza-bouzid
fix test outputs utils
080662a
@kenza-bouzid
attempt to fix test callback
15bf472
@kenza-bouzid
wip fix test
e085846
@kenza-bouzid
enable multi gpu evaluation
bdb8f52
@kenza-bouzid
uncomment test
6df0657
@kenza-bouzid
update tests for ddp inference
9c34f61
@kenza-bouzid
fix after_ddp_cleanup
c63061c
@codecov
Copy link

codecov bot commented Jan 23, 2023 •
edited
Loading

Codecov Report

Merging #761 (2be293a) into main (ca4eb45) will increase coverage by 0.02%.
The diff coverage is 80.95%.

Impacted file tree graph

Flag Coverage Δ
hi-ml 82.24% <89.47%> (+0.02%) ⬆️
hi-ml-azure 25.86% <0.00%> (ø)
hi-ml-cpath 75.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
hi-ml-azure/src/health_azure/logging.py 0.00% <0.00%> (ø)
hi-ml/src/health_ml/run_ml.py 77.55% <85.71%> (+0.06%) ⬆️
hi-ml/src/health_ml/deep_learning_config.py 84.23% <100.00%> (+0.28%) ⬆️
hi-ml/src/health_ml/lightning_container.py 79.46% <100.00%> (+0.18%) ⬆️

@kenza-bouzid kenza-bouzid marked this pull request as ready for review January 24, 2023 11:28
ant0nsc
ant0nsc approved these changes Jan 24, 2023
View reviewed changes
Copy link
Collaborator

@ant0nsc ant0nsc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! Does not look complicated at all anymore

hi-ml/src/health_ml/deep_learning_config.py Outdated Show resolved Hide resolved
hi-ml/src/health_ml/run_ml.py Outdated Show resolved Hide resolved
hi-ml/src/health_ml/run_ml.py Outdated Show resolved Hide resolved
hi-ml/src/health_ml/run_ml.py Show resolved Hide resolved
@kenza-bouzid @ant0nsc
Update hi-ml/src/health_ml/deep_learning_config.py
7ba2a06 
Co-authored-by: Anton Schwaighofer <antonsc@microsoft.com>
@kenza-bouzid kenza-bouzid mentioned this pull request Jan 25, 2023
Open
@kenza-bouzid kenza-bouzid merged commit ec8b795 into main Jan 25, 2023
@kenza-bouzid kenza-bouzid deleted the kenzab/ddp_evaluation branch January 25, 2023 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mebristo mebristo mebristo approved these changes

@ant0nsc ant0nsc ant0nsc approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kenza-bouzid @ant0nsc @mebristo