Gpu enable roar #20
Merged
Gpu enable roar #20
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AndreasMadsen
force-pushed
the
gpu-enable-roar
branch
3 times, most recently
from
94838dc
to
3f45565
Compare
Merged
AndreasMadsen
force-pushed
the
gpu-enable-roar
branch
from
3f45565
to
95ccbe2
Compare
ncmeade
reviewed
Mar 3, 2021
| @@ -133,7 +146,7 @@ def _mask_batch(self, batch): | |||
|
|
|||
| def _mask_dataset(self, dataloader, name): | |||
| outputs = [] | |||
| for batch in tqdm(dataloader(batch_size=self.batch_size, num_workers=0, shuffle=False), | |||
| for batch in tqdm(dataloader(batch_size=8, num_workers=0, shuffle=False), | |||
There was a problem hiding this comment.
Is the batch size fixed here due to memory issues?
There was a problem hiding this comment.
Yes. I haven't found another way. It doesn't appear to affect performance much, but properly something I should look more into.
ncmeade
reviewed
Mar 3, 2021
| @@ -21,4 +21,4 @@ pip3 install --no-index --find-links $HOME/python_wheels \ | |||
| # Install comp550 | |||
| cd $HOME/workspace/comp550 | |||
| pip3 install --no-index --no-deps -e . | |||
| python3 -u -X faulthandler "$@" --use-gpu True --num-workers 4 --persistent-dir $SCRATCH/comp550 | |||
| python3 -u -X faulthandler "$@" --use-gpu True --num-workers 3 --persistent-dir $SCRATCH/comp550 | |||
There was a problem hiding this comment.
Why did we enable the fault handler again? I think you mentioned it in a comment before, but I couldn't find it
There was a problem hiding this comment.
It essentially has no performance impact and in case of a Segmentation Fault, it provides a stack trace based on the coredump. I was getting Segmentation Faults when looking into memory issues.
|
This looks good to merge to me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Depends on #19 landing. I will rebase this once #19 landsI enabled CUDA for ROAR and recallibrated all the walltime estimates. Everything should be working and running in a resonable amount of time now.
I did look into using
Trainer().predict()which was added in 1.2.0. However, it doesn't appear to work and while in the CHANGELOG it is still undocumented.