Skip to content

/robosat.pink forked from ocourtin/robosat

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some performance improvements #81

Open
wants to merge 5 commits into
base: master
from
Open

Some performance improvements #81

wants to merge 5 commits into from

Conversation

@dselivanov
Copy link

dselivanov commented Nov 6, 2019

see commit messages
dselivanov added 4 commits Nov 6, 2019
@dselivanov
read png, jpeg, webp with PIL
de0695c
@dselivanov
disable threading in opencv since we use pytorch multiprocessing and …
ba49c1a 
…shm for data loading. this should help with resource oversubscription
@dselivanov
pin memory in dataloader - generally improves data loading throughput…
f875489 
… quite a lot
@dselivanov
merge upstream
a76f1f9 
Merge remote-tracking branch 'upstream/master'

# Conflicts:
#	robosat_pink/tiles.py
@ocourtin

This comment has been minimized.

Sign in to view
Copy link
Collaborator

ocourtin commented Nov 6, 2019

Hi Dmitriy !

Thanks for this PR.
Did you perform some (quick) performance bench on it ?
@dselivanov

This comment has been minimized.

Sign in to view
Copy link
Author

dselivanov commented Nov 6, 2019 •
edited

In my current case it is ~5-7%. But I've observed cases when it is much higher. I found disabling threading is particularly useful when using torch multiprocess data loaders.
@dselivanov

This comment has been minimized.

Sign in to view
Copy link
Author

dselivanov commented Nov 6, 2019

Also note that latest commit brings back png, wepb, jpeg reading with PIL. Your latest changes caused incorrect data loading (I found it as rs compare stopped work)
@dselivanov
fix loading png, jpg, webp images.
8f92f14
@dselivanov dselivanov force-pushed the dselivanov:master branch 2 times, most recently from 7e59d1d to 8f92f14 Nov 7, 2019
ocourtin added a commit that referenced this pull request Nov 7, 2019
@ocourtin
Revert PIL PNG handling suppresion from 8a3bce9. Thanks to @dselivanov …
8b6078b 
…for the report. Cf #81
@ocourtin

This comment has been minimized.

Sign in to view
Copy link
Collaborator

ocourtin commented Nov 7, 2019

Thanks Dmitryi for this regression report. Just fixed.

On the performance improvments, on the other side, i'm still unable to reproduce any significative results, on training. Looks like i'm more benching some inner cache ^^

Feel free to push more information on your own way to reproduce it, and about the kind of hardware you use.

On a more broader comment, i believe more in our ability to significantly enhance performances with #82 and #10
@dselivanov

This comment has been minimized.

Sign in to view
Copy link
Author

dselivanov commented Nov 10, 2019

I run all the experiments on nvidia DGX-2. Reported performance improvement is for spacenet dataset and building detection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@dselivanov @ocourtin
You can’t perform that action at this time.