Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make and save predictions and do eval chip-by-chip #635

Merged
merged 8 commits into from Jan 24, 2019

Conversation

Projects
None yet
2 participants
@lewfish
Copy link
Contributor

lewfish commented Dec 20, 2018

Overview

Currently, the predict command makes predictions for each chip in a scene, holding all the predictions in memory and then saves them to disk. This uses too much RAM for very large scenes. The same happens for the eval command. This PR makes it so that each prediction (for a chip) is written to disk immediately, and so that the eval is done one window at a time, and not over the whole scene.

The main change supporting this is making the SemanticSegmentationLabels class compute labels dynamically rather than storing them all for the set of windows. This is accomplished by making label_fn a parameter of the constructor for SemanticSegmentationLabels. The label_fn converts a window to a label array, and can implemented in different ways depending on the context. It calls the backend predict method for the predict command, and Rasterio windowed read operations during eval.

Notes

The vectorification process involves loading the whole prediction array into RAM, which means that won't work on very large scenes. If that is a priority, we can fix it in a separate PR.

Testing Instructions

  • I tested this on a private experiment with very large scenes, that previously would crash during each of chip, predict, and eval, and it now works.
  • The correctness of the changes should be captured by the unit and integration tests.
  • I also tested this by running on the Vegas example and checked that the vectorification still works. (Tested using instructions in #658. That bug seems to be coming from the vectorification code. I got it to work by swapping in a known good model into the train dir after the training step completes.)

Closes #632
Closes #651

@lewfish lewfish added the review label Dec 20, 2018

@lewfish lewfish force-pushed the lf/semseg-mem2 branch 4 times, most recently from 79b1313 to c2b0754 Dec 24, 2018

@lewfish lewfish changed the title WIP: Make and save predictions and do eval chip-by-chip Make and save predictions and do eval chip-by-chip Dec 26, 2018

@lewfish lewfish force-pushed the lf/semseg-mem2 branch 2 times, most recently from cc99e41 to 64459d6 Dec 27, 2018

@codecov

This comment has been minimized.

Copy link

codecov bot commented Dec 31, 2018

Codecov Report

❗️ No coverage uploaded for pull request base (develop@0efe56e). Click here to learn what that means.
The diff coverage is 71.09%.

Impacted file tree graph

@@            Coverage Diff             @@
##             develop     #635   +/-   ##
==========================================
  Coverage           ?   72.01%           
==========================================
  Files              ?      171           
  Lines              ?     7878           
  Branches           ?        0           
==========================================
  Hits               ?     5673           
  Misses             ?     2205           
  Partials           ?        0
Impacted Files Coverage Δ
rastervision/backend/backend.py 58.33% <ø> (ø)
rastervision/backend/tf_deeplab.py 15.65% <0%> (ø)
rastervision/predictor.py 15.25% <0%> (ø)
rastervision/data/label_store/label_store.py 80% <100%> (ø)
...stervision/evaluation/classification_evaluation.py 88.57% <100%> (ø)
...label_source/semantic_segmentation_label_source.py 97.56% <100%> (ø)
.../label_store/semantic_segmentation_raster_store.py 23.63% <15.78%> (ø)
rastervision/task/semantic_segmentation.py 21.91% <27.77%> (ø)
rastervision/data/label/labels.py 62.5% <50%> (ø)
...rvision/data/label/semantic_segmentation_labels.py 85.36% <82.85%> (ø)
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0efe56e...160aefd. Read the comment docs.

@codecov

This comment has been minimized.

Copy link

codecov bot commented Dec 31, 2018

Codecov Report

Merging #635 into develop will decrease coverage by 0.18%.
The diff coverage is 72.02%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #635      +/-   ##
===========================================
- Coverage    70.95%   70.76%   -0.19%     
===========================================
  Files          171      171              
  Lines         8000     8052      +52     
===========================================
+ Hits          5676     5698      +22     
- Misses        2324     2354      +30
Impacted Files Coverage Δ
rastervision/backend/backend.py 58.33% <ø> (ø) ⬆️
rastervision/backend/tf_deeplab.py 15.65% <0%> (-0.12%) ⬇️
rastervision/predictor.py 15.25% <0%> (-0.27%) ⬇️
...label_source/semantic_segmentation_label_source.py 97.56% <100%> (+0.26%) ⬆️
rastervision/data/label_store/label_store.py 80% <100%> (+17.5%) ⬆️
...stervision/evaluation/classification_evaluation.py 82.05% <100%> (+0.97%) ⬆️
.../label_store/semantic_segmentation_raster_store.py 15.85% <15%> (-0.37%) ⬇️
rastervision/task/semantic_segmentation.py 21.91% <27.77%> (+0.48%) ⬆️
rastervision/data/label/labels.py 62.5% <50%> (-4.17%) ⬇️
...rvision/data/label/semantic_segmentation_labels.py 83.72% <81.08%> (-12.12%) ⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7eafab6...27c2c5c. Read the comment docs.

@lewfish lewfish force-pushed the lf/semseg-mem2 branch 2 times, most recently from 0f68b2e to 6e68796 Jan 14, 2019

@lewfish lewfish force-pushed the lf/semseg-mem2 branch from 6e68796 to cf2b3d3 Jan 14, 2019

@lewfish lewfish force-pushed the lf/semseg-mem2 branch from cf2b3d3 to 27c2c5c Jan 14, 2019

@lewfish lewfish requested a review from jamesmcclain Jan 16, 2019

@jamesmcclain
Copy link
Member

jamesmcclain left a comment

Given the review instructions, there seems to be little for me to do other than run tests and inspect the code. I looked most carefully at 10d447a and 099c4d7 because they seem to be the most important parts of this multi-parter?

Meta-comment: multi-step changes should probably be in sequences of minimal PRs and unrelated changes (unless they are small) should probably be in separate PRs. I am sure that I am guilty of this, as well.

@lewfish

This comment has been minimized.

Copy link
Contributor Author

lewfish commented Jan 24, 2019

Meta-comment: multi-step changes should probably be in sequences of minimal PRs and unrelated changes (unless they are small) should probably be in separate PRs. I am sure that I am guilty of this, as well.

I agree. But in this case, I thought that the main commits (the first four) represented a cohesive set of changes that should be reviewed together, and that it would be more unwieldy to have a separate PR for each of them.

@lewfish lewfish merged commit 7097791 into develop Jan 24, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@lewfish lewfish removed the review label Jan 24, 2019

@lewfish lewfish deleted the lf/semseg-mem2 branch Jan 24, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.