Port DeepLabV3 segmentation model from `keras_cv.models.legacy` to the `Task` API #1831

soumik12345 · 2023-05-27T07:50:30Z

What does this PR do?

This PR moves the DeepLabV3 segmentation model from keras_cv.models.legacy and aligns it with the Task API.

A Kaggle TPU notebook demostrating a few training steps: https://www.kaggle.com/code/soumikrakshit/deeplabv3-tpu/

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue? Please add a link
to it if that's the case.
Did you write any new necessary tests?
If this adds a new model, can you run a few training steps on TPU in Colab to ensure that no XLA incompatible OP are used?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

soumik12345 · 2023-05-28T23:44:07Z

cc: @jbischof

keras_cv/models/segmentation/deeplab_v3/deeplab_v3.py

ianstenbit

Thanks for the awesome PR!

keras_cv/models/segmentation/deeplab_v3/deeplab_v3.py

keras_cv/models/segmentation/__init__.py

keras_cv/layers/segmentation/segmentation_head.py

soumik12345 · 2023-05-30T04:28:14Z

Hi @ianstenbit
Made the changes requested and updated the notebook linked at #1831 (comment)

ianstenbit

High-level question:

Should we just directly offer DeepLabV3Plus? I see from the papers that this is a relatively straightforward addition on top of DeepLabV3, and I see that you wrote a Keras tutorial on this: https://keras.io/examples/vision/deeplabv3_plus/

Unless there's a reason to offer both, I'd suggest that we update this to implement DeepLabV3Plus, and then we can verify the forward pass numerics against your KerasIO tutorial.

jbischof

Great work!

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py

jbischof · 2023-06-14T23:51:31Z

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py

+
+        inputs = backbone.input
+
+        final_backbone_pyramid_output = backbone.get_layer(


Is it safe to assume that this dictionary is sorted? We may need to extract the max manually.

Also: why not use the output of the backbone as the max layer?

This was done as per this request.

I think in general we probably can just use the backbone output here.

I brought this up originally because I wasn't clear on whether DeepLabV3Plus always wants to use the highest-available P-level here or if it always wants to use e.g. P4. It seems like it may want the highest P-level always, which I think should mean it always matches up with the backbone output.

I don't think the last P-level is exactly the same as the output, although it is the same shape in the width and height dimensions. There are often extra normalization and activations (and even extra conv layers!) afterward that we omit for 1:1 comparability with the previous levels (e.g., ResNetV2, MobileNet)

@jbischof
I attempted to train with the output of the backbone as the features, and it seems that the outputs are significantly worse in that case, compared to the P5 features.

Here's a comparison: https://wandb.ai/geekyrakshit/deeplabv3-keras-cv/reports/Backbone-Outputs-vs-P5-Features--Vmlldzo0NjQ5NDUw

I'm having trouble reading that chart. Which color is P5? Which is backbone?

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus_test.py

jbischof · 2023-06-15T00:05:45Z

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py

+            "backbone": self.backbone,
+            "spatial_pyramid_pooling": self.spatial_pyramid_pooling,
+            "projection_filters": self.projection_filters,
+            "segmentation_head": self.segmentation_head,


I still struggle with serialization. Do we need to explicitly serialize and deserialize a submodel like this for saving to work properly? I see a saving test but I know there are gotchas.

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus_test.py

jbischof · 2023-06-15T03:44:58Z

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py

+            used as a feature extractor for the DeepLabV3+ Encoder. Should
+            either be a `keras_cv.models.backbones.backbone.Backbone` or a
+            `tf.keras.Model` that implements the `pyramid_level_inputs`
+            property with keys "P2", "P3", "P4", and "P5" and layer names as


Do we need all five keys? It just seems like "P2" now

jbischof · 2023-06-15T14:41:52Z

Do we have a reference implementation for this model where we can check the forward pass and maybe grab some pretrained weights?

Also: don't forget to set up presets!

Thanks for your hard work 🚀

ianstenbit · 2023-06-27T18:54:03Z

@soumik12345 do you think you'll be able to finish up this PR soon?

If not I may push some changes to your branch so that we can get this finished -- thank you!

soumik12345 · 2023-06-27T21:45:29Z

@soumik12345 do you think you'll be able to finish up this PR soon?

If not I may push some changes to your branch so that we can get this finished -- thank you!

Hi @ianstenbit
There are probably no changes left to make from an architectural perspective. I can train the model on the Cityscapes fine-annotation set (without extra data) and add some presets.

Please let me know if there are any immediate changes that I can make.

ianstenbit · 2023-06-30T17:03:01Z

@soumik12345 do you think you'll be able to finish up this PR soon?
If not I may push some changes to your branch so that we can get this finished -- thank you!

Hi @ianstenbit There are probably no changes left to make from an architectural perspective. I can train the model on the Cityscapes fine-annotation set (without extra data) and add some presets.

Please let me know if there are any immediate changes that I can make.

Yeah I think the architecture + tests are looking good -- just want to verify the performance by training it against a benchmark 😄

ianstenbit · 2023-07-05T16:05:02Z

keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus_presets.py

+                "pixel-level semantic labeling task, which consists of fine "
+                "annotations for train and val sets (3475 annotated images) "
+                "and 34 classes. This model achieves a final Mean IoU of "
+                "0.36 on the validation set for fine annotations."


How does this compare to the expected performance of this model on this dataset?

ianstenbit · 2023-07-12T20:40:29Z

@soumik12345 we'd like to go ahead and merge this for now (without worrying about verifying the performance just yet, and without porting to Keras Core).

Would you be willing to:

Fix the lint error
Remove the preset for now

and then I can merge it?

I tried to push to to your fork to make the changes myself but I don't have access.

soumik12345 · 2023-07-12T20:54:02Z

@soumik12345 we'd like to go ahead and merge this for now (without worrying about verifying the performance just yet, and without porting to Keras Core).

Would you be willing to:

Fix the lint error

Remove the preset for now

and then I can merge it?

I tried to push to to your fork to make the changes myself but I don't have access.

Hi @ianstenbit removed the presets and fixed linting.
Immense Thanks to you and @jbischof for guiding me through this PR!

ianstenbit · 2023-07-12T20:55:26Z

@soumik12345 we'd like to go ahead and merge this for now (without worrying about verifying the performance just yet, and without porting to Keras Core).
Would you be willing to:

Fix the lint error

Remove the preset for now

and then I can merge it?
I tried to push to to your fork to make the changes myself but I don't have access.

Hi @ianstenbit removed the presets and fixed linting. Immense Thanks to you and @jbischof for guiding me through this PR!

Wonderful, thank you for the contribution! If the CI is happy then I will go ahead and merge this 😄

ianstenbit · 2023-07-12T20:59:38Z

Looks like the linter is still unhappy.

./keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py:28:81: E501 line too long (127 > 80 characters)
./keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py:30:81: E501 line too long (107 > 80 characters)
./keras_cv/models/segmentation/deeplab_v3_plus/deeplab_v3_plus.py:48:81: E501 line too long (132 > 80 characters)

You'll need to add # noqa: E501 to the end of the docstring for DeepLabV3Plus (See YOLOV8Detector as an example of how to do this)

ianstenbit · 2023-07-12T21:16:52Z

Keras Core (incl. accelerator) testing is not yet relevant for this, so this is good to go

will fix during port to KerasCore

…e `Task` API (keras-team#1831) * add: DeepLabV3 backbone + preset placeholders * add: presents for DeepLabV3 backbone * update: minor changes * update: minor changes * add: license info * update: refactored DeepLabV3 as a single Task * add: deeplab_v3 module inside segmentation module * update: refactor segmentation head * add: serialization test for SegmentationHead * remove: legacy segmentation models dir * remove: legacy segmentation imports * update: added weight_decay in compile() * add: docstring for deeplabv3 * update: remove compile() and train_step() from DeepLabV3 * update: imports * update: copyright info * remove: Segmentation Head layer implementation + serialization test * update: refactor DeepLabV3 segmentation head as a keras Sequential * update: made abstractions to implement encoder-decoder architecture * update: _make_deeplabv3_encoder * refactor: docstrings for DeepLabV3 * refactor: docstrings for DeepLabV3 * refactor: DeepLabV3 * update: refactored DeepLabV3 to implement DeepLabV3+ * update: docstring * update: docstring * update: removed segmentation_head_activation * update: removed input_shape and input_tensor * update: remove outdated comment * update: docstring * update: replaced low_level_feature_layer_name with low_level_feature_pyramid_level * update: docastring * update: DeepLabV3 * update: remove unused imports * update: changed DeepLabV3 to DeepLabV3+ * update: remove build * update: docstring * update: add docstring for dropout * update: made encoder output upsampling dynamic * add: tests for DeepLabV3+ * update: order of parameters * update: example in docstring * update: renamed feature_map * update: docs for dropout * update: renamed tests * update: update tests * update: removed test_no_nans * update: fix linting * update: remove cleanup_global_session * update: tests * update: fix linting * update: incorporated feedback * update: incorporated feedback * update: incorporated feedback * add: preset for deeplabv3+ * update: remove presets * fix: linting * update: make CI happy

add: DeepLabV3 backbone + preset placeholders

da1339d

soumik12345 marked this pull request as draft May 27, 2023 07:50

soumik12345 added 12 commits May 27, 2023 08:23

add: presents for DeepLabV3 backbone

df81262

update: minor changes

2d320c3

update: minor changes

e587c5b

add: license info

84a48ba

update: refactored DeepLabV3 as a single Task

40128b0

add: deeplab_v3 module inside segmentation module

0790564

update: refactor segmentation head

586d424

add: serialization test for SegmentationHead

08a98b8

remove: legacy segmentation models dir

7f743c1

remove: legacy segmentation imports

fc45181

update: added weight_decay in compile()

9ff69c9

add: docstring for deeplabv3

59b8748

soumik12345 marked this pull request as ready for review May 28, 2023 23:43

soumik12345 changed the title ~~Port DeepLabV3 segmentation model from keras_cv.models.legacy~~ Port DeepLabV3 segmentation model from keras_cv.models.legacy to Task API May 28, 2023

soumik12345 changed the title ~~Port DeepLabV3 segmentation model from keras_cv.models.legacy to Task API~~ Port DeepLabV3 segmentation model from keras_cv.models.legacy to the Task API May 28, 2023

ianstenbit requested review from jbischof and ianstenbit May 29, 2023 14:35

innat reviewed May 29, 2023

View reviewed changes

keras_cv/models/segmentation/deeplab_v3/deeplab_v3.py Outdated Show resolved Hide resolved

ianstenbit suggested changes May 29, 2023

View reviewed changes

soumik12345 added 5 commits May 30, 2023 03:13

update: remove compile() and train_step() from DeepLabV3

3166e79

update: imports

ecdd0f0

update: copyright info

6d63e76

remove: Segmentation Head layer implementation + serialization test

dea43bd

update: refactor DeepLabV3 segmentation head as a keras Sequential

57652bd

soumik12345 requested a review from ianstenbit May 30, 2023 04:28

ianstenbit reviewed May 30, 2023

View reviewed changes

Merge branch 'keras-team:master' into master

747d977

jbischof reviewed Jun 15, 2023

View reviewed changes

soumik12345 added 3 commits June 15, 2023 02:33

update: incorporated feedback

196aa33

update: incorporated feedback

a4978a7

update: incorporated feedback

8809cf1

jbischof reviewed Jun 15, 2023

View reviewed changes

soumik12345 mentioned this pull request Jun 23, 2023

feat: PyNet model implemented soumik12345/pynet#1

Merged

7 tasks

Merge branch 'keras-team:master' into master

8d6d178

soumik12345 added 2 commits July 2, 2023 16:57

Merge branch 'keras-team:master' into master

54b6ea8

add: preset for deeplabv3+

8878502

soumik12345 requested a review from ianstenbit July 3, 2023 19:15

ianstenbit reviewed Jul 5, 2023

View reviewed changes

Merge branch 'keras-team:master' into master

1d0cfbf

soumik12345 added 3 commits July 13, 2023 02:15

Merge branch 'keras-team:master' into master

bd2f0f9

update: remove presets

9d4fa70

fix: linting

f0a6d19

update: make CI happy

7b4a221

ianstenbit approved these changes Jul 12, 2023

View reviewed changes

ianstenbit merged commit 3b97525 into keras-team:master Jul 12, 2023
6 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port DeepLabV3 segmentation model from `keras_cv.models.legacy` to the `Task` API #1831

Port DeepLabV3 segmentation model from `keras_cv.models.legacy` to the `Task` API #1831

soumik12345 commented May 27, 2023 •

edited

Loading

soumik12345 commented May 28, 2023

ianstenbit left a comment

soumik12345 commented May 30, 2023

ianstenbit left a comment

jbischof left a comment

jbischof Jun 14, 2023

soumik12345 Jun 15, 2023

ianstenbit Jun 15, 2023

jbischof Jun 15, 2023 •

edited

Loading

soumik12345 Jun 15, 2023

jbischof Jun 16, 2023

jbischof Jun 15, 2023

jbischof Jun 15, 2023

jbischof commented Jun 15, 2023 •

edited

Loading

ianstenbit commented Jun 27, 2023

soumik12345 commented Jun 27, 2023

ianstenbit commented Jun 30, 2023

ianstenbit Jul 5, 2023

ianstenbit commented Jul 12, 2023

soumik12345 commented Jul 12, 2023

ianstenbit commented Jul 12, 2023

ianstenbit commented Jul 12, 2023

ianstenbit commented Jul 12, 2023


		inputs = backbone.input

		final_backbone_pyramid_output = backbone.get_layer(

Port DeepLabV3 segmentation model from keras_cv.models.legacy to the Task API #1831

Port DeepLabV3 segmentation model from keras_cv.models.legacy to the Task API #1831

Conversation

soumik12345 commented May 27, 2023 • edited Loading

What does this PR do?

Before submitting

Who can review?

soumik12345 commented May 28, 2023

ianstenbit left a comment

Choose a reason for hiding this comment

soumik12345 commented May 30, 2023

ianstenbit left a comment

Choose a reason for hiding this comment

jbischof left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbischof Jun 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbischof commented Jun 15, 2023 • edited Loading

ianstenbit commented Jun 27, 2023

soumik12345 commented Jun 27, 2023

ianstenbit commented Jun 30, 2023

Choose a reason for hiding this comment

ianstenbit commented Jul 12, 2023

soumik12345 commented Jul 12, 2023

ianstenbit commented Jul 12, 2023

ianstenbit commented Jul 12, 2023

ianstenbit commented Jul 12, 2023

Port DeepLabV3 segmentation model from `keras_cv.models.legacy` to the `Task` API #1831

Port DeepLabV3 segmentation model from `keras_cv.models.legacy` to the `Task` API #1831

soumik12345 commented May 27, 2023 •

edited

Loading

jbischof Jun 15, 2023 •

edited

Loading

jbischof commented Jun 15, 2023 •

edited

Loading