[Feature] Support various scales in RRDBNet #699

anse3832 · 2022-01-07T07:57:38Z

Motivation

Current RRDBNet only supports 4x upsampling.
The parameter "radius" in config file of real_esrgan is wrong.

Modification

Add scale parameter in rrdb_net
Edit config file of real_esrgan for supporting various scales
Edit config file of real_esrgan for fixing some errors (radius=50 -> kernel_size=51)

ckkelvinchan

@anse3832 Thank you very much! Would you mind changing also realesrnet_c64b23g32_12x4_lr2e-4_1000k_df2k_ost.py and adding the unittest for the added support here?

Thanks again!

mmedit/models/backbones/sr_backbones/rrdb_net.py

codecov · 2022-01-07T10:03:51Z

Codecov Report

Merging #699 (e461ca0) into master (4f814a8) will increase coverage by 1.22%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #699      +/-   ##
==========================================
+ Coverage   80.59%   81.81%   +1.22%     
==========================================
  Files         200      211      +11     
  Lines       11419    11896     +477     
  Branches     1831     1912      +81     
==========================================
+ Hits         9203     9733     +530     
+ Misses       1932     1868      -64     
- Partials      284      295      +11

Flag	Coverage Δ
unittests	`81.79% <100.00%> (+1.21%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmedit/models/backbones/sr_backbones/rrdb_net.py	`95.83% <100.00%> (+0.52%)`	⬆️
mmedit/datasets/pipelines/crop.py	`95.72% <0.00%> (-1.63%)`	⬇️
mmedit/datasets/pipelines/blur_kernels.py	`83.08% <0.00%> (-1.48%)`	⬇️
mmedit/models/__init__.py	`100.00% <0.00%> (ø)`
mmedit/datasets/__init__.py	`100.00% <0.00%> (ø)`
mmedit/models/backbones/__init__.py	`100.00% <0.00%> (ø)`
mmedit/datasets/pipelines/loading.py	`96.01% <0.00%> (ø)`
mmedit/datasets/pipelines/__init__.py	`100.00% <0.00%> (ø)`
mmedit/datasets/sr_reds_multiple_gt_dataset.py	`100.00% <0.00%> (ø)`
...edit/models/backbones/encoder_decoders/__init__.py	`100.00% <0.00%> (ø)`
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4f814a8...e461ca0. Read the comment docs.

ckkelvinchan · 2022-01-11T06:24:47Z

tests/test_models/test_backbones/test_sr_backbones/test_sr_backbones.py

+    # x2 model, initialization and forward (cpu)
+    net = RRDBNet(
+        in_channels=3,
+        out_channels=3,
+        scale=2,
+        mid_channels=8,
+        num_blocks=2,
+        growth_channels=4)
+    net_init_weights(pretrained=None)
+    input_shape = (1, 3, 12, 12)
+    img = _demo_inputs(input_shape)
+    output = net(img)
+    assert output.shape == (1, 3, 24, 24)
+
+    # x2 model forward (gpu)
+    if torch.cuda.is_avaliable():
+        net = net.cuda()
+        output = net(img.cuda())
+        assert output.shape == (1, 3, 24, 24)
+


The unittest does not pass. Here you can simply copy the original one and set scale to 2.

ckkelvinchan · 2022-01-12T13:19:54Z

Hi @anse3832, it seems that linting failed. You can first install pre-commit by pip install pre-commit. Then you can type pre-commit install in the root directory of mmediting.

After that every time you commit something, it will check the format for you. Thank you very much for your contributions and we are almost there!

ckkelvinchan · 2022-01-12T13:44:54Z

In addition, unittest also failed. You may try to run the test locally to test whether it has any errors~

nijkah · 2022-01-17T00:27:55Z

I'm a member of the team with anse3832.
I helped his PR, lint and pytest.
Can you check it?

ckkelvinchan · 2022-01-17T03:01:22Z

I'm a member of the team with anse3832. I helped his PR, lint and pytest. Can you check it?

Hi @nijkah, thank you very much! I will check it.

ckkelvinchan · 2022-01-17T03:41:41Z

Hi @anse3832 @nijkah, I think it is almost okay. Could you help adding one more unittest, where scale=3? We want to cover the case when scale is not 1,2 or 4

anse3832 · 2022-01-18T01:27:06Z

@ckkelvinchan
I have some concerns about ESRGAN and Real-ESRGAN

The Real-ESRGAN official code (https://github.com/xinntao/Real-ESRGAN) only works when scale is 1,2 or 4. Therefore, I'm worried about supporting x3 scale.
The ESRGAN and Real-ESRGAN use the same "RRDBNet", but it works slightly different.

Real-ESRGAN uses pixelunshuffle, while ESRGAN doesn't
Real-ESRGAN always uses nearest upsample twice regardless of scale (1, 2, 4), while ESRGAN use different times of nearest upsample according to scale (once in x2, x3 scale and twice in x4 scale)

Considering above concerns, I suggest newly writing the code for "RRDBNet" in Real-ESRGAN as "RealRRDBNet"

When user wants to use ESRGAN, the code calls "RRDBNet" (it supports x2, x3 and x4 scale)
When user wants to use Real-ESRGAN, the code calls "RealRRDBNet" (it supports x1, x2 and x4 scale)

Is it okay to proceed these works? If you have any suggestions, please let me know.

ckkelvinchan · 2022-01-18T01:36:05Z

Thank you for your suggestions! What I meant in the previous reply is that we want to cover the following line of code.

That means we are not going to support x3. We just need to make sure in the unittest that an error will be raised. I checked the official code and it seems that x3 is not supported neither.

anse3832 · 2022-01-18T06:33:36Z

@ckkelvinchan
Do you mean that the unittest code should be like below? (it will raise error because of x3)

ckkelvinchan · 2022-01-18T06:38:43Z

@ckkelvinchan Do you mean that the unittest code should be like below? (it will raise error because of x3)

When you expect an error, you should use with pytest.raises(). You can follow the code here.

By the way, you do not need to separate RealRRDBNet from RRDBNet. That means, we support only 1x, 2x, and 4x only, and for 1x and 2x, we use an additional pixel-unshuffle.

Thanks again!

anse3832 · 2022-01-18T07:07:18Z

@ckkelvinchan
In the official ESRGAN code (https://github.com/xinntao/ESRGAN/blob/master/RRDBNet_arch.py), it only supports x4 scale and it does not use pixelunshuffle.

If it is allowed to use pixelunshuffle in ESRGAN (although the official code does not support), I did not need to separate the code for ESRGAN and Real-ESRGAN. In this case, the code supports for x1, x2 or x4 scale. (Neither of official ESRGAN or Real-ESRGAN supports x3 scale)

RRDBNet (both in ESRGAN and Real-SRGAN): supports x1, x2 or x4 scale and uses pixelunshuffle

If I should follow the official code, I think it's better to separate the code.

RRDBNet (in ESRGAN): supports x4 scale and doesn't use pixelunshuffle
RealRRDBNet (in Real-ESRGAN): supports x1, x2 or x4 scale and uses pixelunshuffle

ckkelvinchan · 2022-01-18T07:33:17Z

Yes it is allowed to use pixel-unshuffle in ESRGAN. Since this modification does not affect the original ESRGAN. We can extend it to support x1 and x2.

Lint & Fix import error

ckkelvinchan

Thank you @anse3832 and @nijkah !

CI failed

ckkelvinchan · 2022-01-18T13:36:57Z

Hi @anse3832 it seems that CI failed. Could you help have a look? Thanks again for your help.

nijkah · 2022-01-19T05:01:05Z

@ckkelvinchan Fixed. I think it works this time. 🙏

nijkah · 2022-01-19T13:52:08Z

Checked lint failure.
@anse3832 Can you approve the PR?
anse3832#4

Feature/real esrgan

ckkelvinchan

Thanks @anse3832 and @nijkah ! All tests passed.

ckkelvinchan · 2022-01-20T02:46:22Z

Your efforts are very much appreciated. Look forward to working with you next time!

wangruohui · 2022-01-26T12:28:15Z

Hi all, it seems that this PR accidentally changes file permissions of some config files from 644 to 755, can anybody help to fix that?

nijkah · 2022-01-26T13:06:51Z

@wangruohui
nijkah@0d41d2c
Is it okay to pull request this branch?

ckkelvinchan · 2022-01-27T05:11:08Z

@wangruohui
nijkah@0d41d2c
Is it okay to pull request this branch?

It is fixed in #718 by @wangruohui. Thanks anyways!

Co-authored-by: nijkah <nijkah@gmail.com> Co-authored-by: hakjinlee <hakjinlee@si-analytics.ai>

OpenMMLab-Assistant003 · 2023-04-04T13:08:18Z

Hi @anse3832！First of all, we want to express our gratitude for your significant PR in the MMEditing project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.

We would also like to invite you to join our Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/UjgXkPWNqA

If you have WeChat account，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤ @anse3832

ckkelvinchan reviewed Jan 7, 2022

View reviewed changes

mmedit/models/backbones/sr_backbones/rrdb_net.py Outdated Show resolved Hide resolved

mmedit/models/backbones/sr_backbones/rrdb_net.py Outdated Show resolved Hide resolved

mmedit/models/backbones/sr_backbones/rrdb_net.py Outdated Show resolved Hide resolved

ckkelvinchan changed the title ~~Support various scales in rrdbnet~~ [Feature] Support various scales in RRDBNet Jan 7, 2022

anse3832 force-pushed the feature/real_esrgan branch from 187d11b to 78f4c5b Compare January 11, 2022 05:29

ckkelvinchan reviewed Jan 11, 2022

View reviewed changes

anse3832 force-pushed the feature/real_esrgan branch from b3193b4 to ffb0cfa Compare January 11, 2022 07:22

anse3832 added 2 commits January 18, 2022 17:47

Support various scales in rrdbnet

2be3180

Add a unit test and refactor

e9e5e25

anse3832 force-pushed the feature/real_esrgan branch from 6982f45 to e9e5e25 Compare January 18, 2022 08:49

nijkah and others added 2 commits January 18, 2022 08:56

Lint & Fix import error

cbed6d8

Merge pull request #2 from nijkah/feature/real_esrgan

3ecf7e2

Lint & Fix import error

ckkelvinchan previously approved these changes Jan 18, 2022

View reviewed changes

nijkah and others added 2 commits January 18, 2022 22:51

Fix syntax error in RRDBNet

44762de

Fix syntax error in RRDBNet

50d9a20

anse3832 force-pushed the feature/real_esrgan branch from 378a5bb to 50d9a20 Compare January 19, 2022 01:23

Lint

b6b2450

Merge branch 'feature/real_esrgan' into feature/real_esrgan

7c5f1a3

Merge pull request #4 from nijkah/feature/real_esrgan

e461ca0

Feature/real esrgan

ckkelvinchan approved these changes Jan 20, 2022

View reviewed changes

ckkelvinchan merged commit 8043791 into open-mmlab:master Jan 20, 2022

Yshuo-Li pushed a commit to Yshuo-Li/mmediting that referenced this pull request Jul 15, 2022

[Feature] Support various scales in RRDBNet (open-mmlab#699)

291c2a1

Co-authored-by: nijkah <nijkah@gmail.com> Co-authored-by: hakjinlee <hakjinlee@si-analytics.ai>

Yshuo-Li pushed a commit to Yshuo-Li/mmediting that referenced this pull request Jul 15, 2022

[Feature] Support various scales in RRDBNet (open-mmlab#699)

3f900f7

Co-authored-by: nijkah <nijkah@gmail.com> Co-authored-by: hakjinlee <hakjinlee@si-analytics.ai>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support various scales in RRDBNet #699

[Feature] Support various scales in RRDBNet #699

anse3832 commented Jan 7, 2022 •

edited by ckkelvinchan

Loading

ckkelvinchan left a comment

codecov bot commented Jan 7, 2022 •

edited

Loading

ckkelvinchan Jan 11, 2022

ckkelvinchan commented Jan 12, 2022

ckkelvinchan commented Jan 12, 2022

nijkah commented Jan 17, 2022 •

edited

Loading

ckkelvinchan commented Jan 17, 2022

ckkelvinchan commented Jan 17, 2022

anse3832 commented Jan 18, 2022 •

edited

Loading

ckkelvinchan commented Jan 18, 2022

anse3832 commented Jan 18, 2022

ckkelvinchan commented Jan 18, 2022

anse3832 commented Jan 18, 2022 •

edited

Loading

ckkelvinchan commented Jan 18, 2022

ckkelvinchan left a comment

ckkelvinchan commented Jan 18, 2022

nijkah commented Jan 19, 2022 •

edited

Loading

nijkah commented Jan 19, 2022 •

edited

Loading

ckkelvinchan left a comment

ckkelvinchan commented Jan 20, 2022

wangruohui commented Jan 26, 2022

nijkah commented Jan 26, 2022

ckkelvinchan commented Jan 27, 2022

OpenMMLab-Assistant003 commented Apr 4, 2023

[Feature] Support various scales in RRDBNet #699

[Feature] Support various scales in RRDBNet #699

Conversation

anse3832 commented Jan 7, 2022 • edited by ckkelvinchan Loading

Motivation

Modification

ckkelvinchan left a comment

Choose a reason for hiding this comment

codecov bot commented Jan 7, 2022 • edited Loading

Codecov Report

ckkelvinchan Jan 11, 2022

Choose a reason for hiding this comment

ckkelvinchan commented Jan 12, 2022

ckkelvinchan commented Jan 12, 2022

nijkah commented Jan 17, 2022 • edited Loading

ckkelvinchan commented Jan 17, 2022

ckkelvinchan commented Jan 17, 2022

anse3832 commented Jan 18, 2022 • edited Loading

ckkelvinchan commented Jan 18, 2022

anse3832 commented Jan 18, 2022

ckkelvinchan commented Jan 18, 2022

anse3832 commented Jan 18, 2022 • edited Loading

ckkelvinchan commented Jan 18, 2022

ckkelvinchan left a comment

Choose a reason for hiding this comment

ckkelvinchan commented Jan 18, 2022

nijkah commented Jan 19, 2022 • edited Loading

nijkah commented Jan 19, 2022 • edited Loading

ckkelvinchan left a comment

Choose a reason for hiding this comment

ckkelvinchan commented Jan 20, 2022

wangruohui commented Jan 26, 2022

nijkah commented Jan 26, 2022

ckkelvinchan commented Jan 27, 2022

OpenMMLab-Assistant003 commented Apr 4, 2023

anse3832 commented Jan 7, 2022 •

edited by ckkelvinchan

Loading

codecov bot commented Jan 7, 2022 •

edited

Loading

nijkah commented Jan 17, 2022 •

edited

Loading

anse3832 commented Jan 18, 2022 •

edited

Loading

anse3832 commented Jan 18, 2022 •

edited

Loading

nijkah commented Jan 19, 2022 •

edited

Loading

nijkah commented Jan 19, 2022 •

edited

Loading