Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in SSIM #213

Merged
merged 13 commits into from
Mar 2, 2021
Merged

Fix bug in SSIM #213

merged 13 commits into from
Mar 2, 2021

Conversation

zakajd
Copy link
Collaborator

@zakajd zakajd commented Jan 21, 2021

Closes #209 and closes #203

Proposed Changes

TODO:

  • Add tests to compare with MATLAB
  • Update table in README

@codecov
Copy link

codecov bot commented Jan 22, 2021

Codecov Report

Merging #213 (051b5a7) into master (d401e8c) will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #213      +/-   ##
==========================================
+ Coverage   95.18%   95.21%   +0.03%     
==========================================
  Files          28       29       +1     
  Lines        1766     1778      +12     
==========================================
+ Hits         1681     1693      +12     
  Misses         85       85              
Flag Coverage Δ
unittests 95.21% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
piq/__init__.py 100.00% <100.00%> (ø)
piq/ms_ssim.py 100.00% <100.00%> (ø)
piq/ssim.py 100.00% <100.00%> (ø)

@zakajd
Copy link
Collaborator Author

zakajd commented Jan 22, 2021

Ready for review.
Test failed due to duplicate code in tests/test_ssim.py and tests/test_ms_ssim.py. This will be fixed during test refactoring (#97 )

Copy link
Collaborator

@denproc denproc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch with downsampling bug!
Minor comments about the documentation is to be resolved. Others are optional and can be discussed.

piq/ms_ssim.py Outdated Show resolved Hide resolved
return msssim_val
ssim_val = ss.mean(dim=(-1, -2))
cs = cs.mean(dim=(-1, -2))
return ssim_val, cs


def _ssim_per_channel_complex(x: torch.Tensor, y: torch.Tensor, kernel: torch.Tensor,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we update the complex-valued SSIM similarly to SSIM?

k1=self.k1, k2=self.k2)
return torch.ones_like(score) - score


def _ssim_per_channel(x: torch.Tensor, y: torch.Tensor, kernel: torch.Tensor,
data_range: Union[float, int] = 1., k1: float = 0.01,
k2: float = 0.03) -> Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case of avoiding Unions as much as possible, we should replace it with List[torch.Tensor].

@snk4tr
Copy link
Contributor

snk4tr commented Jan 27, 2021

Hey guys. Sonar check fails because of code duplication. As far as I can see, all duplications are in tests, which is absolutely fine with me. I will also remove piq/feature_extractors from the coverage report since it contains standard feature extraction model architecture that is not means to be tested.

Let me know if you think different. Otherwise I will modify Sonar rules to ignore duplications in tests and coverage in piq/feature_extractors.

@zakajd
Copy link
Collaborator Author

zakajd commented Jan 27, 2021

@snk4tr disagree.
Let's leave Sonar cloud checks for feature extractors and add proper testing for models instead. At least check that forward call runs without failure.
We can also move new models there (like PieAPP)

@snk4tr
Copy link
Contributor

snk4tr commented Jan 28, 2021

@denproc what do you think?

@snk4tr
Copy link
Contributor

snk4tr commented Jan 30, 2021

@denproc what do you think?

Ok, I see that @denproc liked the idea of not removing feature extractors from the coverage report. Reverted this change.

@snk4tr
Copy link
Contributor

snk4tr commented Feb 15, 2021

@zakajd are you working on the one? Let's finilize this one. Let me know if you need any help.

@zakajd
Copy link
Collaborator Author

zakajd commented Feb 18, 2021

@snk4tr this one can be merged if there is no failing tests.

I didn't renamed variables in ms-ssim as proposed by Denis because its better to do that in a separate PR (along with some other renamings).

@sonarcloud
Copy link

sonarcloud bot commented Feb 26, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@snk4tr
Copy link
Contributor

snk4tr commented Mar 1, 2021

@zakajd are you still working on this one or it is ready?

@zakajd
Copy link
Collaborator Author

zakajd commented Mar 1, 2021

@snk4tr it's ready

Copy link
Contributor

@snk4tr snk4tr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR looks good, ready to merge.
@zakajd FYI: it is easier to follow PRs when they are make to solve a single issue. In this case, the PR is both about fixing the bug and refactoring, which confuses things a little. Consider making separate PRs for these two next time.
P.S. If you did that because of long review time, do not let long processes to break your style 😉

@snk4tr snk4tr merged commit 49c3663 into master Mar 2, 2021
@snk4tr snk4tr deleted the bug/ssim branch March 2, 2021 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug in SSIM: Can't reproduce paper values Split code for SSIM and MS-SSIM into separate files
3 participants