Group warnings by message instead of by test id

[nicoddemus]

Currently pytest shows warnings grouped by node id. What usually happens is that in large codebases the offending code ends up called from multiple tests, generating a lot of output which usually points to the same code over and over again.

For example:

import warnings
import pytest

def func():
    warnings.warn(UserWarning("foo"))

@pytest.mark.parametrize("i", range(5))
def test_foo(i):
    func()

def test_bar():
    func()

Currently generates:

========================= warnings summary ==========================
test-warning-grouping.py::test_foo[0]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[1]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[2]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[3]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[4]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_bar
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 6 passed, 6 warnings in 0.03 seconds ================

This gives the impression that the user has a ton of warnings to fix, even though they all point to the same location. This is even worse when there are multiple warnings involved, because it is hard to find the actual offending code in the flood of messages generated by pytest.

This is the output with this PR:

========================= warnings summary ==========================
test-warning-grouping.py::test_foo[0]
test-warning-grouping.py::test_foo[1]
test-warning-grouping.py::test_foo[2]
test-warning-grouping.py::test_foo[3]
test-warning-grouping.py::test_foo[4]
test-warning-grouping.py::test_bar
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 6 passed, 6 warnings in 0.04 seconds ================

This makes it much easier to see that we only have to fix the warning in one place, instead of several like the previous output, while still showing all the affected tests.

This also fixes #4402, although not for the reasons discussed there.

[codecov]

Codecov Report

Merging #4665 into master will increase coverage by <.01%.
The diff coverage is 92.59%.

@@            Coverage Diff             @@
##           master    #4665      +/-   ##
==========================================
+ Coverage   95.71%   95.71%   +<.01%     
==========================================
  Files         113      113              
  Lines       24882    24893      +11     
  Branches     2458     2458              
==========================================
+ Hits        23815    23826      +11     
  Misses        754      754              
  Partials      313      313

Flag	Coverage Δ
#docs	29.56% <3.7%> (+0.06%)	⬆️
#doctesting	29.56% <3.7%> (+0.06%)	⬆️
#linting	29.56% <3.7%> (+0.06%)	⬆️
#linux	95.54% <92.59%> (+0.01%)	⬆️
#nobyte	92.32% <92.59%> (ø)	⬆️
#numpy	93.12% <92.59%> (-0.02%)	⬇️
#pexpect	42.05% <66.66%> (-0.03%)	⬇️
#py27	93.71% <92.59%> (+0.02%)	⬆️
#py34	91.83% <92.59%> (+0.07%)	⬆️
#py35	91.85% <92.59%> (+0.05%)	⬆️
#py36	91.87% <92.59%> (+0.07%)	⬆️
#py37	93.87% <92.59%> (+0.01%)	⬆️
#trial	93.12% <92.59%> (-0.02%)	⬇️
#windows	93.91% <92.59%> (ø)	⬆️
#xdist	93.76% <92.59%> (+0.01%)	⬆️

Impacted Files	Coverage Δ
testing/test_warnings.py	98.9% <100%> (+0.04%)	⬆️
src/_pytest/terminal.py	91.81% <90%> (+0.05%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba452db...9543d19. Read the comment docs.

[codecov]

Codecov Report

Merging #4665 into master will increase coverage by <.01%.
The diff coverage is 92.59%.

@@            Coverage Diff             @@
##           master    #4665      +/-   ##
==========================================
+ Coverage   95.71%   95.71%   +<.01%     
==========================================
  Files         113      113              
  Lines       24882    24893      +11     
  Branches     2458     2458              
==========================================
+ Hits        23815    23826      +11     
  Misses        754      754              
  Partials      313      313

Flag	Coverage Δ
#docs	29.56% <3.7%> (+0.06%)	⬆️
#doctesting	29.56% <3.7%> (+0.06%)	⬆️
#linting	29.56% <3.7%> (+0.06%)	⬆️
#linux	95.54% <92.59%> (+0.01%)	⬆️
#nobyte	92.32% <92.59%> (ø)	⬆️
#numpy	93.12% <92.59%> (-0.02%)	⬇️
#pexpect	42.05% <66.66%> (-0.03%)	⬇️
#py27	93.71% <92.59%> (+0.02%)	⬆️
#py34	91.83% <92.59%> (+0.07%)	⬆️
#py35	91.85% <92.59%> (+0.05%)	⬆️
#py36	91.87% <92.59%> (+0.07%)	⬆️
#py37	93.87% <92.59%> (+0.01%)	⬆️
#trial	93.12% <92.59%> (-0.02%)	⬇️
#windows	93.91% <92.59%> (ø)	⬆️
#xdist	93.76% <92.59%> (+0.01%)	⬆️

Impacted Files	Coverage Δ
testing/test_warnings.py	98.9% <100%> (+0.04%)	⬆️
src/_pytest/terminal.py	91.81% <90%> (+0.05%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba452db...9543d19. Read the comment docs.

[RonnyPfannschmidt]

Lovely, just awesome, now i know what i felt was missing but couldnt quite figure